Driving safety monitoring method based on eye movement tracking and target perception technology

Abstract

The invention discloses a driving safety monitoring method based on eye movement tracking and a target perception technology, which is suitable for a motor vehicle driving scene. The method comprises the following steps: S1, collecting eye movement data of a driver in different driving scenes through eye movement tracking equipment, and converting the data into a standard format which can be understood by a fusion model; s2, collecting traffic environment data including traffic object information, distance information, speed information and lane line information by using traffic environment sensing equipment integrated with an environment sensing algorithm; and S3, inputting the eye movement data and the traffic environment data into the fusion model, and judging whether a driver notices all traffic objects with interaction risks through calculation so as to realize driving safety monitoring. The method has the advantages that by fusing the eye movement data and the traffic environment information, the attention distribution of the driver can be accurately monitored, and the traffic safety management level is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of driving safety monitoring, and in particular to an evaluation of a driving safety monitoring method based on eye tracking and target perception technology. Background Art

[0002] This method is mainly used in driving scenarios. In complex traffic environments, the driver's visual behavior and reaction ability have an important impact on the occurrence of traffic accidents. Traditional traffic safety monitoring methods mainly rely on the monitoring of information such as vehicle speed, road conditions, and traffic signals, but these methods often cannot fully evaluate the driver's attention allocation and visual response. Eye tracking technology combined with target perception technology can monitor the driver's visual behavior and perception of the traffic environment in real time, thereby providing a more accurate assessment of driving safety. Specific application scenarios include but are not limited to complex road environments such as unsignaled intersections, urban roads with heavy traffic, and highways with high speeds. In these scenarios, the driver's concentration and reaction speed directly affect traffic safety.

[0003] Past research and technology have mainly focused on two major directions: target detection and eye tracking. Target detection technology can identify targets such as vehicles, pedestrians, and traffic signs on the road in real time through image processing and deep learning algorithms, and conduct risk assessment based on this information, such as the Chinese patents with publication numbers CN118247766A and CN118196373A. However, target detection technology often ignores the driver's visual behavior and his perception of traffic objects, and cannot fully reflect the driver's actual attention distribution. On the other hand, eye tracking technology can accurately record the driver's eye movements, analyze its gaze points, scanning paths and other features, and thus evaluate the driver's concentration, such as the Chinese patents with publication numbers CN117409394A and CN105205443B. However, existing eye tracking technology mainly focuses on the attention monitoring of individual drivers, and lacks an effective combination of comprehensive perception of the traffic environment and multi-target interactive analysis.

[0004] Although target detection and eye tracking each have certain advantages, existing technologies are mostly limited to the application of a single technology, and there is a lack of effective practice to combine the two. The integration of target detection and eye tracking technology can simultaneously monitor the driver's visual behavior and environmental perception, providing a more comprehensive and accurate driving safety assessment. However, there are still few studies and practices that combine these two technologies and apply them to actual driving safety monitoring, and there are problems such as difficulty in technical integration and complex data processing. Therefore, there is an urgent need for a new method to effectively combine eye tracking and target perception technology to improve the accuracy and real-time performance of driving safety monitoring. Summary of the invention

[0005] The purpose of the present invention is to provide a driving safety monitoring method based on eye tracking and target perception technology, and to apply the method to complex traffic scenarios, especially to scenarios such as unsignalized intersections, urban roads with heavy traffic, and highways with high speeds, to provide a basis for traffic safety monitoring and driver risk assessment.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A driving safety monitoring method based on eye tracking and target perception technology comprises the following steps:

[0008] S1. Collect the driver’s eye movement data in different driving scenarios through eye tracking equipment, and convert the data into a standard format that can be understood by the fusion model.

[0009] S2. Use traffic environment perception equipment with integrated environment perception algorithm to collect traffic environment data, including traffic object information, distance information, speed information and lane line information.

[0010] S3. Input the eye movement data and traffic environment data into the fusion model, and determine through calculation whether the driver has noticed all traffic objects with interaction risks, so as to realize driving safety monitoring.

[0011] A further improvement of the present invention is that: the step S1, collecting the driver's eye movement data in different driving scenarios, specifically includes:

[0012] S11, eye tracking program calibration;

[0013] Calibration is performed by the driver looking at multiple preset points on the calibration board. During the calibration process, the tester adjusts the relevant parameters of the coordinate system based on the feedback from the eye tracking device;

[0014] Among them, the tester adjusts the initial position of the x-axis and the initial position of the y-axis. These two parameters are only adjusted at the first point, and the initial positions of the remaining points remain fixed;

[0015] At the same time, adjust the x-axis scaling factor and the y-axis scaling factor to calibrate the field of view and coordinate accuracy of the device;

[0016] The calibration is completed when the deviation between the estimated gaze point recorded by the eye tracking device and the actual gaze point is less than a preset error range ξ. The deviation can be calculated by the following formula:

[0017]

[0018] Among them, (x est ,y est) is the estimated gaze point coordinate recorded by the eye tracking device, (x act ,y act ) is the actual gaze point coordinate, Δd is the Euclidean distance between the two points, and ξ is the preset error threshold. When Δd≤ξ, the calibration is considered complete and the device enters normal working state.

[0019] S12, multi-scene eye movement data collection;

[0020] The eye tracking device is used to collect the driver's eye movement data in different driving scenarios, including but not limited to unsignalized intersections, urban roads with heavy traffic, and highways with high speeds;

[0021] The driver's eye movement data includes gaze features and scan features, wherein the gaze features include gaze point position, gaze duration and gaze frequency; and the scan features include scan speed, scan frequency and scan amplitude;

[0022] The key evaluation features in the unsignalized intersection are gaze point location, gaze frequency, scan frequency and scan amplitude;

[0023] The key evaluation features of the urban road section with heavy traffic volume are the gaze point position, gaze frequency, scanning frequency and scanning amplitude;

[0024] The key evaluation features of the highway section with faster vehicle speed are the gaze point position, gaze duration and scanning speed.

[0025] S13. Convert the collected eye movement data into a standard format that can be processed by the fusion model: The details are as follows:

[0026] The fixation features and scan features were processed by data preprocessing and standardization, and the data were scaled using a normalization algorithm.

[0027] Then, the time series analysis algorithm is used to process the dynamic data, extract key time series features, and generate a standardized data input format.

[0028] The steps of extracting key timing features are as follows:

[0029] S131, segment the eye movement data into time series: the details are as follows:

[0030] The eye movement data is divided into fixed time windows, which can be expressed as follows:

[0031] X i ={x (i-1)s+1 ,x (i -1)s+2,...,x (i-1)s+l} (twenty two)

[0032] Among them, Xi is the i-th time window, l is the time window length, and s is the sliding step size.

[0033] S132, time window statistical feature extraction: the details are as follows:

[0034] Calculate the X for each time window i The key statistical characteristics of include mean, standard deviation, maximum and minimum values, which are specifically expressed as:

[0035]

[0036]

[0037] x max =max(X i ) (25)

[0038] x min =min(X i ) (26)

[0039] S133, time series frequency domain feature extraction: the details are as follows:

[0040] The time series is subjected to a fast Fourier transform (FFT) to extract the frequency component and energy, which is specifically expressed as follows:

[0041]

[0042] Among them, the characteristic values ​​include the main frequency f max And the total energy E: specifically expressed as:

[0043]

[0044] S134, feature conversion standardized data format: the details are as follows:

[0045] The above features are standardized to the interval [0, 1] to facilitate input into the model, specifically expressed as:

[0046]

[0047] A further improvement of the present invention is that: the step S2, using a traffic environment perception device integrated with an environment perception algorithm to collect traffic environment data, specifically includes:

[0048] S21. Dataset collection and data enhancement: Details are as follows:

[0049] The dataset is based on the BDD100K and KITTI databases, and the raw data containing traffic environment objects are screened;

[0050] The traffic environment objects include but are not limited to road images, videos, vehicle speeds, traffic signs, lane lines, pedestrians, other traffic objects and other information;

[0051] The data must cover different traffic scenarios, such as intersections without signalization, urban roads with heavy traffic, and highways with high speeds;

[0052] The data enhancement includes random cropping, rotation, and color transformation to expand the diversity of the training set:

[0053] S211, random cropping: specifically expressed as:

[0054] Randomly select an area on the original image for cropping, and the cropped area is used to construct a new training sample;

[0055] The cropping method should first randomly select a cropping window with a size of (α×width, α×height), where α is the cropping ratio.

[0056] S212, random rotation: specifically expressed as:

[0057] Randomly rotating the image to simulate different perspective changes;

[0058] The random rotation method should first select a rotation angle θ∈[-θ max ,θ max ], where θ max is the maximum rotation angle;

[0059] The image is rotated by a corresponding angle to generate a new training sample.

[0060] S213, color change: specifically expressed as:

[0061] The image brightness adjustment has a change range of 50% to 150% of the original image brightness;

[0062] The image contrast adjustment has a transformation range of 50% to 200% of the original image contrast;

[0063] The image saturation adjustment has a transformation range of 50% to 200% of the original image saturation.

[0064] S22. Establish a multi-task model integrating environmental perception algorithms: The details are as follows:

[0065] The multi-task model of the integrated environment perception algorithm includes object detection, instance segmentation, depth estimation and lane detection functions;

[0066] The target detection and instance segmentation functions in the multi-task model of the integrated environment perception algorithm are based on the YOLO (You Only Look Once) algorithm;

[0067] The depth estimation function in the multi-task model of the integrated environment perception algorithm is based on the Monodepth depth estimation algorithm;

[0068] The lane detection function in the multi-task model of the integrated environment perception algorithm is based on the LaneNet algorithm.

[0069] S23. Configure parameters on the enhanced data set and train the above model:

[0070] The parameters that need to be adjusted during the training process include but are not limited to learning rate, batch size, and number of training rounds;

[0071] The training process requires medium or higher GPU devices;

[0072] The training evaluation indicators are accuracy ACC, prediction rate Pr, recall rate Re, and F1-score. The specific calculation formula is as follows:

[0073]

[0074]

[0075]

[0076]

[0077] Among them: Tp is the number of positive classes predicted as positive classes; Fn is the number of positive classes predicted as negative classes; Fp is the number of negative classes predicted as positive classes; Tn is the number of negative classes predicted as negative classes.

[0078] S24. Deploy the model to hardware devices and collect real-time traffic environment data in actual traffic environments: The details are as follows:

[0079] The hardware device includes at least a memory and a processor;

[0080] The memory is used to store a machine-readable program for executing a traffic scene multi-target detection method;

[0081] The memory may be a non-volatile memory, such as a solid state drive (SSD), a hard disk drive (HDD) or a flash memory, or a volatile memory, such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), for storing all necessary information including an algorithm model, training data, configuration parameters and execution code;

[0082] The processor is connected to the memory, can access the machine-readable program stored in the memory, and executes the traffic scene multi-target detection method based on the program;

[0083] The processor accelerates the data processing process through a parallel computing architecture and performs computing processes including but not limited to object detection, instance segmentation, depth estimation, lane line detection, etc.

[0084] The processor coordinates the work of each module by calling the instructions and algorithms in the machine-readable program, processes the data collected by the sensor in real time, performs multi-target detection tasks, and generates corresponding output results.

[0085] A further improvement of the present invention is that the hardware aspects of step S1 and step S2 specifically include:

[0086] The eye tracking device and the traffic environment sensing device used in step S1 and step S2 are integrated into a pair of glasses;

[0087] The eyeglass device includes a lens ring, a middle bridge, temples, anti-slip ear hooks, a front-view camera and a rear-view camera;

[0088] The eyeglass device is made by 3D printing technology;

[0089] The front-view camera is installed above the center beam to obtain perception information of the traffic environment;

[0090] The rear-view camera is installed in front of the center beam through a short cross beam to realize the driver's eye movement information tracking;

[0091] During the test, the driver wearing the eyewear device can simultaneously achieve real-time perception of the traffic environment and synchronous tracking of the driver's eye movement information.

[0092] A further improvement of the present invention is that: the step S3, inputting the eye movement data and the traffic environment data into the fusion model, specifically includes:

[0093] The fusion model is a model based on the combination of Transformer and Long Short-Term Memory Network (LSTM) to process eye movement data and traffic environment data;

[0094] The fusion model treats eye movement data as time series data and conducts in-depth analysis on it through the model, combining the information from target detection and instance segmentation, depth estimation and lane line detection models to determine whether the driver pays sufficient attention to all traffic objects with interaction risks.

[0095] A further improvement of the present invention is that: the step S3, inputting the eye movement data and the traffic environment data into the fusion model, specifically includes:

[0096] S31. Preprocess the eye movement data: the details are as follows:

[0097] The eye movement data is collected through step S1;

[0098] Then, the eye movement data is serialized in time order and passed into the fusion model as input data;

[0099] Specifically, the eye movement data is regarded as time series data, and the eye movement data is segmented using a fixed time window length l to generate data samples, which can be specifically expressed as:

[0100] X={X1,X2,···,X n},X n ∈R l×3 (34)

[0101] Among them, l is the length of the time window and n is the number of samples.

[0102] S32, feature encoding and input: The details are as follows:

[0103] The eye movement data is encoded through a fully connected layer (FC) to generate a higher-dimensional feature vector, which is specifically expressed as:

[0104]

[0105] in, W is the weight matrix, b is the bias term, and m is the encoded feature dimension.

[0106] S33, self-attention mechanism and residual connection: The details are as follows:

[0107] The self-attention mechanism and residual connection step first further captures the global dependency between eye movement data and traffic environment data through the self-attention mechanism, which is specifically expressed as:

[0108]

[0109] Among them, Q is the query vector, K is the key vector, V is the numerical vector, and d k is the dimension of K;

[0110] Multihead(Q,K,V)=Concat(head 1 ,......,head h )W o (37)

[0111]

[0112] Where h is the number of attention heads,

[0113] The model can automatically assign weights to different eye movement data and traffic environment data through a self-attention mechanism, thereby enhancing attention to key traffic objects;

[0114] Then the model is layer normalized, which is specifically expressed as:

[0115]

[0116] S34. Use the ReLU feed-forward network to activate feature information: The details are as follows:

[0117] O f =ReLU(WO1+b) (40)

[0118] S35, perform residual connection and normalization again: the details are as follows:

[0119] O j =LayerNorm( f +O i ) (41)

[0120] S36. Use LSTM network with attention mechanism to model the temporal dependency between eye movement data and traffic environment data:

[0121] The LSTM module includes an input gate, a forget gate and an output gate;

[0122] The LSTM module is updated by the new input data, which is specifically expressed as:

[0123] i t =Sigmod(W i X t +A i h t-1 +b i ) (42)

[0124] f t =Sigmod(W f X t +A f h t-1 +b f ) (43)

[0125] o t =Sigmod(W o X t +A o h t-1 +bo ) (44)

[0126]

[0127]

[0128] Among them, i t , f t , o t They are input gate, forget gate, output gate, and C t is the cell state, h t is the hidden state, tanh() is the hyperbolic tangent activation function, W, A, b are learnable weights and bias terms.

[0129] S37. The weighted sum of each hidden state is as follows:

[0130] The weights are: based on the hidden states generated by the LSTM module, the weights are assigned to each hidden state by introducing the attention mechanism, specifically:

[0131]

[0132] The output of the LSTM network is obtained by weighted summation, which is specifically expressed as:

[0133]

[0134] S38. Determination of driver's attention, as follows:

[0135] The output F of the LSTM network is classified and mapped by the softmax function to determine whether the driver has noticed the traffic object with interaction risk, which is specifically expressed as follows:

[0136] R = softmax(WF+b) (49)

[0137] Where: R is the driver's traffic object perception prediction result, indicating whether the driver notices the traffic object with interaction risk.

[0138] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0139] Compared with the single eye movement data or single traffic environment data analysis commonly used in the prior art, the present invention proposes a driving safety monitoring method based on multi-source data fusion by comprehensively applying eye movement tracking technology and traffic environment perception technology. This method not only takes into account the driver's visual attention characteristics, but also combines multiple factors in the traffic environment for comprehensive analysis, which can more accurately determine whether the driver pays enough attention to potential risk objects in the traffic environment.

[0140] By integrating multiple perception technologies such as target detection and instance segmentation, depth estimation, and lane line detection, the present invention can monitor the driver's attention to interactive risk traffic objects in real time and quantify the driver's risk perception ability. The application of this method can not only improve the accuracy of driving safety monitoring, but also effectively improve the intelligent driving system's ability to respond to complex traffic scenarios.

[0141] In addition, the present invention combines eye movement data and traffic environment data, and uses the Transformer-LSTM model to analyze and judge the driver's attention state, thereby providing real-time driver status feedback and traffic safety warning for the intelligent transportation system. This method has broad application prospects in highways and complex urban traffic environments, and helps to improve road traffic security capabilities and optimize traffic safety management. BRIEF DESCRIPTION OF THE DRAWINGS

[0142] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0143] Figure 1 This is a flow chart of the driving safety monitoring method of the present invention.

[0144] Figure 2 A calibration plate drawn for the eye tracking estimation model parameter calibration process.

[0145] Figure 3 The driving safety monitoring device for providing the driving safety information by analyzing the information obtained from the eye movement device and the traffic environment perception device involved in an embodiment of the present invention is schematically illustrated.

[0146] Figure 4 The modular structure flow chart of the embodiment of the present invention. DETAILED DESCRIPTION

[0147] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0148] The purpose of the present invention is to provide a driving safety monitoring method based on eye tracking and target perception technology, which provides a basis for improving road traffic security capabilities and optimizing traffic safety management in highway and complex urban traffic environments.

[0149] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0150] Figure 1 Flow chart of a driving safety monitoring method based on eye tracking and target perception technology according to an embodiment of the present invention; Figure 1 As shown, the present invention provides a driving safety monitoring method based on eye tracking and target perception technology, comprising the following steps:

[0151] S1. Collect the driver’s eye movement data in different driving scenarios through eye tracking equipment, and convert the data into a standard format that can be understood by the fusion model, as follows:

[0152] S11, eye tracking program calibration;

[0153] In this step, the calibration is performed by the driver looking at multiple preset points on the calibration board. During the calibration process, the tester adjusts the relevant parameters of the coordinate system based on the feedback from the eye tracking device;

[0154] In this step, the tester adjusts the initial position of the x-axis and the initial position of the y-axis. These two parameters are only adjusted at the first point, and the initial positions of the remaining points remain fixed;

[0155] In this step, adjust the x-axis scaling factor and the y-axis scaling factor at the same time to calibrate the field of view and coordinate accuracy of the device;

[0156] In this step, the calibration is completed when the deviation between the estimated gaze point recorded by the eye tracking device and the actual gaze point is less than the preset error range ξ. The deviation can be calculated by the following formula:

[0157]

[0158] Among them, (x est ,y est) is the estimated gaze point coordinate recorded by the eye tracking device, (x act ,y act ) is the actual gaze point coordinate, Δd is the Euclidean distance between the two points, and ξ is the preset error threshold, which is 1 cm in this embodiment. When Δd≤ξ, the calibration is determined to be completed and the device enters the normal working state.

[0159] S12, multi-scene eye movement data collection;

[0160] In this step, the eye tracking device is used to collect the driver's eye movement data in different driving scenarios, including but not limited to unsignalized intersections, urban roads with heavy traffic, and highways with high speeds;

[0161] In this step, the driver's eye movement data includes gaze features and scan features. The gaze features include gaze point position, gaze duration and gaze frequency; the scan features include scan speed, scan frequency and scan amplitude.

[0162] In this step, the key evaluation features in the unsignalized intersection are gaze point position, gaze frequency, scan frequency and scan amplitude;

[0163] In this step, the key evaluation features of the urban road section with heavy traffic volume are the gaze point position, gaze frequency, scanning frequency and scanning amplitude;

[0164] In this step, the key evaluation features of the highway section with faster vehicle speed are the gaze point position, gaze duration and scanning speed.

[0165] S13. Convert the collected eye movement data into a standard format that can be processed by the fusion model: The details are as follows:

[0166] In this step, the gaze features and scan features are preprocessed and standardized, and the data are scaled using a normalization algorithm.

[0167] In this step, the time series analysis algorithm is then used to process the dynamic data, extract key timing features, and generate a standardized data input format.

[0168] In this step, extracting key features is divided into the following steps:

[0169] S131, segment the eye movement data into time series: the details are as follows:

[0170] In this step, the eye movement data is segmented according to fixed time windows, which is specifically expressed as:

[0171] X i ={x (i-1)s+1 ,x (i-1)s+2 ,...,x(i-1)s+l} (51)

[0172] Among them, X i is the i-th time window, l is the time window length, and s is the sliding step size. In this embodiment, l is 10 seconds and s is 5 seconds.

[0173] S132, time window statistical feature extraction: the details are as follows:

[0174] In this step, each time window X is calculated i The key statistical characteristics of include mean, standard deviation, maximum and minimum values, which are specifically expressed as:

[0175]

[0176]

[0177] x max =max(X i ) (54)

[0178] x min =min(X i ) (55)

[0179] S133, time series frequency domain feature extraction: the details are as follows:

[0180] In this step, the time series is subjected to a fast Fourier transform (FFT) to extract the frequency components and energy, which is specifically expressed as:

[0181]

[0182] Among them, the characteristic values ​​include the main frequency f max And the total energy E: specifically expressed as:

[0183]

[0184] S134, feature conversion standardized data format: the details are as follows:

[0185] In this step, the above features are standardized to the interval [0, 1] for easy input into the model, specifically expressed as:

[0186]

[0187] In this step, the hardware aspects of step S1 and step S2 specifically include:

[0188] In this step, the eye tracking device and the traffic environment sensing device used in step S1 and step S2 are integrated into a pair of glasses;

[0189] In this step, the eyeglass device includes a lens ring, a middle bridge, temples, anti-slip ear hooks, a front-view camera and a rear-view camera;

[0190] In this step, the eyewear device is made by 3D printing technology;

[0191] In this step, the front-view camera is installed above the center beam to obtain perception information of the traffic environment;

[0192] In this step, the rear-view camera is installed in front of the middle beam through a short crossbeam to realize the driver's eye movement information tracking;

[0193] In this step, during the test, the driver wears the eyewear device, which can simultaneously achieve real-time perception of the traffic environment and synchronous tracking of the driver's eye movement information.

[0194] S2. Use traffic environment perception equipment with integrated environment perception algorithms to collect traffic environment data, including traffic object information, distance information, speed information, and lane line information, as follows:

[0195] S21. Dataset collection and data enhancement: Details are as follows:

[0196] In this step, the data set is based on the BDD100K and KITTI databases, and the original data containing traffic environment objects is screened;

[0197] In this step, the traffic environment objects include but are not limited to road images, videos, vehicle speeds, traffic signs, lane lines, pedestrians, other traffic objects and other information;

[0198] In this step, the data must cover different traffic scenarios, such as intersections without signalization, urban roads with heavy traffic, and highways with high speeds;

[0199] In this step, the data enhancement includes random cropping, rotation, and color transformation to expand the diversity of the training set, which is specifically expressed as:

[0200] S211, random cropping: specifically expressed as:

[0201] In this step, a region is randomly selected on the original image for cropping, and the cropped region is used to construct a new training sample;

[0202] In this step, the cropping method should first randomly select a cropping window with a size of (α×width, α×height), where α is the cropping ratio.

[0203] S212, random rotation: specifically expressed as:

[0204] In this step, the image is randomly rotated to simulate different perspective changes;

[0205] In this step, the random rotation method should first select the rotation angle θ∈[-θ max ,θ max ], where θ max is the maximum rotation angle;

[0206] In this step, the image is rotated by a corresponding angle to generate a new training sample.

[0207] S213, color change: specifically expressed as:

[0208] In this step, the image brightness is adjusted, and the transformation range is 50% to 150% of the original image brightness;

[0209] In this step, the image contrast is adjusted, and the transformation range is 50% to 200% of the original image contrast;

[0210] In this step, the image saturation is adjusted, and the transformation range is 50% to 200% of the original image saturation.

[0211] S22. Establish a multi-task model integrating environmental perception algorithms: The details are as follows:

[0212] In this step, the multi-task model of the integrated environment perception algorithm includes target detection, instance segmentation, depth estimation and lane line detection functions;

[0213] In this step, the target detection and instance segmentation functions in the multi-task model of the integrated environment perception algorithm are based on the YOLO (You Only Look Once) algorithm;

[0214] In this step, the depth estimation function in the multi-task model of the integrated environment perception algorithm is based on the Monodepth depth estimation algorithm;

[0215] In this step, the lane detection function in the multi-task model of the integrated environment perception algorithm is based on the LaneNet algorithm.

[0216] S23. Configure parameters on the enhanced data set and train the above model:

[0217] In this step, the parameters that need to be adjusted in the training process include but are not limited to learning rate, batch size, and number of training rounds;

[0218] In this step, the training process requires a medium or higher GPU device;

[0219] In this step, the training evaluation indicators are accuracy ACC, prediction rate Pr, recall rate Re, and F1-score. The specific calculation formula is as follows:

[0220]

[0221]

[0222]

[0223]

[0224] Among them: Tp is the number of positive classes predicted as positive classes; Fn is the number of positive classes predicted as negative classes; Fp is the number of negative classes predicted as positive classes; Tn is the number of negative classes predicted as negative classes.

[0225] S24. Deploy the model to hardware devices and collect real-time traffic environment data in actual traffic environments: The details are as follows:

[0226] In this step, the hardware device includes at least a memory and a processor;

[0227] In this step, the memory is used to store a machine-readable program for executing the traffic scene multi-target detection method;

[0228] In this step, the memory may be a non-volatile memory, such as a solid state drive (SSD), a hard disk drive (HDD) or a flash memory, or a volatile memory, such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), for storing all necessary information including the algorithm model, training data, configuration parameters and execution code;

[0229] In this step, the processor is connected to the memory, can access the machine-readable program stored in the memory, and executes the traffic scene multi-target detection method based on the program;

[0230] In this step, the processor accelerates the data processing process through a parallel computing architecture and performs computing processes including but not limited to target detection, instance segmentation, depth estimation, lane line detection, etc.

[0231] In this step, the processor coordinates the work of each module by calling the instructions and algorithms in the machine-readable program, processes the data collected by the sensor in real time, performs the multi-target detection task, and generates corresponding output results;

[0232] In this step, in this embodiment, Figure 3 The driving safety monitoring device 300 is schematically illustrated to analyze the information obtained from the eye movement device and the traffic environment perception device involved in an embodiment of the present invention to provide the driving safety information;

[0233] In this step, in this embodiment, refer to Figure 3 , the driving safety monitoring device 300 may include a data input 301 for receiving data from an eye tracking device and a traffic environment perception device, a memory 302 for storing instructions required for calculation and receiving data, a processor 303 for executing a process corresponding to the instructions stored in the memory 302, a deep learning accelerator 304 for improving data processing efficiency and accelerating deep learning model training and reasoning, and a safety decision module 305 for providing corresponding driving safety suggestions based on the results analyzed by the processor 303 and the deep learning model, combined with the driver's attention state information and driving environment information;

[0234] In this step, the instruction analyzes the driver's eye movement data and traffic environment data collected from the eye tracking device and the traffic environment perception device by using a deep learning model (such as an LSTM neural network), thereby obtaining driving environment information and driver's attention state information;

[0235] In this step, the data input interface 301 receives data transmitted by the environment sensing device;

[0236] In this step, the memory 302 is used to store the instructions and data required for the calculation, including the deep learning model, the training data set, the traffic environment perception data and its related parameters. In this embodiment, the memory uses a 512G solid state drive;

[0237] In this step, the processor 303 can analyze the eye tracking data and the traffic environment data obtained by algorithms such as target detection, instance segmentation and depth estimation through the deep learning model, so as to extract driving environment information and driver status information. Finally, based on the driving environment information and the driver's attention status information, corresponding driving safety information is provided to support the driver's safety decision-making in a complex traffic environment;

[0238] In this step, the deep learning acceleration module 304 is a Nvidia RTX3060Ti GPU, which can process a large number of matrix operations and deep learning model reasoning tasks in parallel, significantly improving the response speed of the real-time driving safety monitoring system;

[0239] S3. Input the eye movement data and traffic environment data into the fusion model, and determine whether the driver has noticed all traffic objects with interaction risks through calculation, so as to realize driving safety monitoring, as follows:

[0240] S31. Preprocess the eye movement data: the details are as follows:

[0241] In this step, the eye movement data is collected through step S1;

[0242] In this step, the eye movement data is serialized in chronological order and passed into the fusion model as input data;

[0243] In this step, specifically, the eye movement data is regarded as time series data, and the eye movement data is segmented using a fixed time window length l to generate data samples, which is specifically expressed as:

[0244] X={X1,X2,···,X n},X n ∈R l×3 (63)

[0245] Among them, l is the length of the time window and n is the number of samples.

[0246] S32, feature encoding and input: The details are as follows:

[0247] In this step, the eye movement data is encoded through a fully connected layer (FC) to generate a higher-dimensional feature vector, which is specifically expressed as:

[0248]

[0249] in, W is the weight matrix, b is the bias term, and m is the encoded feature dimension.

[0250] S33, self-attention mechanism and residual connection: The details are as follows:

[0251] In this step, the self-attention mechanism and residual connection step firstly further capture the global dependency between the eye movement data and the traffic environment data through the self-attention mechanism, which is specifically expressed as:

[0252]

[0253] Among them, Q is the query vector, K is the key vector, V is the numerical vector, and d k is the dimension of K;

[0254] Multihead(Q,K,V)=Concat(head 1 ,......,head h )W o (66)

[0255]

[0256] Where h is the number of attention heads,

[0257] In this step, the model can automatically assign weights to different eye movement data and traffic environment data through a self-attention mechanism, thereby enhancing attention to key traffic objects;

[0258] In this step, the model is layer normalized, which is specifically expressed as:

[0259]

[0260] S34. Use the ReLU feed-forward network to activate feature information: The details are as follows:

[0261] O f =ReLU(WO1+b) (69)

[0262] S35, perform residual connection and normalization again: the details are as follows:

[0263] O j =LayerNorm( f +O i ) (70)

[0264] S36. Use LSTM network with attention mechanism to model the temporal dependency between eye movement data and traffic environment data:

[0265] In this step, the LSTM module includes an input gate, a forget gate and an output gate;

[0266] In this step, the LSTM module is updated by the new input data, which is specifically expressed as follows:

[0267] i t =Sigmod(W i X t +A i h t-1 +b i ) (71)

[0268] f t =Sigmod(W f X t +A f h t-1 +b f ) (72)

[0269] o t =Sigmod(W o X t +A o h t-1 +b o ) (73)

[0270]

[0271]

[0272] Among them, i t , f t , o t They are input gate, forget gate, output gate, and C t is the cell state, h t is the hidden state, tanh() is the hyperbolic tangent activation function, W, A, b are learnable weights and bias terms.

[0273] S37. The weighted sum of each hidden state is as follows:

[0274] In this step, the weight is: based on the hidden state generated by the LSTM module, the weight is assigned to each hidden state by introducing the attention mechanism, which is specifically expressed as:

[0275]

[0276] In this step, the output of the LSTM network is obtained by weighted summation, which is specifically expressed as follows:

[0277]

[0278] S38. Determination of driver's attention, as follows:

[0279] In this step, the output F of the LSTM network is classified and mapped by the softmax function to determine whether the driver has noticed the traffic object with interaction risk, which is specifically manifested as follows:

[0280] R = softmax(WF+b) (78)

[0281] Where: R is the driver's traffic object perception prediction result, indicating whether the driver notices the traffic object with interaction risk.

Claims

1. A driving safety monitoring method based on eye tracking and target perception technology, characterized in that: The method comprises the following steps: S1. Collect the driver’s eye movement data in different driving scenarios through eye tracking equipment, and convert the data into a standard format that can be understood by the fusion model. S2. Use traffic environment perception equipment with integrated environment perception algorithm to collect traffic environment data, including traffic object information, distance information, speed information and lane line information. S3. Input the eye movement data and traffic environment data into the fusion model, and determine through calculation whether the driver has noticed all traffic objects with interaction risks, so as to realize driving safety monitoring.

2. The driving safety monitoring method based on eye tracking and target perception technology according to claim 1, characterized in that: The step S1, collecting the driver's eye movement data in different driving scenarios, specifically includes: S11, eye tracking program calibration; Calibration is performed by the driver looking at multiple preset points on the calibration board. During the calibration process, the tester adjusts the relevant parameters of the coordinate system based on the feedback from the eye tracking device; Among them, the tester adjusts the initial position of the x-axis and the initial position of the y-axis. These two parameters are only adjusted at the first point, and the initial positions of the remaining points remain fixed; Adjust the x-axis scaling factor and the y-axis scaling factor at the same time to calibrate the device's field of view and coordinate accuracy; The calibration is completed when the estimated gaze point recorded by the eye tracking device completely coincides with the actual gaze point. The calibration process is completed, the parameters are determined, and the device enters normal working state. S12, multi-scene eye movement data collection; The eye tracking device is used to collect the driver's eye movement data in different driving scenarios, including but not limited to unsignalized intersections, urban roads with heavy traffic, and highways with high speeds; The driver's eye movement data includes gaze features and scan features, wherein the gaze features include gaze point position, gaze duration and gaze frequency; and the scan features include scan speed, scan frequency and scan amplitude; The key evaluation features in the unsignalized intersection are gaze point location, gaze frequency, scan frequency and scan amplitude; The key evaluation features of the urban road section with heavy traffic volume are the gaze point position, gaze frequency, scanning frequency and scanning amplitude; The key evaluation features of the highway section with faster vehicle speed are the gaze point position, gaze duration and scanning speed. S13. Convert the collected eye movement data into a standard format that can be processed by the fusion model: The details are as follows: The fixation features and scan features were processed by data preprocessing and standardization, and the data were scaled using a normalization algorithm. Then, the dynamic data (such as scanning path and scanning speed) are processed using a time series analysis algorithm to extract key timing features and generate a standardized data input format.

3. The driving safety monitoring method based on eye tracking and target perception technology according to claim 1, characterized in that: The step S2, collecting traffic environment data using a traffic environment perception device integrated with an environment perception algorithm, specifically includes: S21. Dataset collection and data enhancement: Details are as follows: The dataset is based on the BDD100K and KITTI databases, and the raw data containing traffic environment objects are screened; The traffic environment objects include but are not limited to road images, videos, vehicle speeds, traffic signs, lane lines, pedestrians, other traffic objects and other information; The data must cover different traffic scenarios, such as intersections without signalization, urban roads with heavy traffic, and highways with high speeds; The data enhancement includes random cropping, rotation, and color transformation to expand the diversity of the training set. S22. Establish a multi-task model integrating environmental perception algorithms: The details are as follows: The multi-task model of the integrated environment perception algorithm includes object detection, instance segmentation, depth estimation and lane detection functions; The target detection and instance segmentation functions in the multi-task model of the integrated environment perception algorithm are based on the YOLO (You Only Look Once) algorithm; The depth estimation function in the multi-task model of the integrated environment perception algorithm is based on the Monodepth depth estimation algorithm; The lane detection function in the multi-task model of the integrated environment perception algorithm is based on the LaneNet algorithm. S23. Configure parameters on the enhanced data set and train the above model: The parameters that need to be adjusted during the training process include but are not limited to learning rate, batch size, and number of training rounds; The training process requires medium or higher GPU devices; The training evaluation indicators are accuracy ACC, prediction rate Pr, recall rate Re, and F1-score. The specific calculation formula is as follows: Among them: Tp is the number of positive classes predicted as positive classes; Fn is the number of positive classes predicted as negative classes; Fp is the number of negative classes predicted as positive classes; Tn is the number of negative classes predicted as negative classes. S24. Deploy the model to hardware devices and collect real-time traffic environment data in actual traffic environments: The details are as follows: The hardware device includes at least a memory and a processor; The memory is used to store a machine-readable program for executing a traffic scene multi-target detection method; The memory may be a non-volatile memory, such as a solid state drive (SSD), a hard disk drive (HDD) or a flash memory, or a volatile memory, such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), for storing all necessary information including an algorithm model, training data, configuration parameters and execution code; The processor is connected to the memory, can access the machine-readable program stored in the memory, and executes the traffic scene multi-target detection method based on the program; The processor accelerates the data processing process through a parallel computing architecture and performs computing processes including but not limited to object detection, instance segmentation, depth estimation, lane line detection, etc. The processor coordinates the work of each module by calling the instructions and algorithms in the machine-readable program, processes the data collected by the sensor in real time, performs multi-target detection tasks, and generates corresponding output results.

4. The driving safety monitoring method based on eye tracking and target perception technology according to claim 1, characterized in that: The hardware aspects include the following: The eye tracking device and the traffic environment sensing device used in step S1 and step S2 are integrated into a pair of glasses; The eyeglass device includes a lens ring, a middle bridge, temples, anti-slip ear hooks, a front-view camera and a rear-view camera; The eyeglass device is made by 3D printing technology; The front-view camera is installed above the center beam to obtain perception information of the traffic environment; The rear-view camera is installed in front of the center beam through a short cross beam to realize the driver's eye movement information tracking; During the test, the driver wearing the eyewear device can simultaneously achieve real-time perception of the traffic environment and synchronous tracking of the driver's eye movement information.

5. The driving safety monitoring method based on eye tracking and target perception technology according to claim 1, characterized in that: The step S3, inputting the eye movement data and the traffic environment data into the fusion model, specifically includes: The fusion model is a model based on the combination of Transformer and Long Short-Term Memory Network (LSTM) to process eye movement data and traffic environment data; The fusion model treats eye movement data as time series data and conducts in-depth analysis on it through the model, combining the information from target detection and instance segmentation, depth estimation and lane line detection models to determine whether the driver pays sufficient attention to all traffic objects with interaction risks.

6. The driving safety monitoring method based on eye tracking and target perception technology according to claim 1, characterized in that: The step S3, inputting the eye movement data and the traffic environment data into the fusion model, specifically includes: S31. Preprocess the eye movement data: the details are as follows: The eye movement data is collected through step S1; Then, the eye movement data is serialized in time order and passed into the fusion model as input data; Specifically, the eye movement data is regarded as time series data, and the eye movement data is segmented using a fixed time window length l to generate data samples, which can be specifically expressed as: X={X1,X2,···,X n },X n ∈R l×3 (5) Among them, l is the length of the time window and n is the number of samples. S32, feature encoding and input: The details are as follows: The eye movement data is encoded through a fully connected layer (FC) to generate a higher-dimensional feature vector, which is specifically expressed as: in, W is the weight matrix, b is the bias term, and m is the encoded feature dimension. S33, self-attention mechanism and residual connection: The details are as follows: The self-attention mechanism and residual connection step first further captures the global dependency between eye movement data and traffic environment data through the self-attention mechanism, which is specifically expressed as: Among them, Q is the query vector, K is the key vector, V is the numerical vector, and d k is the dimension of K; Multihead(Q,K,V)=Concat(head 1 ,......,head h )W o (8) Where h is the number of attention heads, The model can automatically assign weights to different eye movement data and traffic environment data through a self-attention mechanism, thereby enhancing attention to key traffic objects; Then the model is layer normalized, which is specifically expressed as: S34. Use the ReLU feed-forward network to activate feature information: The details are as follows: A f =ReLU(WO1+b) (11) S35, perform residual connection and normalization again: the details are as follows: THE j =LayerNorm(O f +O i ) (12) S36. Use LSTM network with attention mechanism to model the temporal dependency between eye movement data and traffic environment data: The LSTM module includes an input gate, a forget gate and an output gate; The LSTM module is updated by the new input data, which is specifically expressed as: i t =Sigmod(W i X t +A i h t-1 +b i ) (13) f t =Sigmod(W f X t +A f h t-1 +b f ) (14) o t =Sigmod(W o X t +A o h t-1 +b o ) (15) Among them, i t , f t , o t They are input gate, forget gate, output gate, and C t is the cell state, h t is the hidden state, tanh() is the hyperbolic tangent activation function, W, A, b are learnable weights and bias terms. S37. The weighted sum of each hidden state is as follows: The weights are: based on the hidden states generated by the LSTM module, the weights are assigned to each hidden state by introducing the attention mechanism, specifically: The output of the LSTM network is obtained by weighted summation, which is specifically expressed as: S38. Determination of driver's attention, as follows: The output F of the LSTM network is classified and mapped by the softmax function to determine whether the driver has noticed the traffic object with interaction risk, which is specifically expressed as follows: R = softmax(WF+b) (20) Where: R is the driver's traffic object perception prediction result, indicating whether the driver notices the traffic object with interaction risk.

Images