Driving skill level classification method based on multi-dimensional data fusion

Abstract

The invention discloses a driving skill level classification method based on multi-dimensional data fusion. The method comprises the following steps: (1) acquiring behaviors, physiological data and environmental data of a driver in real time through a driving simulator, eye movement tracking glasses and a physiological sensor; and (2) processing and analyzing the multi-dimensional data by using a CNN-LSTM classifier (CNN-LSTM-attention) so as to evaluate the skill level of the driver. According to the method, control skills and cognitive skills are comprehensively evaluated, key parameters including steering, braking, eye movement frequency, concentration duration and the like are captured in real time by using a driving simulator, eye movement tracking glasses and a physiological sensor, and a CNN-LSTM-attention classifier is constructed. According to the method, the complexity of the scene is quantified, the classifier is trained according to different scenes, the accuracy is improved, and meanwhile, the reliability of driving ability evaluation is enhanced by using multi-dimensional data.

Description

Technical Field

[0001] The present invention belongs to the field of driver safety in transportation, and more specifically, is a driving skill level classification method based on multi-dimensional data fusion. Background Art

[0002] Traditional driving skill assessments often use simple straight roads or segmented curves as the test environment. Although this method is easy to implement and control variables, it lacks simulation of complex traffic environments in the real world. This simplified scenario cannot fully reflect the various challenges that drivers may encounter in actual driving, such as traffic congestion, emergency handling, driving in different weather conditions, etc., thus limiting the external validity and practicality of the assessment results. In addition, drivers have different operating modes in scenarios of different complexity. If composite scenario detection is used directly, the accuracy is low. It is necessary to classify the complexity of the scenarios and detect them separately to improve accuracy.

[0003] Previous studies often rely on subjective questionnaires and basic driving parameters to evaluate skills. This method is easily affected by individual differences, social desirability effects, and other factors, which limits the objectivity and accuracy of the evaluation results. In addition, relying solely on basic driving parameters (such as speed and acceleration) ignores the diversity and complexity of driving behavior and cannot fully capture the driver's cognitive decision-making process and operating skills.

[0004] Previous studies have mostly used rule-based classification models or support vector machines (SVM) and other methods, which require manual feature extraction and information loss in the process of reducing the dimension of time series data. Deep learning combined with LSTM can effectively and automatically extract feature vectors while retaining hidden information in the time series.

[0005] Driving skill assessment is crucial for traffic safety and the design of personalized driver assistance systems. However, traditional driving skill assessment methods have many limitations. First, the test environment is usually oversimplified, using only simple straight roads or segmented curves. Although this is convenient for implementation and control of variables, it cannot simulate complex traffic conditions in the real world, such as traffic congestion, emergency handling, and driving in different weather conditions, thus limiting the external validity and practicality of the assessment results. Second, previous studies often rely on subjective questionnaires and basic driving parameters (such as speed and acceleration) to conduct skill assessment. This method is easily affected by individual differences and social expectation effects, resulting in limited objectivity and accuracy of the assessment results. In addition, methods such as rule-based classification models or support vector machines (SVMs) require manual feature extraction, and information is lost when reducing the dimensionality of time series data. In contrast, deep learning combined with LSTM can automatically extract features and retain hidden information in time series, improving assessment accuracy. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to overcome the defects of the prior art, build a driving skill level detection model, and provide a real-time, high-precision driving skill level classification method based on multidimensional data fusion.

[0007] The present invention uses a CNN-LSTM-attention classifier to comprehensively evaluate the driver's skills in complex environments. First, the driving risk is quantified and a complex test scenario is constructed through the potential field theory model and AHP. Then, the ECG signal is cleaned using a bandpass filter, the AOI is set to record the gaze behavior, and the skill level is evaluated through a questionnaire to provide training labels for the CNN-LSTM-attention model. Finally, the data of the four dimensions are integrated for classifier training, and the three classification results of the driver's driving skills are predicted to be low, medium, and high based on the weight parameters obtained from the training. This can improve the accuracy of the classification of driving skill levels.

[0008] Technical solution: The driving skill level classification method based on multidimensional data fusion of the present invention includes two stages: the first stage is to obtain driving behavior, perception and environmental data; the second stage is a driving skill level detection model, which consists of four core modules: complex scene quantification and classification module; data preprocessing and sample calibration module; multidimensional data fusion and classifier training module; classification model prediction module. The overall process is as follows: Figure 1 shown.

[0009] In the first stage, the present invention uses aSeeStudio eye tracker and PhysioLab physiological detection equipment, with every 20s as a time window to ensure the real-time detection, and obtains a total of 15 feature indicators in 4 dimensions. The results are as follows:

[0010] Cognitive factor dimensions: real-time heart rate, AOI type of gaze point, gaze point dispersion, and blinking frequency.

[0011] Control behavior dimension: steering wheel angle, brake pedal, accelerator pedal

[0012] Vehicle feedback dimensions: speed, acceleration, XY coordinates

[0013] Scenario condition dimensions: traffic flow, following distance, lane width, number of lanes, traffic sign location

[0014] Further,

[0015] Module (1) Quantification and construction of scene complexity: This module establishes a potential field theory model and combines it with the analytic hierarchy process (AHP) to quantitatively evaluate driving risk factors in complex traffic environments. By simulating different traffic conditions, such as the dynamic changes in work areas and fluctuations in traffic flow, it can better evaluate the performance of drivers in various environments. It extracts static and dynamic element features of scenes of different complexity and accurately identifies complex road sections. Finally, the complex scene is divided into sections with different complexities and tested separately to eliminate the impact of scene complexity on the accuracy of driver skill level detection.

[0016] Module (2) Data preprocessing and calibration module: In this module, a bandpass filter is applied to remove noise from the electrocardiogram (ECG) signal to achieve data cleaning and refinement. Three areas of interest (AOIs) are set to record the driver's gaze behavior to analyze the driver's attention characteristics. Finally, the sliding window method is used to generate time series data. At the same time, the driving skill level is evaluated through a questionnaire to provide training labels for the CNN-LSTM-attention model. The questionnaire content includes information such as driving experience, weekly driving frequency, typical driving time, and the number of historical traffic accidents. Through the questionnaire evaluation, the driver's skills can be calibrated into three categories: high, medium, and low.

[0017] Module (3) Multi-dimensional data fusion and classifier training module: This module constructs a CNN-LSTM-attention network structure. It integrates four dimensions of data: control behavior, vehicle feedback, cognitive factors, and environmental conditions. It uses CNN to extract features of each dimension, uses LSTM to process time series data, and uses the attention mechanism to capture global information, fuse multi-dimensional information, and classify skill levels into three categories. The use of multi-dimensional data not only enhances the reliability of driving ability assessment, but also improves classification accuracy.

[0018] Module (4) Integrated classification model prediction module: The recognition performance of the model is evaluated by a 10-fold cross validation method, and the classification accuracy is evaluated on a test set. The model is tested under different data dimensions and scene complexities to achieve stable prediction of skill levels. Compared with the prior art, the present invention effectively improves classification accuracy by designing complex scenes and integrating multidimensional data. This method can more comprehensively understand the performance of drivers in complex traffic environments and provide a scientific basis for the development of driver assistance systems and driver training.

[0019] Further,

[0020] The complex scene quantification model used in module (1) is a potential field theory model, which uses the driving perspective of the autonomous vehicle as a coordinate reference, calculates the complexity of static and dynamic elements, and weights them to obtain the complexity of the traffic environment. Figure 2 As shown in the figure, this method compares the vehicle to an individual in field theory, and abstracts external influencing factors such as road conditions, traffic facilities and meteorological conditions as positive point charges or uniformly charged wires to generate the electric potential field of the regional space. Different electric potential fields can be superimposed to form the final environmental electric potential field, which is similar to the complexity of the traffic environment. The specific expression formula is as follows.

[0021]

[0022] q i is the virtual charge of element i, reflecting the complexity of the element itself. ri is the distance between element i and its own cell. is the electric field strength generated by element i at the vehicle, reflecting the influence of the element on the driver.

[0023] Static element complexity:

[0024]

[0025] is the virtual charge of the ith static element determined by the static element category, is the distance from the viewpoint coordinates to the center of the i-th static element, n is the total number of static environment elements, and r0 represents the equivalent radius of the obstacle.

[0026] Dynamic element complexity:

[0027]

[0028] According to human driving habits and subjective feelings, when environmental elements are in motion, the influence of objects in front will be enhanced, and the influence of objects behind will be weakened, similar to the Doppler effect. Therefore, the proportional enhancement factor The calculation formula is: c is the propagation speed of the electric potential field (a constant that needs to be calibrated and its value should not be lower than the maximum speed of the vehicle), θ is The direction angle, v ix and v iy It is the velocity component of the dynamic element on the X-axis and Y-axis, and n is the number of lanes between the lane where the vehicle is located and the lane where the obstacle is located.

[0029] Different potential fields can be accumulated to form the final environmental potential field. The complexity of static elements and dynamic elements is weighted and summed to obtain the traffic environment complexity CE. The formula is:

[0030] C E =α·C J +β·C D

[0031] Where α and β are the weights of the complexity of static elements and the complexity of dynamic elements, and their values ​​are 0.35 and 0.65 respectively. E >80 is marked as a high-complexity section, C E <40 is a low complexity section, and the others are medium complexity. E The numerical value can accurately identify complex road sections. Based on the existing traffic video data, this paper divides the existing scenes into three complexity levels and reconstructs these scenes in the simulator for experiments. At the same time, based on the characteristics of the three types of complex road sections, multiple complex road sections are designed due to actual tests.

[0032] In module (2), we synchronized the psychological, eye tracking, vehicle dynamics, and environmental data. This included removing missing data or existing outliers in the dataset. To clean the noise in the electrocardiogram (ECG) signal, we applied a 0.5-35Hz bandpass filter. Subsequently, we applied the Savitzky-Golay smoothing technique to further refine the data. The R peak was obtained from the ECG signal using a classical QRS detection algorithm to obtain the driver's dynamic heart beat (RR) interval, as well as real-time heart rate data. RR intervals that were less than 0.3 seconds or greater than 2 seconds and differed from the mean of the previous three valid RR intervals by more than 25% were also considered outliers and replaced with the mean of the three RR intervals before and after.

[0033] For the eye tracking data, we set three areas of interest (AOI) to record the driver's gaze behavior: (i) instrument panel; (ii) road view including the side mirrors and the central part of the driving simulator screen; and (iii) traffic sign area (dynamic area, combined with video). The fixation duration of these areas was recorded for analysis. The monitoring ratio was obtained from the eye tracking data.

[0034]

[0035]

[0036] where t road , t tablet and t sign They are the time spent viewing the road AOI, instrument panel AOI, and traffic sign AOI. speed is the speed awareness monitoring rate, which reflects the frequency of the driver's observation of the instrument panel and reflects the driver's skill level in speed perception. sign is the traffic sign monitoring rate, which reflects that the driver can recognize the traffic sign in time. In order to reduce the differences between participants, all data were standardized to have a mean of zero and a standard deviation of one. The preprocessing process was performed using MATLAB.

[0037] We used a sliding window method to process the dataset. Specifically, each record data in each dimension is divided into a 20-second time window t win , and t step It represents the moving step of the window, which is set to 2s for complex sections and 10s for ordinary sections. step , we can achieve more sophisticated data enhancement in highly complex road sections, thereby improving the performance of the model in these key areas. This approach not only helps to improve the diversity and representativeness of the data, but also provides a solid foundation for subsequent behavior pattern analysis and prediction.

[0038] At the same time, the driving skill level is evaluated through a questionnaire to provide training labels for the CNN-LSTM-attention model. Based on the self-assessment of the driver's driving experience, weekly driving frequency, typical driving time, and number of historical traffic accidents, the AHP comprehensive evaluation questionnaire score is used to mark the driving skills into three categories: high, medium, and low.

[0039] In module (3), we integrate data from four dimensions - control behavior, vehicle dynamics, cognitive factors, and environmental conditions. The feature extraction process of the CNN-LSTM-attention model involves inputting the segmented time windows in the training dataset into the 1D-CNN to automatically learn features. The model consists of three 1D convolutional layers, three maximum pooling layers, and two layers of LSTM. The convolutional layer uses a sliding filter to extract effective features and adopts the ELU activation function to achieve faster convergence and stronger robustness. The maximum pooling layer reduces the data complexity by halving the data complexity, and a dropout layer is added after the pooling layer to prevent overfitting. After the convolution and pooling layers, the high-dimensional feature map is directly input into the LSTM network, which processes the time series data using a gating mechanism (forget gate, input gate, and output gate) to control the discarding or adding of information. These feature maps are then converted into hidden states through the LSTM network.

[0040] In the fusion step, we combine the hidden states of physiological data, eye data, vehicle dynamics data, and environmental data to create a new feature map. These hidden states are aggregated into a vector representation H = (h1, h2, ..., h n ). The attention layer calculates the alignment coefficient α for each hidden state t , and then use these coefficients to weight the sum of these states h t , forming the final vector representation s. This vector representation can be input into the SoftMax layer for pressure level classification.

[0041] u t =tanh(Wh t +B)

[0042]

[0043] The self-attention mechanism involves a fully connected layer with a tanh activation function followed by a SoftMax normalization to obtain the alignment coefficient. The dimension of the context information parameter vector (u) is an important hyperparameter, and studies have shown that increasing its dimension can improve the accuracy and false positive rate of the model. After extensive experimental evaluation, the optimal dimension of u is set to 64. During the training process, the weight matrix W, the bias vector b, and the parameter vector u are randomly initialized. The choice of these initial values ​​is crucial for the convergence of the training process and the overall performance of the model.

[0044] Based on the above model, the subjects were classified into three levels of skill level. Compared with the traditional algorithm, the manual feature extraction step is omitted, which improves efficiency and accuracy accordingly.

[0045] The implementation details of module (4) of the present invention are as follows: the recognition performance of the model is evaluated by a 10-fold cross validation method, and the classification accuracy is evaluated on a test set. The model is tested under different data dimensions and scene complexities to achieve stable prediction of skill levels. Compared with the prior art, the present invention effectively improves classification accuracy by effectively identifying complex scenes and fusing multidimensional data. This method can more comprehensively understand the performance of drivers in complex traffic environments and provide a scientific basis for the development of driver assistance systems and driver training.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] 1. Multi-dimensional index extraction: The present invention obtains multi-dimensional indicators based on simulators, eye tracking, and physiological detection equipment, and uses a 20-second time window as a sample point to detect the driving skill level through different road sections within the same task.

[0048] 2. Quantification and classification of scene complexity: This invention quantifies and classifies the complexity of driving scenes through potential field theory, effectively solving the impact of scene complexity on the accuracy of skill level detection. The vehicle is compared to an individual in field theory, and external influencing factors such as road conditions, traffic facilities and meteorological conditions are abstracted as positive point charges or uniformly charged wires to generate an electric potential field in the regional space to simulate scene complexity for subsequent detection.

[0049] 3. Efficient deep learning strategy: Combining the advantages of CNN, LSTM, and attention mechanisms, the present invention constructs a powerful deep learning model. This model can effectively integrate multi-dimensional data, which not only improves the prediction accuracy, generalization, and robustness, but also enables the model to capture the complex patterns and associations in the driver's multi-dimensional data, thereby significantly improving the efficiency and effectiveness of driving skill level detection.

[0050] 4. Feature screening and data enhancement: Based on the perception-decision-control driving process, four dimensions, including control behavior, vehicle dynamics, cognitive factors and environmental conditions, and 15 feature indicators were selected for analysis. Since data samples on complex roads are relatively rare, dynamic time windows are used to enhance the training data, which effectively overcomes the limitations of small sample data sets and improves the stability and accuracy of the model in practical applications.

[0051] 5. Real-time, high-precision skill level detection: Finally, the present invention realizes real-time, high-precision detection of driving skill level through the CNN-LSTM-attention model. This strategy not only improves the accuracy and stability of prediction, but also enhances the adaptability and flexibility of the model. It can dynamically select the most appropriate prediction model according to the specific characteristics of the input data and the complexity of the scene, ensuring that efficient prediction performance can be maintained under different scene complexities and data dimensions. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0053] Figure 2 It is a schematic diagram of quantifying scene complexity of the present invention;

[0054] Figure 3 It is a structural schematic diagram of the CNN-LSTM-attention prediction model of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further described below.

[0056] The driving skill level classification method based on multidimensional data fusion of this embodiment includes two stages: the first stage is to obtain driving behavior, perception and environmental data; the second stage is to detect the driving skill level. The model consists of four core modules: complex scene quantification and classification module; data preprocessing and sample calibration module; multidimensional data fusion and classifier training module; classification model prediction module. The overall process is as follows: Figure 1 shown.

[0057] In the first stage, the present invention uses aSeeStudio eye tracker and PhysioLab physiological detection equipment, with every 20s as a time window to ensure the real-time detection, and obtains a total of 15 feature indicators in 4 dimensions. The results are as follows:

[0058] Cognitive factor dimensions: real-time heart rate, AOI type of gaze point, gaze point dispersion, and blinking frequency.

[0059] Control behavior dimension: steering wheel angle, brake pedal, accelerator pedal

[0060] Vehicle feedback dimensions: speed, acceleration, XY coordinates

[0061] Scenario condition dimensions: traffic flow, following distance, lane width, number of lanes, traffic sign location

[0062] In the stage of establishing the driving skill level detection model, it is mainly divided into the following four modules:

[0063] Module (1) Quantification and construction of scene complexity: This module establishes a potential field theory model and combines it with the analytic hierarchy process (AHP) to quantitatively evaluate driving risk factors in complex traffic environments. By simulating different traffic conditions, such as the dynamic changes in work areas and fluctuations in traffic flow, it can better evaluate the performance of drivers in various environments. It extracts static and dynamic element features of scenes of different complexity and accurately identifies complex road sections. Finally, the complex scene is divided into sections with different complexities and tested separately to eliminate the impact of scene complexity on the accuracy of driver skill level detection.

[0064] Module (2) Data preprocessing and calibration module: In this module, a bandpass filter is applied to remove noise from the electrocardiogram (ECG) signal to achieve data cleaning and refinement. Three areas of interest (AOIs) are set to record the driver's gaze behavior to analyze the driver's attention characteristics. Finally, the sliding window method is used to generate time series data. At the same time, the driving skill level is evaluated through a questionnaire to provide training labels for the CNN-LSTM-attention model. The questionnaire content includes information such as driving experience, weekly driving frequency, typical driving time, and the number of historical traffic accidents. Through the questionnaire evaluation, the driver's skills can be calibrated into three categories: high, medium, and low.

[0065] Module (3) Data preprocessing and calibration module: This module integrates data from four dimensions: control behavior, vehicle feedback, cognitive factors, and environmental conditions. CNN is used to extract features from each dimension, LSTM is used to process time series data, and an attention mechanism is used to capture global information and fuse multi-dimensional information to classify skill levels into three categories. The use of multi-dimensional data not only enhances the reliability of driving ability assessment, but also improves classification accuracy.

[0066] Module (4) Integrated classification model prediction module: The recognition performance of the model is evaluated by a 10-fold cross validation method, and the classification accuracy is evaluated on a test set. The model is tested under different data dimensions and scene complexities to achieve stable prediction of skill levels. Compared with the prior art, the present invention effectively improves classification accuracy by designing complex scenes and integrating multidimensional data. This method can more comprehensively understand the performance of drivers in complex traffic environments and provide a scientific basis for the development of driver assistance systems and driver training.

[0067] Further,

[0068] The complex scene quantification model used in module (1) is a potential field theory model, which uses the driving perspective of the autonomous vehicle as a coordinate reference, calculates the complexity of static and dynamic elements, and weights them to obtain the complexity of the traffic environment. Figure 2 As shown in the figure, this method compares the vehicle to an individual in field theory, and abstracts external influencing factors such as road conditions, traffic facilities and meteorological conditions as positive point charges or uniformly charged wires to generate the electric potential field of the regional space. Different electric potential fields can be superimposed to form the final environmental electric potential field, which is similar to the complexity of the traffic environment. The specific expression formula is as follows.

[0069]

[0070] q i is the virtual charge of element i, reflecting the complexity of the element itself. ri is the distance between element i and its own cell. is the electric field strength generated by element i at the vehicle, reflecting the influence of the element on the driver.

[0071] Static element complexity:

[0072]

[0073] is the virtual charge of the ith static element determined by the static element category, is the distance from the viewpoint coordinates to the center of the i-th static element, n is the total number of static environment elements, and r0 represents the equivalent radius of the obstacle.

[0074] Dynamic element complexity:

[0075]

[0076] According to human driving habits and subjective feelings, when environmental elements are in motion, the influence of objects in front will be enhanced, and the influence of objects behind will be weakened, similar to the Doppler effect. Therefore, the proportional enhancement factor The calculation formula is: c is the propagation speed of the electric potential field (a constant that needs to be calibrated and its value should not be lower than the maximum speed of the vehicle), θ is The direction angle, v ix and v iy It is the velocity component of the dynamic element on the X-axis and Y-axis, and n is the number of lanes between the lane where the vehicle is located and the lane where the obstacle is located.

[0077] Different potential fields can be accumulated to form the final environmental potential field. The complexity of static elements and dynamic elements is weighted and summed to obtain the traffic environment complexity CE. The formula is:

[0078] C E =α·C J +β·C D

[0079] Where α and β are the weights of the complexity of static elements and the complexity of dynamic elements, and their values ​​are 0.35 and 0.65 respectively. E >80 is marked as a high-complexity section, C E <40 is a low complexity section, and the others are medium complexity. E The numerical value can accurately identify complex road sections. Based on the existing traffic video data, this paper divides the existing scenes into three complexity levels and reconstructs these scenes in the simulator for experiments. At the same time, based on the characteristics of the three types of complex road sections, multiple complex road sections are designed due to actual tests.

[0080] In module (2), we synchronized the psychological, eye tracking, vehicle dynamics, and environmental data. This included removing missing data or existing outliers in the dataset. To clean the noise in the electrocardiogram (ECG) signal, we applied a 0.5-35Hz bandpass filter. Subsequently, we applied the Savitzky-Golay smoothing technique to further refine the data. The R peak was obtained from the ECG signal using a classical QRS detection algorithm to obtain the driver's dynamic heart beat (RR) interval, as well as real-time heart rate data. RR intervals that were less than 0.3 seconds or greater than 2 seconds and differed from the mean of the previous three valid RR intervals by more than 25% were also considered outliers and replaced with the mean of the three RR intervals before and after.

[0081] For the eye tracking data, we set three areas of interest (AOI) to record the driver's gaze behavior: (i) instrument panel; (ii) road view including the side mirrors and the central part of the driving simulator screen; and (iii) traffic sign area (dynamic area, combined with video). The fixation duration of these areas was recorded for analysis. The monitoring ratio was obtained from the eye tracking data.

[0082]

[0083] where t road , t tablet and t sign They are the time spent viewing the road AOI, instrument panel AOI, and traffic sign AOI. speed is the speed awareness monitoring rate, which reflects the frequency of the driver's observation of the instrument panel and reflects the driver's skill level in speed perception. sign is the traffic sign monitoring rate, which reflects that the driver can recognize the traffic sign in time. In order to reduce the differences between participants, all data were standardized to have a mean of zero and a standard deviation of one. The preprocessing process was performed using MATLAB.

[0084] We used a sliding window method to process the dataset. Specifically, each record data in each dimension is divided into a 20-second time window t win , and t step It represents the moving step of the window, which is set to 2s for complex sections and 10s for ordinary sections. step , we can achieve more sophisticated data enhancement in highly complex road sections, thereby improving the performance of the model in these key areas. This approach not only helps to improve the diversity and representativeness of the data, but also provides a solid foundation for subsequent behavior pattern analysis and prediction.

[0085] At the same time, the driving skill level is evaluated through a questionnaire to provide training labels for the CNN-LSTM-attention model. Based on the self-assessment of the driver's driving experience, weekly driving frequency, typical driving time, and number of historical traffic accidents, the AHP comprehensive evaluation questionnaire score is used to mark the driving skills into three categories: high, medium, and low.

[0086] In module (3), we integrate data from four dimensions - control behavior, vehicle dynamics, cognitive factors, and environmental conditions. The feature extraction process of the CNN-LSTM-attention model involves inputting the segmented time windows in the training dataset into the 1D-CNN to automatically learn features. The model consists of three 1D convolutional layers, three maximum pooling layers, and two layers of LSTM. The convolutional layer uses a sliding filter to extract effective features and adopts the ELU activation function to achieve faster convergence and stronger robustness. The maximum pooling layer reduces the data complexity by halving the data complexity, and a dropout layer is added after the pooling layer to prevent overfitting. After the convolution and pooling layers, the high-dimensional feature map is directly input into the LSTM network, which processes the time series data using a gating mechanism (forget gate, input gate, and output gate) to control the discarding or adding of information. These feature maps are then converted into hidden states through the LSTM network.

[0087] In the fusion step, we combine the hidden states of physiological data, eye data, vehicle dynamics data, and environmental data to create a new feature map. These hidden states are aggregated into a vector representation H = (h1, h2, ..., h n ). The attention layer calculates the alignment coefficient α for each hidden state t , and then use these coefficients to weight the sum of these states h t , forming the final vector representation s. This vector representation can be input into the SoftMax layer for pressure level classification.

[0088] u t =tanh(Wh t +B)

[0089]

[0090] The self-attention mechanism involves a fully connected layer with a tanh activation function followed by a SoftMax normalization to obtain the alignment coefficient. The dimension of the context information parameter vector (u) is an important hyperparameter, and studies have shown that increasing its dimension can improve the accuracy and false positive rate of the model. After extensive experimental evaluation, the optimal dimension of u is set to 64. During the training process, the weight matrix W, the bias vector b, and the parameter vector u are randomly initialized. The choice of these initial values ​​is crucial for the convergence of the training process and the overall performance of the model.

[0091] Based on the above model, the subjects were classified into three levels of skill level. Compared with the traditional algorithm, the manual feature extraction step is omitted, which improves efficiency and accuracy accordingly.

[0092] The implementation details of module (4) of the present invention are as follows: the recognition performance of the model is evaluated by a 10-fold cross validation method, and the classification accuracy is evaluated on a test set. The model is tested under different data dimensions and scene complexities to achieve stable prediction of skill levels. Compared with the prior art, the present invention effectively improves classification accuracy by effectively identifying complex scenes and fusing multidimensional data. This method can more comprehensively understand the performance of drivers in complex traffic environments and provide a scientific basis for the development of driver assistance systems and driver training.

[0093] The specific real-time method is as follows:

[0094] Step 1. Prepare the equipment needed for the experiment: a fixed driving simulator that can collect data on vehicle dynamics (speed, acceleration, and XY coordinates) and driver behavior (steering wheel angle, throttle, brake). Eye tracking glasses that can collect data on pupil diameter, dispersion of gaze points on the X and Y axes, and blink frequency. A physiological analysis system for recording ECG (electrocardiogram). Also prepare synchronization software to synchronize the above-mentioned data to a timeline.

[0095] Step 2: Based on a large number of real-life driving videos and drone video data, different driving scenarios are extracted. Using the potential field theory, the static elements (traffic signs, plants, markings, and ancillary facilities) and dynamic elements (vehicles and pedestrians) in the scenarios are analyzed to calculate the complexity C of all real-life scenarios. E . E >80 is marked as a high-complexity section, C E <40 is a low complexity section, and the others are medium complexity. Five sections of each complexity are selected, a total of 15 sections, and these scenes are reconstructed in the simulator for experiments. At the same time, according to the characteristics of the three complexity sections, multiple complexity sections are designed for actual testing.

[0096] Step 3: Recruit experimental personnel, and divide them into three driving skill levels: low, medium, and high, based on the questionnaire answers filled out by the applicants (driving experience, accident rate, driving frequency, etc.), using the AHP comprehensive questionnaire answers. The results are used as labels for subsequent training models.

[0097] Step 4. After completing the above steps, the simulator experiment can be carried out. Before the formal experiment begins, participants are asked to drive a simple route to familiarize themselves with the simulator environment and vehicle controls. Each participant is then instructed to rest for 2-3 minutes to establish a baseline for physiological measurements. Afterwards, participants will drive through all driving scenarios of varying complexity. In these scenarios, data is continuously collected, including ECG signals, eye tracking data, vehicle handling, vehicle dynamics, and environmental measurements.

[0098] Step 5. Preprocess the obtained data. For ECG, apply a 0.5-35Hz bandpass filter and Savitzky-Golay smoothing technique to the eye movement data to remove noise, and use the QRS detection algorithm to extract the R wave peak, and finally calculate the real-time heart rate data. For eye movement data, it is necessary to combine the video gaze and the gaze point coordinates recorded by the eye tracker to calculate the AOI type to which the gaze point belongs and the gaze time of each type of AOI in real time. Outliers are removed from the driving simulator data. The scene file is converted into environmental condition dimension data on the timeline. The final four dimensions contain 15 features. The data of the above four dimensions are divided into 20-second time windows t using the sliding window method. win The moving step size of the window for high-complexity sections is 2s, and that for other sections is set to 10s to avoid too little data in high-complexity sections and avoid overfitting.

[0099] Step 6: Input the obtained four-dimensional time window data into the CNN-LSTM-attention network for training, evaluate the recognition performance of the model by 10-fold cross-validation, and evaluate the classification accuracy on the test set. Test this model under different data dimensions and scene complexity to achieve stable prediction of skill level.

[0100] Step 7: Apply the model to new scenarios of different complexity designed in step 2, and conduct driving skill level tests on new subjects. The results can be used for the development of driving assistance systems and driving training.

[0101] The present invention quantitatively classifies driving scenarios of high, medium and low complexity, and trains a driving skill level detection model in scenarios of different complexity. Participants with different driving skill levels were recruited for simulator experiments. The experiment collected data from four dimensions: cognitive factors, control behavior, vehicle feedback, and scenario conditions. After preprocessing, these data were input into the CNN-LSTM-attention network for training and evaluation. Finally, the model achieved stable prediction of driving skill levels in scenarios of different complexity, and can be applied to the testing of new scenarios, providing support for the development and training of driver assistance systems.

[0102] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A driving skill level classification method based on multidimensional data fusion, characterized in that: The following steps are involved: (1) Obtain the driver's behavior, physiological data, and environmental data in real time through driving simulators, eye tracking glasses, and physiological sensors; (2) Use the CNN-LSTM-attention classifier to process and analyze multidimensional data to evaluate the driver's skill level.

2. The driving skill level classification method based on multidimensional data fusion according to claim 1 is characterized in that: The CNN-LSTM-attention classifier includes a scene complexity quantification and construction module, a data preprocessing and calibration module, a multi-dimensional classifier training module and an integrated classification model prediction module.

3. The driving skill level classification method based on multidimensional data fusion according to claim 2 is characterized in that: The scenario complexity quantification and construction module quantitatively evaluates driving risk factors in complex traffic environments by establishing a potential field theory model and combining it with the hierarchical analysis method. The module extracts static and dynamic element features of scenarios of different complexity, comprehensively evaluates the complexity of the scenarios, and classifies them.

4. The driving skill level classification method based on multidimensional data fusion according to claim 3 is characterized in that: This method compares the vehicle to an individual in field theory, and abstracts the external influencing factors as positive point charges or uniformly charged wires to generate the electric potential field of the regional space. Different electric potential fields can be superimposed to form the final environmental electric potential field, which is similar to the complexity of the traffic environment. The formula is as follows: q i is the virtual charge of element i, reflecting the complexity of the element itself, ri is the distance between element i and its own workshop, is the electric field strength generated by element i at the ego-vehicle, ε0 is the electric constant of vacuum, which has no practical significance in the complexity quantification and can be taken as 1. Static element complexity: is the virtual charge of the ith static element determined by the static element category, is the distance from the viewpoint coordinates to the center of the i-th static element, n is the total number of static environment elements, and r0 represents the equivalent radius of the obstacle. Dynamic element complexity: is the virtual power of the i-th dynamic element determined by the dynamic element category. According to human driving habits and subjective feelings, when the environmental elements are in motion, the influence of the front object will be enhanced, and the influence of the rear object will be weakened, similar to the Doppler effect. Therefore, the proportional enhancement factor The calculation formula is: c is the propagation speed of the electric potential field, θ is The direction angle, v ix and v iy is the velocity component of the dynamic element on the X-axis and Y-axis, n is the number of lanes between the lane where the vehicle is located and the lane where the obstacle is located, Different potential fields can be accumulated to form the final environmental potential field. The complexity of static elements and dynamic elements is weighted and summed to obtain the traffic environment complexity C E , the formula is: C E =α CJ+β CD Among them, α and β are the weights of static element complexity and dynamic element complexity, and their values ​​are 0.35 and 0.65 respectively. E >80 is marked as a high-complexity section, C E <40 is a low complexity section, and the others are medium complexity. E Numerical value, identifying complex road sections.

5. The driving skill level classification method based on multidimensional data fusion according to claim 2 is characterized in that: The data preprocessing and calibration module uses a bandpass filter to remove noise from the electrocardiogram signal, sets multiple focus areas to record the driver's gaze behavior, and analyzes his attention characteristics. At the same time, the driving skill level is evaluated through a questionnaire to provide training labels for the CNN-LSTM-attention classifier.

6. The driving skill level classification method based on multidimensional data fusion according to claim 5 is characterized in that: Clean the noise from the ECG signal, apply a bandpass filter of 0.5-35 Hz, and apply the Savitzky-Golay smoothing technique to further refine the data; For eye tracking data, the monitoring ratio is obtained from the eye tracking data. where t road , t tablet and t sign are the viewing time of the road AOI, instrument panel AOI and traffic sign AOI, respectively. speed is the speed awareness monitoring rate, r sign For the traffic sign monitoring rate, in order to reduce the differences between participants, all data were standardized to have a mean of zero and a standard deviation of one. The preprocessing process was performed using MATLAB.

7. The driving skill level classification method based on multidimensional data fusion according to claim 2 is characterized in that: The multidimensional classifier training module integrates data from four dimensions: control behavior, vehicle dynamics, cognitive factors, and environmental conditions. It uses CNN to extract features from each dimension, processes time series data through LSTM, and combines the attention mechanism to capture global information, thereby achieving the fusion of multidimensional information and dividing skill levels into multiple categories.

8. The driving skill level classification method based on multidimensional data fusion according to claim 7 is characterized in that: The model consists of three 1D convolutional layers, three maximum pooling layers, and two LSTM layers. The convolutional layer uses a sliding filter to extract effective features and adopts the ELU activation function to achieve faster convergence and stronger robustness. The maximum pooling layer reduces the data complexity by halving the data complexity, and a dropout layer is added after the pooling layer to prevent overfitting. After the convolution and pooling layers, the high-dimensional feature maps are directly input into the LSTM network, which processes time series data using a gating mechanism to control the discarding or adding of information. Then, these feature maps are converted into hidden states through the LSTM network.

9. The driving skill level classification method based on multidimensional data fusion according to claim 7 is characterized in that: In the fusion step, the hidden states of physiological data, eye data, vehicle dynamic data, and environmental data are integrated to create a new feature map. These hidden states are aggregated into a vector representation H = (h1, h2, ..., h n ), the attention layer calculates the alignment coefficient α for each hidden state t , and then use these coefficients to weight the sum of these states h t , forming the final vector representation s, which can be input into the SoftMax layer for pressure level classification. u t =tanh(Wh t +B) The self-attention mechanism involves a fully connected layer with a tanh activation function, followed by a SoftMax normalization to obtain the alignment coefficient. The dimension of the information parameter vector (u) is set to 64. During the training process, the weight matrix W, bias vector b, and parameter vector u are randomly initialized.

10. The driving skill level classification method based on multidimensional data fusion according to claim 2 is characterized in that: The ensemble classification model prediction module uses a 10-fold cross-validation method to evaluate the recognition performance of the model, evaluates the classification accuracy on the test set, and tests the model under different data dimensions and scenario complexities to achieve stable prediction of skill levels.

Images