CNN-GRU-Attention-based driving behavior prediction method

Abstract

The invention relates to a CNN-GRU-Attention-based driving behavior prediction method, and belongs to the technical field of intelligent driving. The method comprises the following steps: acquiring a multi-modal physiological signal of a driver and a vehicle driving parameter in a driving process, and performing preprocessing; feature extraction and feature screening are carried out on the preprocessed physiological signals and the vehicle driving parameters, data calibration is carried out according to driving behaviors generated in the driving process, and then data samples are divided into a training set, a test set and a verification set; inputting a training set sample into the driving behavior prediction model, taking the multi-modal physiological signals and vehicle driving parameters as input, taking a driving behavior prediction result as output, continuously adjusting a network structure and parameters, training the driving behavior prediction model according to training loss and prediction precision, and visualizing a confusion matrix. After training is completed, verification set sample data are input into the model for driving behavior prediction. The driving behavior prediction accuracy is higher.

Description

Technical Field

[0001] The invention relates to a driving behavior prediction method based on CNN-GRU-Attention, belonging to the technical field of intelligent driving. Background Art

[0002] In the human-vehicle co-driving mode, the complex interaction between the human and the vehicle is still constrained by the driving environment and other conditions. Especially when facing complex road conditions, there are complex interactions between the driver, the vehicle and the driving environment. Therefore, building a model that can comprehensively analyze the information of human-vehicle-road conditions is of positive significance for achieving accurate prediction of driving behavior.

[0003] Existing studies often only select driver physiological signals or vehicle operating parameters as input parameters and predict driving behavior based on data-driven methods. However, physiological signals are time-series and suitable for predicting future trends, but are susceptible to sensor noise interference; and in practical applications, there is a lack of systematic and global methods or ideas for people-vehicle-road conditions to build a complete model to describe driving behavior, so as to identify or predict driver driving behavior, and then realize the early warning of dangerous driving behavior and improve driving safety. Summary of the invention

[0004] In view of the above technical requirements and problems, the present invention aims to provide a human-vehicle-road integrated model based on CNN-GRU-Attention (convolutional neural network-gated recurrent unit-attention mechanism), which can collect the driver's physiological data and vehicle driving parameters in real time during driving and accurately predict driving behavior. The collected multimodal physiological signals are filtered, denoised, interpolated, and normalized, thereby effectively improving the accuracy of the driving behavior prediction model. On this basis, short-time Fourier transform (STFT) and ErgoLAB cloud platform are used to extract EEG (power spectral density), eye movement (annotation order), pulse wave signal and electromyographic signal (average power frequency), and the vehicle driving parameters are feature screened based on Pearson correlation analysis, and four vehicle driving parameters of steering wheel angle, steering wheel angular velocity, accelerator pedal opening, and brake pedal opening are screened as driving behavior feature parameters. Finally, based on CNN-GRU-Attention, the driver's driving behavior feature signal is fused, and a human-vehicle-road condition integrated relationship model is established to achieve accurate prediction of driving behavior.

[0005] To achieve the above objectives, a driving behavior prediction method based on CNN-GRU-Attention is proposed. The prediction method mainly includes the following steps:

[0006] S1: Collect the driver's multimodal physiological signals (electroencephalogram signals, pulse wave signals, eye movement signals, electromyography signals) and vehicle driving parameters during driving, and perform preprocessing;

[0007] S2: Extract and screen the features of the preprocessed physiological signals and vehicle driving parameters, calibrate the data according to the driving behavior generated during driving, and then divide the data samples into training set, test set, and validation set;

[0008] S3: Input the training set samples into the constructed CNN-GRU-Attention model, with multimodal physiological signals and vehicle driving parameters as input and driving behavior prediction results as output. According to the training loss and prediction accuracy, the network structure and parameters are continuously adjusted to train the driving behavior prediction model and visualize the confusion matrix. After the training is completed, the test set sample data is input into the model for driving behavior prediction.

[0009] Furthermore, the EEG signals, pulse wave signals, eye movement signals, electromyographic signals and vehicle driving parameters were preprocessed, mainly including bandpass filtering, baseline correction and time domain / frequency domain threshold method to deal with the interference of eye movement artifacts and motion artifacts in EEG; linear interpolation and sliding mean filtering were used to deal with the missing and noise of eye movement signals; notch filtering and bandpass filtering methods were selected to eliminate interference and improve the quality of pulse wave signals; low-pass filtering was used to process the original electromyographic signals to remove power supply interference and high-frequency noise. Finally, the data samples were normalized.

[0010] Furthermore, the Hanning window function was used for signal truncation and the power spectral density of four frequency bands in the driver's EEG signal was extracted based on the short-time Fourier transform method; the standard deviation of all cardiac intervals (Standard Deviation of Normal to Normal intervals, SDNN) was selected as the time domain characteristic index of the pulse wave to evaluate the overall change of the pulse wave; the average power frequency was used to extract the features of the collected electromyographic signals; a method based on the position threshold algorithm was used to extract the gaze point in the eye movement signal; the Pearson correlation analysis was used to perform feature screening on the collected vehicle driving parameters, and four vehicle driving parameters including steering wheel angle, steering wheel angular velocity, accelerator pedal opening and brake pedal opening were selected as driving behavior characteristic parameters.

[0011] Furthermore, a human-vehicle-road integrated model based on CNN-GRU-Attention is constructed to effectively predict driving behavior. Among them, the CNN network effectively captures the spatial features in the driving behavior data through convolution operations, including 1 convolution layer with an input size of 64 and a convolution kernel size of 3, 1 batch normalization layer, 1 pooling layer with a window size of 2, 1 convolution layer with an input size of 128 and a convolution kernel size of 3, 1 batch normalization layer, and 1 pooling layer with a window size of 2. The convolution layer performs convolution operations on the input samples, and its expression is:

[0012]

[0013] Where: Represents the i-th feature map of the l-1th layer; represents the weight matrix; Represents the bias corresponding to the convolution kernel; represents the output of the jth feature map of the lth layer; M j represents the jth convolution area of ​​the l-1th layer; f() represents the activation function, and the activation function selects the Relu function.

[0014] The GRU network captures and memorizes long-term dependencies in time series, extracts temporal features, and learns the temporal dynamic characteristics of driving behavior. It contains two GRU layers (50 units), sets a Dropout mechanism in the GRU module, and introduces an attention mechanism between the GRU layer and the output layer, assigning different weight parameters to the input feature vectors to highlight the feature vectors that are valuable for key tasks, while reducing attention to task-irrelevant features, thereby adjusting the importance ratio of features and optimizing the network model. The attention mechanism calculation formula is:

[0015]

[0016] Where: W s is the input vector, h k is the hidden state vector of the input, a i-k is the normalized score of the attention weight, a k is the attention weight, b s is the bias, and tanh is the activation function for calculating the attention weight matrix.

[0017] Furthermore, the training set samples are input into the driving behavior prediction model, and the Adam method is used as the optimizer algorithm to iteratively train the model, calculate the LOSS value and accuracy, and save the optimal hyperparameters.

[0018] The validation set samples are input into the CNN-GRU-Attention driving behavior prediction model for verification, and the driving behavior prediction results are output through the Softmax function.

[0019] In general, compared with the prior art solutions, the driving behavior prediction method based on CNN-GRU-Attention provided by the present invention has the following beneficial effects:

[0020] 1. The real-time collected multimodal physiological signals (electroencephalogram signals, eye movement signals, pulse wave signals, electromyography signals) and vehicle driving parameters are input into the driving behavior prediction model. Compared with relying solely on physiological signals or vehicle driving parameters, multimodal data fusion can make full use of the interactive characteristics between various data, and the driving behavior prediction accuracy is higher.

[0021] 2. After combining the CNN structure with the GRU structure, the model can not only extract the spatial characteristics of driving behavior, but also effectively capture the global dependencies in the time series, significantly improving the accuracy of driving behavior prediction. After introducing the Attention mechanism, the weights of different time steps can be dynamically assigned to highlight the characteristics of key time steps, thereby improving the model's ability to recognize important features. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a driving behavior prediction model based on CNN-GRU-Attention is provided in an embodiment of the present invention.

[0023] Figure 2 Pearson correlation coefficient absolute value heat map

[0024] Figure 3 This is the CNN-GRU network structure diagram.

[0025] Figure 4 It is the LOSS value convergence curve of the driving behavior prediction model.

[0026] Figure 5 Confusion matrix for the driving behavior prediction model. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and implementation cases, but it should be understood that the embodiments are used to explain the present invention and are not used to limit the present invention.

[0028] The invention provides a driving behavior prediction method based on CNN-GRU-Attention. The provided prediction method is completed in the following steps: first, based on multi-sensor collection of multi-modal physiological signals (electroencephalogram signals, eye movement signals, electromyographic signals, pulse wave signals) of the driver during driving and vehicle driving parameters, the signals are filtered, denoised, interpolated and normalized; second, feature extraction is performed on the multi-modal physiological signals to extract effective features that can characterize the driving behavior, and feature screening is performed on the vehicle driving parameters based on the Pearson correlation analysis method; third, data calibration is performed on the data set according to the driving behavior actually generated during the driving process, and the training set, test set and validation set are divided into a training set, a test set and a validation set in a ratio of 8:1:1; finally, a driving behavior prediction model based on CNN-GRU-Attention is constructed, the training set samples are input for iterative testing, the network parameter settings are continuously adjusted according to the LOSS value and the prediction accuracy in the training process, and the validation set samples are input after the model training is completed to verify the model validity, so as to obtain the driving behavior prediction result.

[0029] The driving behavior prediction method based on CNN-GRU-Attention mainly includes the following steps:

[0030] Step 1: Data collection, preprocessing

[0031] Collect multimodal physiological signals (electroencephalogram signals, pulse wave signals, eye movement signals, electromyography signals) and vehicle driving parameters during simulated driving and perform preprocessing.

[0032] Step 1 includes the following sub-steps:

[0033] (11) Processing the interference of eye movement artifacts and motion artifacts in EEG based on bandpass filtering (filter range 0.1-40HZ), baseline correction and time domain threshold method;

[0034] (12) Linear interpolation and sliding mean filtering are used to deal with missing eye movement signals and noise;

[0035] (13) Select notch filtering (eliminating power supply interference frequency with a specific frequency of 50 Hz) and bandpass filtering method (filter range 0.5-5 Hz) to eliminate interference and improve the quality of pulse wave signals;

[0036] (14) Low-pass filtering (cut-off frequency is set to 0.2 Hz to remove power supply interference and high-frequency noise) is used to process the original electromyographic signal to remove power supply interference and high-frequency noise;

[0037] (15) Perform maximum-minimum normalization on the data and normalize the data to the range of [0,1].

[0038] Step 2: Sample data feature extraction, feature screening, and data calibration

[0039] The preprocessed multimodal physiological signals are feature extracted and the vehicle driving parameters are feature screened based on Pearson correlation analysis to screen the features that can characterize driving behavior. Finally, the data set is labeled and divided into training set, test set, and validation set.

[0040] Step 2 includes the following sub-steps:

[0041] (21) The Hanning window function is used for signal truncation. The role of the window function in the short-time Fourier transform is mainly to process the windowing of the signal in the time domain so as to perform Fourier transform in the frequency domain. It can perform signal truncation, balance the time and frequency resolution, and improve the accuracy of spectrum analysis. On this basis, the power spectral density of the four frequency bands in the driver's EEG signal is extracted based on STFT. The STFT expression is:

[0042]

[0043] Where S(t,f) is the short-time Fourier output result, s(t) is the time domain signal, and w(t) is the window function with a fixed length.

[0044] (22) SDNN is selected as the time domain characteristic index of the pulse wave to evaluate the overall change of the pulse wave. SDNN is used to evaluate the overall HRV size and reflect the slow change of heart rate. It is a sensitive indicator for evaluating sympathetic nerve function. Its calculation formula is:

[0045]

[0046] Where n represents the total number of normal heart beats, rr i represents the length of the i-th adjacent sinus RR interval, Represents the average value of n RR intervals.

[0047] (23) The average power frequency is used to extract features from the collected electromyographic signals. The power spectral density of the signal is integrated or weighted to calculate the frequency of the average power and provide information about the frequency distribution of the signal. Compared with certain spectral features (such as peak frequency), the average power frequency is usually more numerically stable in the presence of noise or fluctuations in the signal. Its calculation formula is as follows:

[0048]

[0049] Where P(f) is the power spectral density function of the signal and f is the frequency.

[0050] (24) A method based on a position threshold algorithm is used to extract the fixation point in the eye movement signal. This method helps to identify the fixation point in continuous viewpoints by comparing the position difference between two adjacent viewpoints in time sequence with a pre-set threshold.

[0051] (25) Pearson correlation analysis was used to perform feature screening on the collected vehicle driving parameters. Parameters with absolute values ​​of correlation coefficients greater than 0.5 were excluded. The remaining parameters were parameters with low correlation. The four vehicle driving parameters, namely steering wheel angle, steering wheel angle speed, accelerator pedal opening, and brake pedal opening, were used as driving behavior characteristic parameters. The correlation heat map is shown in Figure 2 As shown. The Pearson correlation analysis expression is as follows:

[0052]

[0053] Where cov(X,Y) is the covariance of random variables X and Y; σ X With σ Y are the standard deviations of X and Y respectively. X,Y The closer the value is to 1, the higher the correlation between the two variables.

[0054] (26) The data samples are divided into training set, test set, and validation set in a ratio of 8:1:1.

[0055] Step 3: Build a driving behavior prediction model based on CNN-GRU-Attention to predict driving behavior

[0056] The training set samples are input into the constructed CNN-GRU-Attention model, with multimodal physiological signals and vehicle driving parameters as input and driving behavior prediction results as output. According to the training loss and prediction accuracy, the network structure and parameters are continuously adjusted to train the driving behavior prediction model and visualize the confusion matrix. After the training is completed, the test set sample data is input into the model for driving behavior prediction. The CNN-GRU model structure is as follows: Figure 3 shown.

[0057] Step 3 includes the following sub-steps:

[0058] (31) The CNN local feature extraction network and the GRU global feature extraction network are integrated into a driver driving behavior prediction model. The output of the CNN local feature extraction network is used as the input of the GRU global feature extraction network, and the Attention mechanism is introduced between the GRU layer and the output layer to weight the importance of different channels and optimize the feature extraction process. The attention mechanism calculation formula is:

[0059]

[0060] Where: W s is the input vector, b s is the bias, and tanh is the activation function for calculating the attention weight matrix.

[0061] (32) The training set samples are input into the driving behavior prediction model, and the Adam method is used as the optimizer algorithm to iteratively train the model, calculate the LOSS value and accuracy, and save the optimal hyperparameters. The hyperparameter settings are shown in Table 1.

[0062] Table 1 Hyperparameter settings

[0063]

[0064] (33) The validation set samples are input into the CNN-GRU-Attention driving behavior prediction model for verification, and the driving behavior prediction results are output through the Softmax function. The network model convergence process and confusion matrix are shown in Figure 4 and Figure 5 .

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should analyze that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A driving behavior prediction method based on CNN-GRU-Attention, It is characterized in that The following steps are involved: S1 collects the driver's multimodal physiological signals and vehicle driving parameters during driving and performs preprocessing; S2 extracts and screens the features of the preprocessed physiological signals and vehicle driving parameters, calibrates the data according to the driving behaviors generated during driving, and then divides the data samples into training set, test set, and validation set; S3 inputs the training set samples into the constructed CNN-GRU-Attention driving behavior prediction model, takes multimodal physiological signals and vehicle driving parameters as input, and outputs driving behavior prediction results. According to the training loss and prediction accuracy, it continuously adjusts the network structure and parameters, trains the driving behavior prediction model, and visualizes the confusion matrix. After the training is completed, the validation set sample data is input into the model for driving behavior prediction.

2. The driving behavior prediction method based on CNN-GRU-Attention as claimed in claim 1, Features: The multimodal physiological signals collected in step S1 include electroencephalogram signals, eye movement signals, electromyography signals, and pulse wave signals; the vehicle driving parameters include vehicle speed, steering wheel angle, accelerator pedal opening and closing degree, brake pedal opening and closing degree, acceleration and other parameters.

3. The driving behavior prediction method based on CNN-GRU-Attention as claimed in claim 1 or 2, Features: Step S2 includes the following sub-steps: (21) The Hanning window function was used to perform signal truncation and the power spectral density of four frequency bands in the driver's EEG signal was extracted based on the short-time Fourier transform method; (22) The standard deviation of all cardiac intervals (SDNN) was selected as the time domain characteristic index of the pulse wave to evaluate the overall changes of the pulse wave; (23) Selecting the average power frequency to extract features from the collected electromyographic signals; (24) used a method based on position threshold algorithm to extract the fixation point in the eye movement signal; (25) Pearson correlation analysis was used to perform feature screening on the collected vehicle driving parameters, and four vehicle driving parameters, namely steering wheel angle, steering wheel angle speed, accelerator pedal opening, and brake pedal opening, were selected as driving behavior characteristic parameters.

4. The driving behavior prediction method based on CNN-GRU-Attention as claimed in claim 3, Features: Step S3 includes the following sub-steps: (31) A driving behavior prediction model based on CNN-GRU-Attention was constructed, in which the CNN network structure included 1 convolution layer with an input size of 64 and a convolution kernel size of 3, 1 batch normalization layer, 1 pooling layer with a window size of 2, 1 convolution layer with an input size of 128 and a convolution kernel size of 3, 1 batch normalization layer, and 1 pooling layer with a window size of 2. The GRU network structure included 2 GRU layers with 50 units. The Dropout mechanism was set in the GRU module and the attention mechanism was introduced between the GRU layer and the output layer. (32) The training samples are input into the driving behavior prediction model, the Adam method is used as the optimizer algorithm and the model is iteratively trained. The prediction accuracy of the validation set driving behavior and the convergence of the LOSS value are used as the model accuracy evaluation indicators, and the network parameters are continuously adjusted; (33) The validation set samples are input into the CNN-GRU-Attention driving behavior prediction model for verification, and the driving behavior prediction results are output through the Softmax function.

Images