A driver emotion recognition method based on multi-modal feature fusion

Abstract

The application discloses a driver emotion recognition method based on multi-modal feature fusion, belongs to the technical field of emotion recognition, and embeds a multi-modal dynamic fusion module based on a dual adaptive weighting mechanism into a driver emotion recognition model to recognize driver emotions in real time, specifically comprising the following steps: constructing a multi-modal feature extraction network composed of four parallel sub-networks to extract feature vectors of high-level semantic information in four modes; using a multi-modal dynamic fusion module MDFFM based on a dual adaptive weighting mechanism to jointly optimize mode-level weights and channel-level weights; performing ablation on different modal combinations and the MDFFM, projecting the fused features onto three driver emotions, and using the fused features to recognize the emotions of the driver. The driver emotion recognition method based on multi-modal feature fusion is helpful to improve the safety and human-computer interaction experience of an intelligent driving system, and provides a new path for further optimization of multi-modal emotion recognition technology.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, and in particular to a driver emotion recognition method based on multimodal feature fusion. Background Technology

[0002] Driver behavior changes influenced by emotions are a significant factor contributing to traffic safety risks. Existing research indicates that negative emotions and high stress levels can both increase collision risk. Compared to drivers in a neutral or sad emotional state, drivers in a happy state tend to exhibit shorter reaction times and gaze durations in dangerous situations. Furthermore, the lasting effects are significant, particularly with anxiety being associated with higher speeds and more frequent speeding. Therefore, accurate perception of driver emotions will become an indispensable part of intelligent driving systems.

[0003] The complexity of human emotional expression in dynamic driving scenarios suggests that single-modal data may be less robust and reliable, prompting researchers to turn to driver emotion recognition methods that integrate multimodal data. Research on driver emotion recognition primarily involves three aspects from the perspective of data sources: visual data, vehicle control data, and physiological data. First, the road environment and facial expressions receive considerable attention in the visual data captured by cameras. The road environment, as an external driving condition, influences the driver's experience and, consequently, their emotions. Compared to urban environments, highway environments with fewer stress-inducing objects are more conducive to positive driver emotions. Certain road objects, such as larger vehicles (e.g., trucks), road users (e.g., cyclists), and infrastructure elements (e.g., intersections), are highly correlated with higher levels of subjective stress. Facial expressions, as a nonverbal communication tool, are the most direct feature of external responses to human emotions. However, facial expressions can be misleading because how humans react with facial expressions in a particular situation depends on the region or territory in which they are located.

[0004] While multimodal fusion-based driver emotion recognition has become a popular trend, aiming to provide more reliable recognition results and improve driving safety, the fused modalities are usually limited to a finite combination of two or three modalities, mainly focusing on facial expressions or EEG signals, making it difficult to comprehensively reflect the driver's emotional state in complex driving environments. Furthermore, the fusion module relies on the user's prior knowledge and extensive experiments to search for the optimal fusion coefficients, lacking flexibility and adaptability. Summary of the Invention

[0005] The purpose of this invention is to provide a driver emotion recognition method based on multimodal feature fusion to solve the problems existing in the background technology.

[0006] To achieve the above objectives, this invention provides a driver emotion recognition method based on multimodal feature fusion. A multimodal dynamic fusion module MDFFM based on a dual adaptive weighting mechanism is embedded in the driver emotion recognition model DER-MDF to identify driver emotions in real time. The specific execution steps are as follows: S1. Obtain modal data and construct a multimodal feature extraction network consisting of four parallel sub-networks. Extract feature vectors of high-level semantic information from the modal data through the constructed multimodal feature extraction network. S2. Use the multimodal dynamic fusion module MDFFM based on a dual adaptive weighting mechanism to jointly optimize modal-level weights and channel-level weights; S3. Ablation is performed on different modal combinations and MDFFM, and the fused features are projected onto three driver emotions to identify the driver's emotions.

[0007] Preferably, in step S1: The acquired modal data includes four types of images: road environment images, facial expression images, electroencephalograms (EEGs), and spectrograms generated from vehicle speed data; these four images serve as input data for four parallel sub-networks, respectively. The four parallel sub-networks include a road environment image sub-network, a facial expression sub-network, a brain electronics network, and a vehicle speed sub-network; each sub-network includes 8 convolutional blocks, 4 pooling layers, and 2 fully connected layers.

[0008] Preferably, the specific process of step S1 is as follows: S11. Extract key features from the four modal data and output the original road environment feature vector, original facial expression feature vector, original EEG feature vector, and original vehicle speed feature vector. S12. Introduce GLI-CAM modules into each sub-network to assign weights to features of different channels.

[0009] Preferably, step S12 contains the following: S121. The feature maps in the backbone network are aggregated by global average pooling (GAP) to obtain aggregated features. S122. Perform cross-channel interaction and adaptive convolution operations on the aggregated features to extract deep semantic information. S123. Introduce learnable position embeddings to fuse multi-level feature information and generate the final channel attention weights.

[0010] Preferably, step S2 contains the following: S21. The original feature vectors of the four modes are represented as the original road environment feature vectors. Original facial feature vector Original EEG feature vector and the original vehicle speed feature vector All are located In the diagram, B represents the batch size and C represents the channel depth. S22. Apply channel attention extraction to the original feature vector of each modality to obtain the refined feature vector of each modality, which is used to capture local information between feature channels; S23. Obtain the refined feature vectors of each mode through step S22. , , , The refined feature vectors of each modality are fused by adding the corresponding elements to obtain the initial fused feature vector. The initial fused feature vector Processed through 1D convolutional layers in the channel dimension; S24. After extracting channel attention in step S22 and improving the initial fusion features in step S23, the channel enhancement feature vector is... The algorithm captures information between different feature channels, applies one-dimensional convolution for further feature extraction, and passes the extraction results through a sigmoid activation function to generate channel-level weights. ; S25, will , , , The pieces are stitched together along the feature dimension along the channel to form a shape. The concatenated tensors are passed through a fully connected layer and then activated by the softmax function to generate modal weights. ; S26. Summing the corresponding elements of each mode using the modality-level weights from step S25, and adjusting the importance of feature channels using the channel-level weights from S24, to generate the final fusion vector. .

[0011] Preferably, the calculation formula in step S21 is as follows: ; in, This represents a feature vector that incorporates global information across the feature channels; Includes all four modes; and These are two-dimensional convolutions, with kernel sizes of [missing information]. and R=4; It represents the feature vectors derived from four modalities: road environment images, facial expression images, electroencephalograms, and spectrograms derived from vehicle speed data.

[0012] Preferably, the calculation formulas for the initial fused feature vector and the channel-enhanced feature vector in step S23 are as follows: ; ; in, Represents the channel-enhanced feature vector; and The kernel sizes are respectively and One-dimensional convolution; The ratio is approximately 16.

[0013] Preferably, the calculation formula for step S24 is as follows: ; in, As channel-level weights, they are applied fully to the feature vectors of the four modalities. , , , To obtain the final fusion vector, , , , All located in In the diagram, B represents the batch size and C represents the channel depth. This represents one-dimensional convolution; This represents the Sigmoid activation function.

[0014] Preferably, the calculation formula for step S25 is as follows: ; in, These are the modal-level weights assigned to the four modes, satisfying... and .

[0015] Preferably, the calculation formula for step S26 is as follows: ; in, This represents the final fusion vector.

[0016] Therefore, the driver emotion recognition method based on multimodal feature fusion described above, as used in this invention, has the following beneficial effects: (1) A multimodal driver emotion recognition model integrating four modalities—road environment data, facial expression data, electroencephalogram data, and vehicle speed data—was constructed, which effectively expanded the input modalities of emotion recognition and provided a foundation for more comprehensive and refined driver emotion perception. (2) An MDFFM fusion module was designed. Through a dual adaptive mechanism, the robustness and expressiveness of cross-modal feature fusion were improved, and the model was guided to pay more attention to the features that contribute most to emotion recognition. (3) While ensuring high accuracy, it also has excellent reasoning speed, taking into account both accuracy and real-time performance, and showing good potential for practical application.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is an embodiment of the overall framework of a driver emotion recognition method based on multimodal feature fusion according to the present invention; Figure 2 This is a schematic diagram of the functional architecture of GLI-CAM according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of MDFFM according to an embodiment of the present invention; wherein, (a) is a schematic diagram of channel-level weights and modal-level weights generation; and (b) is a schematic diagram of the dual adaptive weighting mechanism. Figure 4 This is a schematic diagram of vehicle speed data segmentation based on time series-assisted prediction according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the electroencephalogram (EEG) construction process according to an embodiment of the present invention, wherein (a) is a schematic diagram of a head-mounted EEG device; (b) is a schematic diagram of the EEG electrode positions; and (c) is a schematic diagram of EEG for three emotions. Figure 6 This is a schematic diagram of the convergence curves of accuracy and loss on the training set and validation set in an embodiment of the present invention; wherein, (a) is a schematic diagram of the accuracy curves on the training set and validation set; and (b) is a schematic diagram of the loss curves on the training set and validation set. Figure 7 These are visualization diagrams of confusion matrices for driver emotion recognition using different methods in embodiments of the present invention; wherein, (a) is a visualization diagram of the FAN confusion matrix; (b) is a visualization diagram of the Former-DFER confusion matrix; (c) is a visualization diagram of the DECNet confusion matrix; and (d) is a visualization diagram of the DER-MDF confusion matrix. Figure 8 This is a schematic diagram comparing the inference speed of an embodiment of the present invention. Detailed Implementation

[0019] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0020] Please see Figure 1 A driver emotion recognition method based on multimodal feature fusion. The multimodal dynamic fusion module MDFFM, based on a dual adaptive weighting mechanism, is embedded in the driver emotion recognition model DER-MDF to identify driver emotions in real time. The specific execution steps are as follows: S1. Acquire modal data and construct a multimodal feature extraction network consisting of four parallel sub-networks. Extract feature vectors containing high-level semantic information from the modal data using this network. The modal data acquired in step S1 includes four types of images: road environment images, facial expression images, electroencephalograms (EEGs), and a spectrogram generated from vehicle speed data. These four images serve as input data for the four parallel sub-networks, respectively. The four parallel sub-networks include a road environment image sub-network, a facial expression sub-network, a brain-computer interface network, and a vehicle speed sub-network; each sub-network includes 8 convolutional blocks, 4 pooling layers, and 2 fully connected layers, such as... Figure 2 .

[0021] S11. Key features are extracted from the four modalities of data. The road environment image sub-network extracts the original road environment feature vector perceived by the driver in their forward field of vision. The facial expression sub-network extracts the original vehicle speed feature vector related to the driver's emotions. The brain electronics network extracts the original electroencephalogram (EEG) feature vector of brain electrical activity states. The vehicle speed sub-network is used to extract the original vehicle speed feature vector during driving.

[0022] S12. Introduce GLI-CAM modules into each sub-network to assign weights to features of different channels.

[0023] The feature maps in the backbone network are aggregated by global average pooling (GAP) to obtain aggregated features. Cross-channel interaction and adaptive convolution operations are performed on the aggregated features to extract deep semantic information. Learnable position embeddings are introduced to fuse multi-level feature information and generate the final channel attention weights.

[0024] S2. The multimodal dynamic fusion module MDFFM, based on a dual adaptive weighting mechanism, is used to jointly optimize modal-level weights and channel-level weights. The MDFFM structure is as follows: Figure 3 .

[0025] S21. The original feature vectors of the four modes are represented as the original road environment feature vectors. Original facial feature vector Original EEG feature vector and the original vehicle speed feature vector All are located In the diagram, B represents the batch size and C represents the channel depth.

[0026] S22. Apply channel attention extraction to the original feature vector of each modality to obtain the refined feature vector of each modality, which is used to capture local information between feature channels. ; in, This represents a feature vector that incorporates global information across the feature channels; Includes all four modes; and These are two-dimensional convolutions, with kernel sizes of [missing information]. and R=4; It represents the feature vectors derived from four modalities: road environment images, facial expression images, electroencephalograms, and spectrograms derived from vehicle speed data.

[0027] S23. Obtain the refined feature vectors of each mode through step S22. , , , The refined feature vectors of each modality are fused by adding the corresponding elements to obtain the initial fused feature vector. The initial fused feature vector The channel dimension is processed using 1D convolutional layers; the initial fused feature vector and channel-enhanced feature vector are calculated using the following formulas: ; ; in, Represents the channel-enhanced feature vector; and The kernel sizes are respectively and One-dimensional convolution; The reduction ratio is 16; S24. After extracting channel attention in step S22 and improving the initial fusion features in step S23, the channel enhancement feature vector is... The algorithm captures information between different feature channels, applies one-dimensional convolution for further feature extraction, and passes the extraction results through a sigmoid activation function to generate channel-level weights. The calculation formula is as follows: ; in, As channel-level weights, they are applied fully to the feature vectors of the four modalities. , , , To obtain the final fusion vector, , , , All located in In the diagram, B represents the batch size and C represents the channel depth. This represents one-dimensional convolution; This represents the Sigmoid activation function.

[0028] S25, will , , , The pieces are stitched together along the feature dimension along the channel to form a shape. The concatenated tensors are passed through a fully connected layer and then activated by the softmax function to generate modal weights. The calculation formula is as follows: ; in, These are the modal-level weights assigned to the four modes, satisfying... and .

[0029] S26. Calculate the weighted sum of the original modal features using modal-level weights. Summate the corresponding elements of each modality using the modal-level weights from step S25. Adjust the importance of feature channels using the channel-level weights from S24 to generate the final fusion vector. The dual adaptive weighting mechanism controls the contribution of each modality to the fusion result through modality-level weights and finely adjusts the importance of feature channels through channel-level weights, thereby achieving hierarchical feature optimization. The calculation formula is as follows: ; in, This represents the final fusion vector.

[0030] Preserve the gradient in the formula ( This ensures end-to-end trainability while maintaining computational efficiency.

[0031] S3. Ablation is performed on different modal combinations and MDFFM, and the fused features are projected onto three driver emotions to identify the driver's emotions.

[0032] Example PPB-Emo is a publicly available multimodal driver emotion dataset that has been widely used in driver emotion classification research in recent years. This dataset was collected from 240 valid driving tasks completed by 40 participants in a simulated highway driving scenario. It includes physiological signals, facial videos, driving behavior data, and discrete emotion labels. The experiment employed an emotion-inducing paradigm: drivers performed driving tasks after watching emotionally stimulating audio-visual materials. The dataset shows a balanced distribution of samples from happy, neutral, and sad emotions. Participants ranged in age from 19 to 58 years (mean ± standard deviation: 28.10 ± 9.47), including 31 men and 9 women. All participants held valid driver's licenses and had at least one year of driving experience (driving experience range: 1–32 years; mean ± standard deviation: 5.58 ± 6.02 years).

[0033] During training, this embodiment crops the input landscape image, facial expression image, and EEG data to 224×224 pixels. Then, the spectrogram obtained from vehicle speed data via short-time Fourier transform is sampled to 257×113 pixels. The network training epoch is set to 100, and the batch size is set to 16. The network model uses stochastic gradient descent (SGD) optimization, with momentum set to 0.9 and weight decay set to 1 × 10⁻⁶. -4 The initial learning rate was set to 0.25 × 10⁻⁶. -4 The learning rate is reduced by a factor of 10 when the epoch is between 50 and 100, and by a factor of 10 when the epoch is greater than 100. In the speech spectrogram generation function, this embodiment sets the parameter npreseg to 512 and the parameter noverlap to 483. The network model in this embodiment is implemented in the PyTorch framework, and all models are trained and tested on a server equipped with a GeForce RTX 3060 GPU.

[0034] The landscape and facial expression images in the PPB-Emo dataset were acquired as video at a frame rate of 30 FPS. Vehicle speeds were collected at a frequency of 60 Hz. EEG signals were directly acquired and processed using NIC2 software at a sampling frequency of 500 Hz. According to the model input requirements of this embodiment, the network needs to simultaneously receive landscape images, facial expression images, EEG images, and spectrograms converted from vehicle speed data at a specific moment. Therefore, to meet the data input requirements for model training, this embodiment preprocessed the data from all four modalities.

[0035] For landscape images and facial expression images, this embodiment extracts the last frame of each second from the corresponding video as the representative image input for that moment. For vehicle speed data, this embodiment converts it into a spectrogram to reflect the one-dimensional speed variation characteristics in the time-frequency domain. The introduction of the spectrogram helps enhance the feature representation capability of speed information. Considering that time series information contains richer variation characteristics, it can compensate for the lack of information in speed values ​​at a single moment. This embodiment further employs a time series-assisted prediction method with overlapping windows for sequence enhancement. Specifically, the first 2 seconds (recorded as X) are used to assist the current time prediction, that is, the number of seconds of overlapping speed data between two adjacent predictions is X. For example, the speed sequence used for predicting the driver's emotion at the 3rd second is the data from the 1st to the 3rd second, and the prediction of the driver's emotion at the 4th second uses the data from the 2nd to the 4th second (see...). Figure 4 Furthermore, considering that the PPB-Emo dataset only contains 60 velocity sampling points per second, directly generating a spectrogram may result in insufficient time-frequency resolution. Therefore, this embodiment employs the cubic interpolation method of the interp1d interpolation function to extend the velocity time series, and further converts the extended time series information into a spectrogram format.

[0036] Regarding EEG signals, the PPB-Emo dataset preprocesses the raw EEG signals to suppress noise, remove artifacts, and extract useful information. The EEG signals, after bandpass filtering at 2–40 Hz, approximately follow a Gaussian distribution. Figure 5 As shown in (b), the electrodes are arranged according to the international 10-10 system layout. This embodiment uses a non-overlapping 1-second time window to extract the differential entropy features of the EEG signal. The differential entropy features corresponding to each second can be further mapped to an EEG image to represent the spatial distribution of brain activity at that moment. Differential entropy is a stable and high-performing EEG feature widely used in emotion recognition research; it is essentially used to quantify the uncertainty in the probability distribution of continuous random variables. Figure 5 (c) in the image represents the generated brain topographic images for three different emotions: happiness, neutrality, and sadness. Color changes indicate different levels of brain activation intensity. The data range is normalized to 0 to 1, with activation intensity ranging from low to high. The calculation formula is as follows: ; in, It is a random variable. yes The probability density function. Assume a random variable... Follows a Gaussian distribution ,but The differential entropy can be expressed as: .

[0037] The three sets of experimental data were divided proportionally into training, validation, and test sets. By using stratified sampling, this embodiment ensures that the training, validation, and test sets provide data in equal proportions regarding driver road type and driving behavior. The training, validation, and test sets contain 2664, 333, and 333 roadside images, respectively.

[0038] The proposed DER-MDF model was developed using the PyTorch framework and trained and tested on a server equipped with a GeForce RTX 3060 GPU. During training, road environment images, facial expression images, and EEG images were cropped to 224×224 pixels. Vehicle speed data underwent short-time Fourier transform, resulting in spectrogram samples of 257×113. The model was tested over 100 cycles with a batch size of 16 and a size of 0.25×10⁻⁶. -4 The system is trained using an initial learning rate. Stochastic gradient descent (SGD) is used for optimization with a momentum of 0.9 and weight decay of 1 × 10⁻⁶. -4 This is used for parameter optimization. When the number of epochs is between 50 and 100, the learning rate is reduced by an order of magnitude, and when the number of epochs exceeds 100, the learning rate is reduced by the same order of magnitude again. In the generation of the spectrogram, npreseg and noverlap are set to 512 and 483, respectively. Figure 6 Figures (a) and (b) show the accuracy and loss curves for the training and validation sets. DER-MDF exhibits a fast convergence speed and achieves good convergence results in both training and validation. The final training accuracy of the model is approximately 99.84%, and the validation accuracy is approximately 97.89%. The final training loss is approximately 0.003, while the validation loss is approximately 0.048. These results validate the effectiveness of the proposed DER-MDF model in recognizing driver emotions.

[0039] Table 1 shows the test results of the proposed DER-MDF model on the PPB-Emo dataset and its comparison with other commonly used methods (such as Frame Attention Network (FAN), Dynamic Facial Expression Recognition Converter (Former-DFER), and Bimodal Non-Contact Driver Emotion Classification Network (DECNet)). Compared with other commonly used methods, DER-MDF achieves the best results in driver emotion recognition. Specifically, DER-MDF has an accuracy of 98.49% and an F1 score of 0.985, which are 18.72% and 18.77% higher than other methods on average, respectively. Its precision is 0.986 and recall is 0.985, which also outperforms all other methods in both evaluation metrics. Figure 7A visualization of the confusion matrix is ​​presented, where all scores are standardized to the range of 0 to 1. In the confusion matrix, the i-th row represents the true class, and the j-th column corresponds to the predicted class. The element (i, j) represents the percentage of samples in class i that are classified as class j. Figure 7 (d) in the other Figure 7 Comparing the subgraphs in the model, DER-MDF performs consistently and satisfactorily in recognizing the three types of driver emotions (HD, ND, and SAD). However, other commonly used methods show differences in recognizing these three types of driver emotions, indicating an imbalance in recognition performance. When FAN is applied to driver emotion recognition, 74% of HD samples are correctly classified as HD, 88% of ND samples are classified as ND, and 72% of SAD samples are classified as SAD. Similarly, DECNet achieves a 71% correct classification rate for HD, a 90% correct classification rate for ND, and an 88% correct classification rate for SAD. These findings validate the effectiveness of the network architecture and multimodal fusion strategy proposed in this embodiment, contributing to driver emotion recognition.

[0040] Table 1. Performance comparison of DER-MDF with other methods on the PPB-Emo dataset.

[0041] In terms of computation, DER-MDF performs exceptionally well. Its inference speed is 155 frames per second, demonstrating superior performance compared to other driver-driven emotion recognition models. Figure 8As shown, the inference speeds of the compared models range from 7 to 140 frames per second. Among these models, the fastest inference speed is You Only Look Once version 7 (YOLOv7) (140 frames per second), 15 frames per second slower than DER-MDF. Furthermore, DER-MDF has 55.81 MB of model parameters, which can be considered a lightweight model in the field of autonomous driving. Therefore, DER-MDF maintains a fast inference speed while controlling the size of its model parameters, indicating its potential for real-time processing, especially in driver emotion recognition where near-instantaneous analysis of large numbers of image or video frames is required. The ability of a model to quickly process images and detect emotions is crucial for real-time applications in interactive systems. Models with shorter inference times and higher frames per second (FPS) are better suited for these scenarios. For example, security systems that monitor crowd behavior in real time require models with high FPS to detect and respond to potential threats or disturbances in a timely manner. While more complex models may offer higher accuracy, they may also result in slower inference times, making them less suitable for real-time applications. Consider, for example, a virtual assistant that relies on emotion detection to provide personalized responses. A highly accurate but slow model may cause significant delays in interaction, thus degrading the user experience.

[0042] ablation experiment This embodiment not only evaluated the emotion recognition performance when including all modal data, but also systematically analyzed the contribution and differences of different modality combination schemes to the recognition results. This was done to observe the effectiveness of multimodal fusion, and the resulting evaluation metrics are shown in Table 2. When only two modalities were fused, the recognition accuracy of the DER-MDF network ranged from 68.17% to 97.19%, and the F1 score ranged from 0.680 to 0.969. After introducing a third modality, the model's recognition accuracy improved to 96.09% to 97.51%, and the F1 score improved to 0.962 to 0.977. Finally, the model fusing all four modalities achieved the highest performance, with an accuracy of 98.49% and an F1 score of 0.985. The results show that the model fusing all modalities has an accuracy 1.30%–30.32% higher than the model using only two modalities, and an accuracy 0.98%–2.4% higher than the model using three modalities. This fully demonstrates that fusing data from four modalities makes a significant positive contribution to driver emotion recognition tasks and can effectively promote the sharing of feature information among modalities. Furthermore, under the same number of fused modalities, the model including facial expression data consistently demonstrates higher recognition accuracy than the model including vehicle speed data.

[0043] Furthermore, with the same number of fused modalities, the model including the facial expression modality achieved higher recognition accuracy than the model including the vehicle speed modality. When fusing two modalities, the model including the facial expression modality had an average accuracy of 94.57%, while the model including the vehicle speed modality had an average accuracy of 77.27%, a difference of 17.3%. When fusing three modalities, the highest accuracy combination was the combination of road environment (excluding vehicle speed), facial expression, and EEG, achieving an accuracy of 97.51%. This indicates that the rich feature information contained in the facial expression modality plays a very important role in driver emotion recognition tasks.

[0044] Table 2 Ablation study on the impact of different modal combinations on driver emotion recognition performance

[0045] To verify the importance of the MDFFM proposed in this embodiment for driver emotion recognition, an ablation experiment was conducted on MDFFM, and the results are shown in Table 3. Since the MDFFM proposed in this embodiment jointly optimizes modality-level weights and channel-level weights, it effectively integrates feature vectors containing high-level semantic information from all modal data, thus achieving optimal results. Specifically, compared to the model without MDFFM, the accuracy of DER-MDF improved by 1.5%. It can be seen that MDFFM improves the accuracy of DER-MDF for driver emotion recognition by increasing the degree of fusion between different modalities.

[0046] Table 3 Ablation study on the impact of MDFFM on driver emotion recognition performance

[0047] Therefore, this invention adopts the aforementioned driver emotion recognition method based on multimodal feature fusion and proposes a novel driver emotion recognition model (DER-MDF). This model integrates four modalities from three aspects: visual perception, physiological signals, and driving behavior. The model identifies three emotions in drivers: happiness, neutrality, and sadness, with high accuracy and fast inference speed, providing a valuable solution for real-time driver emotion recognition and human-computer interaction in intelligent driving systems. To fully utilize the different importance of different modalities to promote feature fusion, a multimodal dynamic fusion module (MDFFM) based on a dual adaptive weighting mechanism is proposed. This not only improves the feature representation of the network but also reduces the dependence on prior knowledge, enhancing the model's adaptability and robustness in the multimodal fusion process.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A driver emotion recognition method based on multimodal feature fusion, characterized in that, The multimodal dynamic fusion module MDFFM, based on a dual adaptive weighting mechanism, is embedded in the driver emotion recognition model DER-MDF to identify driver emotions in real time. The specific execution steps are as follows: S1. Obtain modal data and construct a multimodal feature extraction network consisting of four parallel sub-networks. Extract feature vectors of high-level semantic information from the modal data through the constructed multimodal feature extraction network. S2. Use the multimodal dynamic fusion module MDFFM based on a dual adaptive weighting mechanism to jointly optimize modal-level weights and channel-level weights; S3. Ablation is performed on different modal combinations and MDFFM, and the fused features are projected onto three driver emotions to identify the driver's emotions.

2. The driver emotion recognition method based on multimodal feature fusion according to claim 1, characterized in that, In step S1: The acquired modal data includes four types of images: road environment images, facial expression images, electroencephalograms (EEGs), and spectrograms generated from vehicle speed data; these four images serve as input data for four parallel sub-networks, respectively. The four parallel sub-networks include a road environment image sub-network, a facial expression sub-network, a brain electronics network, and a vehicle speed sub-network; each sub-network includes 8 convolutional blocks, 4 pooling layers, and 2 fully connected layers.

3. The driver emotion recognition method based on multimodal feature fusion according to claim 2, characterized in that, The specific process of step S1 is as follows: S11. Extract key features from the four modal data and output corresponding road environment feature vector, facial expression feature vector, electroencephalogram feature vector and vehicle speed feature vector. S12. Introduce GLI-CAM modules into each sub-network to assign weights to features of different channels.

4. The driver emotion recognition method based on multimodal feature fusion according to claim 3, characterized in that, Step S12 is as follows: S121. The feature maps in the backbone network are aggregated by global average pooling (GAP) to obtain aggregated features. S122. Perform cross-channel interaction and adaptive convolution operations on the aggregated features to extract deep semantic information. S123. Introduce learnable position embeddings to fuse multi-level feature information and generate the final channel attention weights.

5. The driver emotion recognition method based on multimodal feature fusion according to claim 4, characterized in that, Step S2 is as follows: S21. The original feature vectors of the four modes are represented as the original road environment feature vectors. Original facial feature vector Original EEG feature vector and the original vehicle speed feature vector All are located In the diagram, B represents the batch size and C represents the channel depth. S22. Apply channel attention extraction to the original feature vector of each modality to obtain the refined feature vector of each modality, which is used to capture local information between feature channels; S23. Obtain the refined feature vectors of each mode through step S22. , , , The refined feature vectors of each modality are fused by adding the corresponding elements to obtain the initial fused feature vector. The initial fused feature vector Processed through 1D convolutional layers in the channel dimension; S24. After extracting channel attention in step S22 and improving the initial fusion features in step S23, the channel enhancement feature vector is... The algorithm captures information between different feature channels, applies one-dimensional convolution for further feature extraction, and passes the extraction results through a sigmoid activation function to generate channel-level weights. ; S25, will , , , The pieces are stitched together along the feature dimension along the channel to form a shape. The concatenated tensors are passed through a fully connected layer and then activated by the softmax function to generate modal weights. ; S26. Summing the corresponding elements of each mode using the modality-level weights from step S25, and adjusting the importance of feature channels using the channel-level weights from S24, to generate the final fusion vector. .

6. The driver emotion recognition method based on multimodal feature fusion according to claim 5, characterized in that, The calculation formula in step S22 is as follows: ； in, This represents a feature vector that incorporates global information across the feature channels; Includes all four modes; and These are two-dimensional convolutions, with kernel sizes of [missing information]. and R=4; This represents the feature vectors from the four modalities.

7. The driver emotion recognition method based on multimodal feature fusion according to claim 6, characterized in that, The formulas for calculating the initial fused feature vector and the channel-enhanced feature vector in step S23 are as follows: ； ； in, It is the initial fused feature vector; Represents the channel-enhanced feature vector; and The kernel sizes are respectively and One-dimensional convolution; The ratio is approximately 16.

8. The driver emotion recognition method based on multimodal feature fusion according to claim 7, characterized in that, The calculation formula for step S24 is as follows: ； in, As channel-level weights, they are applied fully to the feature vectors of the four modalities. , , , To obtain the final fusion vector, , , , All located in In the diagram, B represents the batch size and C represents the channel depth. This represents one-dimensional convolution; This represents the Sigmoid activation function.

9. The driver emotion recognition method based on multimodal feature fusion according to claim 8, characterized in that, The calculation formula for step S25 is as follows: ； in, These are the modal-level weights assigned to the four modes, satisfying... and .

10. A driver emotion recognition method based on multimodal feature fusion according to claim 9, characterized in that, The calculation formula for step S26 is as follows: ； in, This represents the final fusion vector.

Images