Driver emotion recognition method based on multi-modal data fusion

Abstract

The invention belongs to the technical field of emotion recognition, and provides a multi-modal data fusion driver emotion recognition method and device, and the method comprises the steps: obtaining a multi-modal data set from a collected original data set of driver eyes, an original data set of a vehicle, and an original data set of an environment; performing feature extraction on the data in the multi-modal data set to obtain audio features, text features and visual features; performing feature alignment on the audio features, the text features and the visual features to obtain modal alignment features; performing feature fusion on the modal alignment features by using a mixed attention mechanism to obtain modal fusion features; and performing emotion recognition processing on the modal fusion features to obtain an emotion recognition result. According to the method, efficient fusion of the multi-modal data in the feature layer is realized, and the accuracy of driver emotion recognition is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to a driver emotion recognition method based on multimodal data fusion. Background Art

[0002] Emotions play an important role in expressing a person's mental state, and studies have shown that emotions affect drivers' decision-making, alertness, and driving behavior. In recent years, in order to reduce traffic accidents and improve road safety, the study of driver emotions has received more and more attention. Therefore, the development of automatic driver emotion recognition and corresponding human-computer interaction systems, taking certain actions to alleviate drivers' negative emotions and avoid traffic accidents are of great value to improving driving safety. However, due to the particularity of the driver's driving scene and the characteristics of multimodal technology itself, it is a huge challenge to fully integrate information from different modalities to accurately identify the driver's emotions. Summary of the invention

[0003] In view of the defects in the prior art, the present invention provides a driver emotion recognition method based on multimodal data fusion, which can improve the efficiency and accuracy of emergency treatment, effectively deal with injury risks, and ensure the life safety of patients.

[0004] A driver emotion recognition method based on multimodal data fusion, comprising:

[0005] Acquire a multimodal dataset from the collected original dataset of the driver's eyes, the original dataset of the vehicle, and the original dataset of the environment;

[0006] Perform feature extraction on the data in the multimodal dataset to obtain audio features, text features, and visual features;

[0007] Align audio features, text features, and visual features to obtain modality alignment features;

[0008] Use the hybrid attention mechanism to fuse the modality alignment features to obtain the modality fusion features;

[0009] The modal fusion features are processed for emotion recognition to obtain the emotion recognition results.

[0010] Furthermore, the audio features, text features and visual features are aligned to obtain modality alignment features, including:

[0011] Align audio features, text features, and visual features across modalities to obtain modality-invariant features;

[0012] Align audio features, text features, and visual features within a sample to obtain modality-specific features;

[0013] The modality-invariant features and modality-specific features are concatenated to obtain the modality-aligned features.

[0014] Furthermore, the audio features, text features and visual features are aligned across modalities to obtain modality-invariant features, including:

[0015] The audio features, text features and visual features are encoded into a modality-invariant subspace using a shared encoder to obtain a shared subspace audio feature, a shared subspace text feature and a shared subspace visual feature;

[0016] The shared subspace audio features, shared subspace text features, and shared subspace visual features are combined using a multi-head self-attention mechanism to obtain the initial modality-invariant features.

[0017] The maximum mean difference matrix is ​​used to enhance the similarity between the shared subspace audio features, shared subspace text features and shared subspace visual features in the initial modality invariant features to obtain the modality invariant features.

[0018] Furthermore, the audio features, text features, and visual features are aligned within the sample to obtain modality-specific features, including:

[0019] Encoding the audio features, text features and visual features into a modality-specific subspace using a separate encoder to obtain subspace-specific audio features, subspace-specific text features and subspace-specific visual features;

[0020] A supervised contrastive learning method is used to reduce the distances between subspace-specific audio features, subspace-specific text features and subspace-specific visual features to obtain modality-specific features.

[0021] Furthermore, the modality alignment features include audio alignment features, text alignment features, and visual alignment features.

[0022] Furthermore, the hybrid attention mechanism is used to fuse the modality alignment features to obtain modality fusion features, including:

[0023] Perform cross-fusion processing on the modality alignment features to obtain cross-fusion features;

[0024] Perform self-attention weighted processing on the cross-fusion features to obtain weighted cross-fusion features;

[0025] The weighted cross-fusion features are subjected to self-attention fusion processing to obtain modality fusion features.

[0026] Furthermore, the modality alignment features are cross-fused to obtain cross-fused features, including:

[0027] Calculate the dot product of the audio alignment feature, text alignment feature, and visual alignment feature in the modality alignment feature to obtain a similarity matrix;

[0028] Use the Softmax function to process the similarity matrix to obtain the cross attention matrix;

[0029] The audio alignment features, text alignment features and visual alignment features in the modality alignment features are fused using the cross-attention matrix to obtain the cross-fusion features.

[0030] Furthermore, the cross-fusion features are self-attention weighted to obtain weighted cross-fusion features, including:

[0031] Perform linear transformation on the cross-fusion features to obtain the self-attention vector;

[0032] Use the Softmax function to normalize the self-attention vector to obtain the self-attention weight;

[0033] The cross-fusion features are weighted using the self-attention weights to obtain weighted cross-fusion features.

[0034] An emotion recognition device for multimodal data fusion, characterized in that it comprises an acquisition unit, a feature extraction unit, a feature alignment unit, a feature fusion unit and a recognition unit, wherein:

[0035] The acquisition unit is used to acquire a multimodal data set from the collected original data set of the driver's eyes, the original data set of the vehicle, and the original data set of the environment;

[0036] The feature extraction unit is used to extract features from the data in the multimodal data set to obtain audio features, text features and visual features;

[0037] The feature alignment unit is used to align audio features, text features and visual features to obtain modality alignment features;

[0038] The feature fusion unit is used to fuse the modality alignment features using a hybrid attention mechanism to obtain modality fusion features;

[0039] The recognition unit is used to perform emotion recognition processing on the modal fusion features to obtain emotion recognition results.

[0040] The present invention proposes a novel multimodal fusion framework based on convolutional long short-term memory network and hybrid attention mechanism, which realizes the efficient fusion of multimodal data at the feature layer, provides richer multimodal information for accurately realizing the driver emotion recognition task, and makes the present invention have higher accuracy in driver emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific embodiments of the present invention, the following will briefly introduce the drawings required for the specific embodiments or the prior art description. In all the drawings, each element or part is not necessarily drawn according to the actual scale.

[0042] Figure 1 A flowchart of a driver emotion recognition method using multimodal data fusion provided by an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of the structure of a driver emotion recognition device with multimodal data fusion provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0045] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0046] In one embodiment, Figure 1 As shown, a driver emotion recognition method based on multimodal data fusion is provided, including:

[0047] 1. Obtain a multimodal dataset from the collected original datasets of the driver's eyes, the vehicle's original dataset, and the environment's original dataset;

[0048] An advanced driving simulator is used to collect original data sets of the driver's eyes, the vehicle's original data set, and the environment's original data set. This multimodal data set includes data in three modalities: vision, audio, and text.

[0049] 2. Extract features from the data in the multimodal dataset to obtain audio features, text features, and visual features;

[0050] a, t and v are used to represent audio data, text data and visual data, v a 、v t and v v Represents audio features, text features, and visual features, and Represents audio feature vector space, text feature vector space, and visual feature vector space.

[0051] Preferably, the pre-trained Wav2Vec2.0 model is used to extract audio features from the original waveform in the multimodal dataset, represented as

[0052] Audio features: Audio can reflect the speaker’s emotions through changes in speaking style, tone, pitch, and other related attributes. To extract audio features, each audio clip is first converted into a raw waveform of fixed length with a sampling rate of 16kHz, using zero padding for shorter audio.

[0053] The pre-trained sentence-level BERT model is used to extract text features from the multimodal dataset, represented as

[0054] The pre-trained image EfficientNet model is used to extract visual features from the multimodal dataset, represented as

[0055] Visual features: Reflect the speaker’s emotions by analyzing changes in relevant attributes such as facial expressions, body language, and visual background in video frames in a multimodal dataset.

[0056] 3. Align the audio features, text features and visual features to obtain modality alignment features;

[0057] Preferably, the audio features, text features and visual features are aligned to obtain modality alignment features, including:

[0058] (1) Align audio features, text features, and visual features across modalities to obtain modality-invariant features;

[0059] The inter-modal alignment of audio features, text features, and visual features aims to reduce the modality gap between audio features, text features, and visual features while maintaining the diversity of each modality.

[0060] Preferably, the audio features, text features and visual features are aligned across modalities to obtain modality-invariant features, including:

[0061] ① Using a shared encoder to encode audio features, text features and visual features into a modality-invariant subspace, to obtain shared subspace audio features, shared subspace text features and shared subspace visual features; and Among them, v ca 、v ct and v cv They represent shared subspace audio features, shared subspace text features and shared subspace visual features respectively, d represents the dimension size, and R represents the vector space.

[0062] Specifically, the following calculation formula is used to encode the audio features, text features and visual features into a modality-invariant subspace to obtain a shared subspace audio feature, a shared subspace text feature and a shared subspace visual feature:

[0063]

[0064]

[0065]

[0066] Among them, X, Y and Z are the text feature set, audio feature set and visual feature set extracted from the data in the multimodal dataset, n, m and k represent the vector size of text feature, audio feature and visual feature, respectively, L mmd (X, Z) represents the mmd loss function of text features and visual features, L mmd (X,Y),L mmd (Y, Z) Similarly, Φ represents the mapping function of the shared encoder, which maps audio features, text features, and visual features to a reproducing kernel Hilbert space (a modality-invariant subspace). ||.|| represents the L2 norm. x i ∈X,y i ∈Y and z i ∈Z represents the ith text feature in the text feature set, the ith audio feature in the audio feature set, the ith visual feature in the visual feature set, and the term and Represents the normalization factors of the text feature set, audio feature set, and visual feature set X, Y, and Z.

[0067] ② Use the multi-head self-attention mechanism to combine the shared subspace audio features, shared subspace text features, and shared subspace visual features to obtain the initial modality invariant features;

[0068] ③ Use the maximum mean difference matrix to enhance the similarity between the shared subspace audio features, shared subspace text features and shared subspace visual features in the initial modal invariant features to obtain the modal invariant features.

[0069] (2) Align audio features, text features, and visual features within the sample to obtain modality-specific features;

[0070] In order to preserve the different characteristics of each modality, separate encoders are used for the features extracted from text data, audio data, and visual data in the multimodal dataset, resulting in different subspaces.

[0071] Preferably, the audio features, text features and visual features are aligned within the sample to obtain modality-specific features, including:

[0072] ① Using a separate encoder to encode audio features, text features and visual features into a modality-specific subspace, to obtain subspace-specific audio features, subspace-specific text features and subspace-specific visual features;

[0073] ② A supervised contrastive learning method is used to reduce the distance between subspace-specific audio features, subspace-specific text features, and subspace-specific visual features to obtain modality-specific features.

[0074] For within-sample alignment, a supervised contrastive learning (SCL) method is used to learn discriminative representations that label subspace-specific audio features, subspace-specific text features, and subspace-specific visual features, which helps to align audio features, text features, and visual features of different modalities at the sample level, thereby further reducing the distance between subspace-specific audio features, subspace-specific text features, and subspace-specific visual features, narrowing the modality gap.

[0075] The SCL loss for intra-sample alignment can be expressed as:

[0076]

[0077] Where Lcl represents the SCL loss of intra-sample alignment, D represents all samples (each sample includes corresponding audio features, text features, and visual features), and P (j) represents all samples of the same category as the jth sample, A (j) =D,{j}, It represents the representation of the current j-th sample after feature fusion. Indicates P (j) The representation of a sample after feature fusion. Indicates A (j) The representation of a sample in after feature fusion. In the logarithmic part, the numerator is all positive similarities and the denominator is the similarity of all samples. τ is the temperature. After calculating the SCL loss, use a multi-head transformer to stack Processing is performed and the features of each modality are updated through the cross-model self-attention mechanism.

[0078] The previous part is to calculate the inter-modal alignment loss, and here we calculate the intra-sample alignment loss.

[0079] (3) Concatenate the modal invariant features and modal specific features to obtain the modal alignment features.

[0080] Both modality-invariant features and modality-specific features have features of the three modalities of audio, text, and vision, so the obtained modality-aligned features also have features of the three modalities of audio, text, and vision. Among them, modality-invariant features include audio-invariant features, text-invariant features, and visual-invariant features, modality-specific features include audio-specific features, text-specific features, and visual-specific features, and modality-aligned features include audio-aligned features, text-aligned features, and visual-aligned features.

[0081] Specifically, the modal invariant features and the modal specific features are concatenated using the following formula to obtain the modal alignment features.

[0082]

[0083]

[0084]

[0085] in, and Represent audio alignment features, text alignment features, and visual alignment features, respectively. They represent the text-specific features, audio-specific features, and visual-specific features in the modality-specific features after in-sample alignment, They respectively represent the text invariant features, audio invariant features, and visual invariant features in the modality-specific features after inter-modality alignment.

[0086] 4. Use the hybrid attention mechanism to fuse the modality alignment features to obtain the modality fusion features;

[0087] Preferably, a hybrid attention mechanism is used to fuse the modality alignment features to obtain modality fusion features, including:

[0088] (1) Performing cross-fusion processing on the audio alignment feature, text alignment feature, and visual alignment feature in the modal alignment feature to obtain a cross-fusion feature;

[0089] Preferably, the audio alignment feature, the text alignment feature and the visual alignment feature in the modality alignment feature are cross-fused to obtain the cross-fused feature, including:

[0090] ① Calculate the dot product of the audio alignment feature, text alignment feature, and visual alignment feature in the modal alignment feature to obtain the similarity matrix;

[0091] ②Use the Softmax function to process the similarity matrix to obtain the cross attention matrix;

[0092] ③ Use the cross-attention matrix to fuse the audio alignment features, text alignment features, and visual alignment features in the modal alignment features to obtain cross-fusion features.

[0093] Specifically, the cross-fusion feature of the audio alignment feature and the text alignment feature is calculated using the following formula:

[0094]

[0095] in, It can be described as a linear combination of audio alignment features and text alignment features, or it can be regarded as text features learned from audio alignment features. Similarly, text features learned from text alignment features and text features learned from visual alignment features can be obtained.

[0096] The text features learned from the text alignment features, the text features learned from the audio alignment features, and the text features learned from the visual alignment features are connected in series to obtain the text cross-fusion features, as follows:

[0097]

[0098] Among them, X T , X A and X V They represent text cross-fusion features, audio cross-fusion features, and visual cross-fusion features respectively. represents the text features learned from the text alignment features, represents the text features learned from the audio alignment features, represents the text features learned from the visual alignment features. The audio cross-fusion features and visual cross-fusion features are obtained in the same way.

[0099] The cross-fusion features include text cross-fusion features, audio cross-fusion features and visual cross-fusion features.

[0100] (2) Perform self-attention weighted processing on the cross-fusion features to obtain weighted cross-fusion features;

[0101] Preferably, the cross-fusion features are subjected to self-attention weighted processing to obtain weighted cross-fusion features, including:

[0102] ①Perform linear transformation on the cross-fusion features to obtain the self-attention vector;

[0103] ②Use the Softmax function to normalize the self-attention vector to obtain the self-attention weight;

[0104] ③ Use the self-attention weight to weight the cross-fusion features to obtain weighted cross-fusion features.

[0105] (3) The weighted cross-fusion features are subjected to self-attention fusion to obtain modal fusion features. Then, these cross-fusion features are further weighted through the self-attention mechanism.

[0106] Therefore, the cross-fusion features of different modalities are concatenated and expressed as follows:

[0107] K=[X T ,X A ,X V ]

[0108] in, represents the output matrix of the serial operation. Here, N represents the total number of samples, n c Represents the number of features after concatenation of the three modalities.

[0109] K can also be expressed in matrix form:

[0110]

[0111] Among them, K t =(k t_1 ,k t_2 ,...,k t_N ) T ,(t∈[1,n c ]), is a column vector representing the fused features through cross attention, and k represents the element in the vector.

[0112] Specifically, the cross-fusion features are weighted using the self-attention weights according to the following formula to obtain weighted cross-fusion features, and then the weighted cross-fusion features are subjected to self-attention fusion processing to obtain modality fusion features:

[0113] u t =tanh(w·K t +b)

[0114]

[0115]

[0116] Here, self-attention is defined as SA(·)=Attention(·).

[0117] Among them, K t =(k t_1 ,k t_2 ,...,k t_N ) T ,(t∈[1,n c]), is the column vector representing the fusion feature through cross attention, k represents the element in the vector, u is the self-attention vector obtained by linearly transforming the cross fusion feature, w is the weight matrix, and b is the bias vector. α t is the self-attention weight obtained by normalizing the product of u, and s represents the modal fusion feature obtained by self-attention fusion processing on the weighted cross-fusion feature. Therefore, the output vector of the self-attention mechanism is SA(K) = s(modal fusion feature).

[0118] In one of the embodiments, before using the hybrid attention mechanism to fuse the modal alignment features to obtain the modal fusion features, it also includes using a stacked convolutional long short-term memory model to extract the spatiotemporal features of the modal alignment features to obtain audio spatiotemporal features, text spatiotemporal features and visual spatiotemporal features, and then using the hybrid attention mechanism as above to fuse the audio spatiotemporal features, text spatiotemporal features and visual spatiotemporal features to obtain modal fusion features, and then using a fully connected neural network to perform emotion recognition processing on the modal fusion features to obtain emotion recognition results.

[0119] Specifically, the stacked convolutional long short-term memory model is composed of two stacked convolutional long short-term memory networks, and the convolutional long short-term memory network includes a ConvLSTM layer, a batch normalization layer, a dropout layer, and a three-dimensional average pooling layer.

[0120] 5. Perform emotion recognition processing on the modal fusion features to obtain emotion recognition results.

[0121] The fully connected neural network is used to perform emotion recognition processing on the modal fusion features to obtain the emotion recognition results.

[0122] The fully connected neural network consists of a fully connected layer and a softmax activation function.

[0123] Specifically, the fully connected layer in the fully connected neural network can be expressed as:

[0124] FC(SA(K))=s·W F

[0125] Among them, FC(SA(K)) is the output prediction value of the fully connected layer, W F It is the weight matrix of the fully connected layer, which determines the number of final output emotion categories.

[0126] In addition, the output prediction values ​​of the fully connected layer are normalized using the Softmax function, and the prediction error is evaluated using the cross entropy loss function.

[0127] Preferably, negative log-likelihood loss is used to optimize prediction accuracy, which is defined as:

[0128]

[0129] in, represents the output of the final classification layer of the fully connected neural network. The variable N is the total number of samples in the training dataset, and C is the total number of emotion categories. If the label of the jth sample corresponds to category c, then y j,c Set to 1 if yes, otherwise set to 0.

[0130] The final loss function is expressed as:

[0131] L all =α(L mmd (X,Y)+L mmd (X,Z)+L mmd (Y,Z))+βL cl +L nll

[0132] Among them, L mmd (X,Y) represents the MMD loss between the text feature set X and the audio feature set Y, L mmd (X,Z) and L mmd (Y, Z) is the same. cl represents the SCL loss of the sample, L nll Denotes negative log-likelihood loss, α and β denote weight factors of MMD and SCL losses, respectively. The MMD and SCL losses are assigned α and β, respectively. The best performance was obtained when α = β = 0.01, where the neural network focused more on emotion recognition, since this is the main task. In addition, the network also needs to reduce the MMD loss and SCL loss to achieve inter-modal alignment and intra-sample alignment, thereby achieving the main task by reducing the modality gap.

[0133] Finally, the parameters of the multimodal fusion framework (a multimodal fusion framework composed of a hybrid attention mechanism + a fully connected neural network) are updated through the back-propagation algorithm.

[0134] The present invention obtains a multimodal dataset based on an original dataset of the driver's eyes and an original dataset of the vehicle, then performs feature extraction on the data in the multimodal dataset to obtain audio features, text features and visual features, and then performs inter-modality alignment and intra-sample alignment on the audio features, text features and visual features, thereby achieving two-level alignment between modalities and within samples, reducing the modality gap, improving the fusion of multimodal features, and effectively aligning the audio features, text features and visual features between modalities and at the sample level, thereby significantly improving the accuracy of driver emotion recognition.

[0135] The present invention also proposes a new multimodal fusion framework based on convolutional long short-term memory network and hybrid attention mechanism, which realizes the efficient fusion of multimodal data at the feature layer, provides richer multimodal information for accurately realizing the driver emotion recognition task, and makes the present invention have higher accuracy in driver emotion recognition.

[0136] In one embodiment, a multimodal data fusion emotion recognition device is provided, comprising an acquisition unit, a feature extraction unit, a feature alignment unit, a feature fusion unit and a recognition unit, wherein:

[0137] The acquisition unit is used to acquire a multimodal data set from the collected original data set of the driver's eyes, the original data set of the vehicle, and the original data set of the environment;

[0138] The feature extraction unit is used to extract features from the data in the multimodal data set to obtain audio features, text features and visual features;

[0139] The feature alignment unit is used to align audio features, text features, and visual features.

[0140] Obtaining modal alignment features;

[0141] The feature fusion unit is used to fuse the modality alignment features using a hybrid attention mechanism.

[0142] Obtain modal fusion features;

[0143] The recognition unit is used to perform emotion recognition processing on the modal fusion features to obtain emotion recognition results.

[0144] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A multimodal data fusion emotion recognition method, characterized in that: include: Acquire a multimodal dataset from the collected original dataset of the driver's eyes, the original dataset of the vehicle, and the original dataset of the environment; Extracting features from the data in the multimodal dataset to obtain audio features, text features, and visual features; Align audio features, text features, and visual features to obtain modality alignment features; Using a hybrid attention mechanism to fuse the modality alignment features to obtain modality fusion features; The modality fusion features are processed for emotion recognition to obtain an emotion recognition result.

2. The emotion recognition method based on multimodal data fusion according to claim 1, characterized in that: The step of aligning the audio features, text features, and visual features to obtain modality alignment features includes: Align audio features, text features, and visual features across modalities to obtain modality-invariant features; Align audio features, text features, and visual features within a sample to obtain modality-specific features; The modality-invariant features and modality-specific features are concatenated to obtain the modality-aligned features.

3. The emotion recognition method based on multimodal data fusion as claimed in claim 2, characterized in that: The step of performing inter-modal alignment of audio features, text features, and visual features to obtain modality-invariant features includes: The audio features, text features and visual features are encoded into a modality-invariant subspace using a shared encoder to obtain a shared subspace audio feature, a shared subspace text feature and a shared subspace visual feature; The shared subspace audio features, shared subspace text features, and shared subspace visual features are combined using a multi-head self-attention mechanism to obtain the initial modality-invariant features. The maximum mean difference matrix is ​​used to enhance the similarity between the shared subspace audio features, shared subspace text features and shared subspace visual features in the initial modality invariant features to obtain the modality invariant features.

4. The emotion recognition method based on multimodal data fusion according to any one of claims 2 or 3, characterized in that: The audio features, text features and visual features are aligned within the sample to obtain modality-specific features, including: Encoding the audio features, text features and visual features into a modality-specific subspace using a separate encoder to obtain subspace-specific audio features, subspace-specific text features and subspace-specific visual features; A supervised contrastive learning method is used to reduce the distances between subspace-specific audio features, subspace-specific text features and subspace-specific visual features to obtain modality-specific features.

5. The emotion recognition method based on multimodal data fusion as claimed in claim 2, characterized in that: The modality alignment features include audio alignment features, text alignment features and visual alignment features.

6. The emotion recognition method of multimodal data fusion according to claim 5, characterized in that: The hybrid attention mechanism is used to fuse the modality alignment features to obtain the modality fusion features, including: Perform cross-fusion processing on the modality alignment features to obtain cross-fusion features; Perform self-attention weighted processing on the cross-fusion features to obtain weighted cross-fusion features; The weighted cross-fusion features are subjected to self-attention fusion processing to obtain modality fusion features.

7. The emotion recognition method based on multimodal data fusion as claimed in claim 6, characterized in that: The cross-fusion processing of the modality alignment features to obtain the cross-fusion features includes: Calculate the dot product of the audio alignment feature, text alignment feature, and visual alignment feature in the modality alignment feature to obtain a similarity matrix; Use the Softmax function to process the similarity matrix to obtain the cross attention matrix; The audio alignment features, text alignment features and visual alignment features in the modality alignment features are fused using the cross-attention matrix to obtain the cross-fusion features.

8. The emotion recognition method of multimodal data fusion according to claim 7, characterized in that: The self-attention weighted processing is performed on the cross-fusion features to obtain weighted cross-fusion features, including: Performing a linear transformation on the cross-fusion features to obtain a self-attention vector; Normalize the self-attention vector using the Softmax function to obtain a self-attention weight; The cross-fusion features are weighted using the self-attention weights to obtain weighted cross-fusion features.

9. An emotion recognition device based on multimodal data fusion, characterized in that: It includes an acquisition unit, a feature extraction unit, a feature alignment unit, a feature fusion unit and a recognition unit, wherein: The acquisition unit is used to acquire a multimodal data set from the collected original data set of the driver's eyes, the original data set of the vehicle, and the original data set of the environment; The feature extraction unit is used to extract features from the data in the multimodal data set to obtain audio features, text features and visual features; The feature alignment unit is used to align the audio features, text features and visual features to obtain modality alignment features; The feature fusion unit is used to fuse the modality alignment features using a hybrid attention mechanism to obtain a modality fusion feature; The recognition unit is used to perform emotion recognition processing on the modal fusion features to obtain emotion recognition results.

Images