Dual-modal emotion recognition method based on multi-granularity attention and feature distribution calibration

Abstract

The application discloses a multi-granularity attention and feature distribution calibration bimodal emotion recognition method, and steps are as follows: step 1, a sample set is acquired; step 2, an MGA-FDC model is built; step 3, the MGA-FDC model is trained by using data segments in a training set, and a trained MGA-FDC model is obtained; step 4, the trained MGA-FDC model is tested by using data segments in a test set, and an emotion recognition model is obtained; and step 5, a sample to be recognized is input into the above emotion recognition model, and EEG signal and image sample emotion recognition of the sample to be recognized can be performed. The MGA-FDC model built in the application has optimal performance compared with current methods, can more effectively complete a bimodal emotion recognition task based on EEG signals and face images, and has more practical significance for actual application.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition, specifically involving a dual-modal emotion recognition method based on multi-granularity attention and feature distribution calibration. Background Technology

[0002] In the field of bimodal emotion recognition, the combination of physiological signals, represented by electroencephalography (EEG), and physical signals, represented by facial images, has become a research hotspot. This combination yields richer and more complete feature information, effectively improving emotion recognition performance and leading to its widespread application in areas such as safe driving, distance education, and healthcare. However, due to the non-stationarity and low signal-to-noise ratio of EEG signals, EEG feature learning is more challenging than facial image feature learning. How to extract more generalizable EEG features for emotion recognition and how to fully utilize multimodal information to capture the consistency and complementarity of emotional semantics have become key research focuses.

[0003] Regarding how to extract more generalizable EEG features for emotion recognition: existing EEG feature extraction methods mainly include manual feature extraction, EEG topography, and deep learning. However, manual feature extraction methods ignore the correlation between EEG signal channels, limiting model performance; EEG topography methods depend on the quality of manual features; existing deep learning methods learn features from EEG signals through simple convolution operations, ignoring the fact that EEG signals are inherently complex time-series data, making it difficult for models to fully model the complex spatiotemporal relationships within EEG signals, thus affecting the accuracy of final emotion recognition. Regarding how to fully utilize multimodal information to capture the consistency and complementarity of emotional semantics: feature-level fusion and decision-level fusion are the most common strategies. However, feature-level fusion methods cannot overcome the semantic gap between modalities and cannot fully exploit the semantic consistency between modalities; decision-level fusion methods only focus on extracting the specific semantic information of each modality, failing to extract the shared semantic information between modalities, thus limiting emotion recognition performance.

[0004] As can be seen from the above, how to learn features with more significant emotional semantics from nonlinear multi-channel time series such as EEG signals for emotion recognition, and how to fully utilize multimodal information to capture the consistency and complementarity of emotional semantics, thereby improving the performance of emotion recognition models, have become key issues that need to be addressed in this field. To this end, this application proposes a bimodal emotion recognition method based on multigranular attention and feature distribution calibration, namely the Multigranular Attention and Feature Distribution Calibration (MGA-FDC) model, to improve the accuracy of bimodal emotion recognition of EEG signals and face images. Summary of the Invention

[0005] The purpose of this invention is to provide a bimodal emotion recognition method based on multi-granularity attention and feature distribution calibration, which can effectively improve the accuracy of bimodal emotion recognition of EEG signals and face images.

[0006] The technical solution adopted in this invention is a dual-modal emotion recognition method based on multi-granularity attention and feature distribution calibration, and the specific steps are as follows:

[0007] Step 1: Obtain the sample set and divide each sample in the sample set into multiple 1-second data segments to obtain the data segment set. Divide the data segment set into 5 parts, take 1 part as the test set, and the rest as the training set.

[0008] Step 2: Build the MGA-FDC model;

[0009] Step 3: Train the MGA-FDC model using data fragments from the training set to obtain the trained MGA-FDC model;

[0010] Step 4: Test the trained MGA-FDC model using data fragments from the test set to obtain the emotion recognition model;

[0011] Step 5: Input the sample to be identified into the above emotion recognition model to perform EEG signal and emotion recognition of the image sample.

[0012] The invention is further characterized by:

[0013] The sample set in step 1 was obtained from the publicly available DEAP and MAHNOB-HCI datasets. Each sample in the sample set contains a face image video and its corresponding EEG signal and emotion category label.

[0014] The MGA-FDC model in step 2 consists of five parts arranged layer by layer from top to bottom: an EEG signal feature learning model, a face image feature learning model, a multi-granularity attention module, a feature distribution calibration module, and a classifier. The construction process of the MGA-FDC model is as follows:

[0015] Step 2.1: Construct an EEG signal feature learning model to obtain the output features of the EEG modes of the EEG signal samples in the sample set;

[0016] Step 2.2: Construct a face image feature learning model to obtain the output features of face image modalities from the sample set of face image videos;

[0017] Step 2.3: The output features of the EEG modality and the output features of the face image modality are fused through multi-granularity attention to obtain the processed bimodal features, EEG modal features, and face image modal features;

[0018] Step 2.4: Perform feature distribution calibration on the processed bimodal features, EEG modal features, and face image modal features through the feature distribution calibration module to obtain the calibrated EEG modal features and face image modal features;

[0019] Step 2.5: Perform a concatenation operation on the two calibrated modal features obtained in Step 2.4, and input the result into the classifier to complete the construction of the MGA-FDC model.

[0020] The EEG signal feature learning model includes a hierarchical first two-dimensional convolutional module, a depthwise separable convolutional module, and a second two-dimensional convolutional module. The specific data processing procedure of this model is as follows:

[0021] Step 2.1.1: Use the EEG signal samples in the sample set as the input of the first two-dimensional convolutional module, and set the convolutional kernel size to half of the sampling rate f, so that the model can capture frequency information of 2Hz and above, and obtain the pre-processed EEG signal features.

[0022] Step 2.1.2: Use the pre-processed EEG signal features obtained in Step 2.1.1 as the input features of the depthwise separable convolution module. Use depthwise convolution to capture the spatial relationship of the pre-processed EEG signal features and use separable convolution to capture the temporal features of the pre-processed EEG signal features. By combining depthwise convolution and separable convolution, EEG signal features containing spatiotemporal relationships are obtained.

[0023] Step 2.1.3: Use the EEG signal features containing spatiotemporal relationships as the input of the second 2D convolution module, and further learn the EEG signal features through 2D convolution to obtain the output features E1 of the EEG mode.

[0024] The face image feature learning model includes convolutional neural networks and convolutional long short-term memory networks. The data processing procedure of this model is as follows:

[0025] Step 2.2.1: Randomly select 5 frames from any 1s data segment as face image data input to the pre-trained convolutional neural network. The pre-trained convolutional neural network processes each face image sample to obtain the face image features corresponding to each face image sample.

[0026] Step 2.2.2: The face image features obtained in Step 2.2.1 are sequentially fed into the convolutional long short-term memory network, and then the output features of the convolutional long short-term memory network are flattened to capture the temporal features between each face image frame.

[0027] Step 2.2.3: The face image features obtained in step 2.2.2 are sequentially fed into the convolutional neural network to adjust the size of the output feature dimension, so as to obtain the output feature F1 of the face image modality.

[0028] The multi-granularity attention module consists of n-layer MGA modules, and its specific processing procedure is as follows:

[0029] Step 2.3.1: For the first-level MGA module, EF1 is obtained by element-wise addition of E1 and F1. For each subsequent MGA module, E1 is obtained by element-wise addition of F1. n F n With EF n-1 The element-wise addition operation yields EF n Features, the formula is as follows:

[0030]

[0031] Where N represents the highest level of the MGA, and n represents the level of any MGA between 1 and N. E represents element-wise addition. n F n EEG modal features and face image modal features, respectively, are input to the nth layer MGA module. n To pass through E n F n With EF n-1 Element-level addition operations yield bimodal features;

[0032] Step 2.3.2, based on EEG modal features E n Bimodal feature EF n Facial image modal features F n The three types of features generate Q respectively. E Q EF Q F Three query vectors, simultaneously capturing F n Internal key feature information, based on F n Generate key vector K F The details are as follows:

[0033]

[0034] in, These represent the modules in the nth-level MGA module used to generate the query vector Q. E Query vector Q EF Query vector Q F Key vector K F The parameter matrix;

[0035] Step 2.3.3, based on F n Combined with query vector QE Q EF Q F Generate α(Q) E ,K F ), α(Q EF ,K F ), α(Q F ,K F Three sets of attention scores, the process is as follows:

[0036]

[0037] Where, d k Represents the scaling dot product factor, α(Q) E ,K F ), α(Q EF ,K F ), α(Q F ,K F ) respectively represent F n Combined with query vector Q E Q EF Q F The generated attention score, Indicates K F Perform a transpose operation;

[0038] Step 2.3.4, based on α(Q) E ,K F ), α(Q EF ,K F ), α(Q F ,K F The three attention scores are further used to generate the final attention score α(Q,K), and a weighted vector V is added based on α(Q,K). F , to obtain F n+1 The following formula is used to complete feature learning within the facial image modality:

[0039] α(Q,K)=softmax(α(Q E ,K F )+α(Q EF ,K F )+α(Q F ,K F (9)

[0040] F n+1 =V F α(Q,K) (10)

[0041] Where α(Q,K) is the final attention score, and V F The weighted vector of α(Q,K), F n+1 For n+1 layers of face image modal features, α(QE ,K F ), α(Q EF ,K F ), α(Q F ,K F ) represents F n Combined with query vector Q E Q EF Q F The generated attention score;

[0042] Step 2.3.5: Repeat steps 2.3.1-2.3.4 to obtain the EEG modal features E of the (n+1)th layer MGA module. n+1 This completes the feature learning within the EEG modality.

[0043] The data processing procedure of the feature distribution calibration module is as follows:

[0044] Step 2.4.1 and Step 2.3.1 generate EF n The method is the same, generating EF, and EF is generated by E. N+1 F N+1 and EF N The element-wise addition is obtained as follows:

[0045]

[0046] Step 2.4.2: Generate the scaling weight w corresponding to F. F The following steps are taken: Global features of F in the c and l dimensions are obtained through adaptive global average pooling. Then, dimensionality reduction of these global features is performed using 1D convolution. Next, batch normalization and ReLU activation are added to reduce the divergence of data distribution in the c and l dimensions of F, thereby reducing the difficulty of network training and completing non-linear activation. Finally, 1D convolution is used again to increase the dimensionality of the features, resulting in more representative features F in the c and l dimensions. c F l Finally, based on F c F l The dot product operation yields w. F The formula is as follows:

[0047] F c =Conv1d(ReLU(BN(Conv1d(Avgpool(EF))))) (12)

[0048] F l =Conv1d(ReLU(BN(Conv1d(Avgpool(Reshape(EF)))))) (13)

[0049] wF =F c ·F l (14)

[0050] Where Reshape(·) represents the dimension transformation operation, AvgPool(·) represents the global average pooling operation, Conv1d(·) represents the 1D convolution operation, BN(·) represents the batch normalization operation, and ReLU(·) represents the ReLU activation function.

[0051] Step 2.4.3: Generate the offset weight b corresponding to F. F First, dimensionality reduction of F is achieved through 1D convolution, followed by batch normalization and ReLU activation. Then, dimensionality increase of the features is achieved through 1D convolution. Finally, feature normalization is achieved through the Sigmoid activation function. The offset weights are implemented using the following formula:

[0052] b F =Sigmoid(Conv1d(ReLU(BN(Conv1d(EF))))) (15)

[0053] Step 2.4.4: Based on scaling weight w F , offset weight b F The calibration of the characteristic distribution of F is completed using the following formula:

[0054]

[0055] Where ⊙ represents element-wise multiplication, F′ is the calibrated face image modal feature, and b F For the offset weight, w F To scale the weights;

[0056] Step 2.4.5: Repeat steps 2.4.1-2.4.4 to obtain... E′ represents the calibrated EEG modal characteristics, b E For the offset weight, w E This is for scaling weights.

[0057] The specific processing procedure of the classifier is as follows: E′ and F′ after the feature distribution calibration module are concatenated and flattened into a one-dimensional vector, which is then fed into the classifier to obtain the constructed MGA-FDC model, as detailed below:

[0058] Predict=Classifiaction(concat(E′,F′)) (17)

[0059] Wherein, Concat(·) is used to represent the concatenation operation, Classification(·) is used to represent the classifier, which includes three parts: a fully connected layer, a Dropout layer, and an activation function, F′ is the calibrated face image modal feature, and E′ is the calibrated EEG modal feature.

[0060] The specific process of step 3 is as follows: each data segment sample in the training set is fed into the MGA-FDC model built in step 2, and its predicted sentiment category is output respectively. The predicted sentiment category of each data segment sample is compared with its sentiment category label, and the loss between the two is minimized. The network parameters in the model are updated through the backpropagation algorithm until the model maintains stable sentiment prediction performance on the training set. The model is then saved to obtain the trained model.

[0061] The specific process of step 4 is as follows: Test the trained model with the test set and save the experimental results of the model on this part of the test set. Test five times in total, ensuring that the data segments used in the test set do not overlap each time. Then take the average of the five test results as the final test result of the model. If the test result meets the expected requirements, save the model and obtain the emotion recognition model. If the test result does not meet the expected requirements, adjust the model architecture and parameters, and repeat steps 1 to 4 until the test result meets the expected requirements. Save the model and obtain the emotion recognition model.

[0062] The beneficial effects of this invention are:

[0063] (1) A bimodal emotion recognition method based on multi-granularity attention and feature distribution calibration. In order to fully capture the feature information of each modality and learn more significant emotional semantic features, a multi-granularity attention (MGA) module was designed. First, bimodal features were generated by element-level addition. Then, query vectors were generated based on each modal feature and the bimodal features. Three attention matrices were generated based on the three different query vectors. Finally, the final attention score was generated based on the three sets of attention matrices and applied to the input modal features. This enabled the model to distinguish the different importance of features specific to the modality and features shared between modalities in a more granular way.

[0064] (2) In order to alleviate the semantic gap between modalities and capture the emotional semantic consistency between modalities, and effectively complete feature fusion, a feature distribution calibration (FDC) module is proposed. Based on the bimodal features, scaling weights and offset weights corresponding to each modality are generated, and the input features are weighted based on these two sets of weights. By calibrating the distribution of high-level semantic features of each modality through bimodal features, the semantic differences of each modality feature are effectively reduced.

[0065] (3) Extensive experiments were conducted on the DEAP and MAHNOB-HCI datasets. Two experimental methods, cross-trial validation and 5-fold cross-validation, were used to demonstrate the feasibility and effectiveness of the proposed method. Attached Figure Description

[0066] Figure 1 This is an architecture diagram of the MGA-FDC model built in the bimodal emotion recognition method of multi-granularity attention and feature distribution calibration of the present invention;

[0067] Figure 2 This is a schematic diagram of the face image feature learning module in the MGA-FDC model built in the dual-modal emotion recognition method of multi-granularity attention and feature distribution calibration of the present invention;

[0068] Figure 3 This is a schematic diagram of the structure of the multi-granularity attention module (E&F→F) in the MGA-FDC model built in the dual-modal emotion recognition method of multi-granularity attention and feature distribution calibration of the present invention;

[0069] Figure 4 This is a schematic diagram of the feature distribution calibration module (EF→F) structure in the MGA-FDC model built in the bimodal emotion recognition method of multi-granularity attention and feature distribution calibration of the present invention. Detailed Implementation

[0070] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0071] Example 1:

[0072] In this embodiment, a 3-layer MGA-FDC model is constructed, with the architecture as follows: Figure 1 As shown, the specific steps are as follows:

[0073] Step 1: Extract multiple data samples from a public dataset, divide each sample into multiple 1-second segments, and use these as experimental data samples. Each experimental data sample contains two modal signals: EEG signal data and face image data. The public datasets used in this invention include the DEAP dataset and the MAHNOB-HCI dataset.

[0074] Step 2: Build the MGA-FDC model;

[0075] The MGA-FDC model consists of five parts: an EEG signal feature learning model, a face image feature learning model, a multi-granularity attention module, a feature distribution calibration module, and a classifier.

[0076] The EEG signal feature learning model includes a hierarchical first two-dimensional convolutional module, a depthwise separable convolutional module, and a second two-dimensional convolutional module. The specific data processing procedure of this model is as follows:

[0077] Step 2.1.1: Use the EEG signal samples in the sample set as the input of the first two-dimensional convolutional module, and set the convolutional kernel size to half of the sampling rate f, so that the model can capture frequency information of 2Hz and above, and obtain the pre-processed EEG signal features.

[0078] Step 2.1.2: The pre-processed EEG signal features obtained in Step 2.1.1 are used as input features of the depthwise separable convolution module. Depthwise convolution is used to capture the spatial relationship of the pre-processed EEG signal features, and separable convolution is used to capture the temporal features of the pre-processed EEG signal features. By combining depthwise convolution and separable convolution, EEG signal features containing spatiotemporal relationships are obtained. The model avoids excessive convolution operations and has lower computational cost.

[0079] Step 2.1.3: Use the EEG signal features containing spatiotemporal relationships as the input of the second 2D convolution module, and further learn the EEG signal features through 2D convolution to obtain the output features of the EEG mode.

[0080] The facial image feature learning model includes a convolutional neural network and a convolutional long short-term memory network. The data processing procedure of this model is as follows: Figure 2 As shown, the details are as follows:

[0081] Step 2.2.1: Randomly select 5 frames from any 1s data segment as face image data input to the pre-trained convolutional neural network. The pre-trained convolutional neural network processes each face image sample to obtain the face image features corresponding to each face image sample.

[0082] A convolutional neural network pre-trained on the AFEW-VA dataset was used to process each frame of face image samples to obtain the face image features corresponding to each frame of face image samples, as shown in Table 1, which are the specific parameters of the convolutional neural network.

[0083] Table 1

[0084] enter \ \ \ 3×64×64 Convolutional layer 3×3 1 ReLU 32×62×62 Max pooling 32@3×3 2 \ 32×30×30 Convolutional layer 64@3×3 1 ReLU 64×28×28 Max pooling 3×3 2 \ 64×13×13 Convolutional layer 128@3×3 1 ReLU 128×13×13 Max pooling 3×3 2 \ 128×6×6

[0085] Step 2.2.2: The face image features obtained in Step 2.2.1 are sequentially fed into the convolutional long short-term memory network, and then the output features of the convolutional long short-term memory network are flattened to capture the temporal features between each face image frame.

[0086] Step 2.2.3: Feed the face image features obtained in Step 2.2.2 into the convolutional neural network in sequence and adjust the size of the output feature dimension to obtain the output features of the face image modality.

[0087] The specific process of building a multi-granularity attention module is as follows: Figure 3 As shown, the details are as follows:

[0088] Step 2.3.1: Construct a multi-granularity attention (E&F→F) module. For the first-layer MGA module, EF1 is obtained by element-wise addition of E1 and F1. For each subsequent MGA module, such as the third-layer MGA module, EF3 is obtained by element-wise addition of E3, F3, and EF2, as shown in the following formula:

[0089]

[0090] Where N represents the highest level of the MGA. This represents element-wise addition operations.

[0091] Step 2.3.2: Generate Q based on the three types of features E3, EF3, and F3 respectively. E Q EF Q F Three query vectors are used, and to avoid computational redundancy, F is fully captured. n Internal key feature information, generating key vector K based on F3. F The details are as follows:

[0092]

[0093] in, These represent the modules used to generate Q in the third-level MGA module. E Q EF Q F K F The parameter matrix;

[0094] Step 2.3.3, based on F n Combined with query vector Q E Q EF Q F For α(Q) E ,K F ), α(Q EF ,K F ), α(Q F ,K F The generation of three sets of attention scores is a strategy that not only effectively reduces computation but also forces the model to learn more reasonable and consistent key vectors, which helps improve the model's generalization performance. The process is as follows:

[0095]

[0096]

[0097] Where, d k This represents the scaling factor for the dot product.

[0098] Step 2.3.4, based on α(Q) E ,K F ), α(Q EF ,K F ), α(Q F ,K F The three attention scores are further used to generate the final attention score α(Q,K), and a weighted vector V is added based on α(Q,K). F , to obtain F n+1 The following formula is used to complete feature learning within the facial image modality:

[0099] α(Q,K)=softmax(α(Q E ,K F )+α(Q EF ,K F )+α(Q F ,K F (9)

[0100] F n+1 =V F α(Q,K) (10)

[0101] Step 2.3.5: Repeat steps 2.3.1-2.3.4 to build a multi-granularity attention (E&F→E) module and obtain the EEG modal features E. n Bimodal feature EF n Facial image modal features F n Using the input features of the nth layer MGA module, we obtain the EEG modal features E of the (n+1)th layer MGA module. n+1 This completes the feature learning within the EEG modality.

[0102] The process of building the feature distribution calibration module is as follows: Figure 4 As shown, the details are as follows:

[0103] Step 2.4.1: Build the feature distribution calibration (EF→F) model and compare it with EF. n The generation method is the same; EF is obtained by adding the element-wise elements of E4, F4, and EF3, that is...

[0104]

[0105] Step 2.4.2: Generate the scaling weight w corresponding to F. FConsidering F is processed separately along the c and l dimensions. First, adaptive global average pooling is used to obtain the global features of F in the c and l dimensions respectively. Then, dimensionality reduction of the features is achieved through 1D convolution. Next, batch normalization and ReLU activation function are added to reduce the divergence of data in the feature distribution, reduce the difficulty of network training, and complete non-linear activation. Finally, 1D convolution is used again to increase the dimensionality of the features, resulting in more representative features F in the c and l dimensions. c F l Finally, based on F c F l The dot product operation yields w. F The formula for implementing this part is as follows:

[0106] F c =Conv1d(ReLU(BN(Conv1d(Avgpool(EF))))) (12)

[0107] F l =Conv1d(ReLU(BN(Conv1d(Avgpool(Reshape(EF)))))) (13)

[0108] w F =F c ·F l (14)

[0109] Where Reshape(·) represents the dimension transformation operation, AvgPool(·) represents the global average pooling operation, Conv1d(·) represents the 1D convolution operation, BN(·) represents the batch normalization operation, and ReLU(·) represents the ReLU activation function.

[0110] Step 2.4.3: Generate the offset weight b corresponding to F. F First, dimensionality reduction of the features F is achieved through 1D convolution, followed by batch normalization and ReLU activation. Then, dimensionality increase of the features is achieved through 1D convolution, and finally, feature normalization is achieved through the Sigmoid activation function. The offset weights are implemented using the following formula:

[0111] b F =Sigmoid(Conv1d(ReLU(BN(Conv1d(EF))))) (15)

[0112] Step 2.4.4: Based on scaling weight w F , offset weight b F Complete the calibration of the characteristic distribution of F. The formula is as follows:

[0113]

[0114] Where ⊙ represents element-wise multiplication;

[0115] Step 2.4.5: Repeat steps 2.4.1-2.4.4 to complete the construction of the feature distribution calibration (EF→E) module, and obtain... E′ represents the calibrated EEG modal characteristics, b E For the offset weight, w E This is for scaling weights.

[0116] The classifier construction process is as follows: E′ and F′, after passing through the feature distribution calibration module, are concatenated and flattened into a one-dimensional vector, which is then fed into the classifier to complete the final sentiment prediction. The formula for this part is as follows:

[0117] Predict=Classifiaction(concat(E′,F′)) (17)

[0118] Concat(·) is used to represent the concatenation operation, and Classification(·) is used to represent the classifier, which includes three parts: a fully connected layer, a Dropout layer, and an activation function.

[0119] In the above steps, E and F are E without loss of generality. m and F n m and n are natural numbers.

[0120] The dimensionality of the input and output features of each module within the MGA-FDC model is shown in Table 2:

[0121] Table 2

[0122]

[0123] Step 3: Input each data segment sample in the training set into the MGA-FDC model built in Step 2, and output its predicted sentiment category. Compare the predicted sentiment category of each data segment sample with its sentiment category label, and minimize the loss between the two. Update the network parameters in the model through the backpropagation algorithm until the model maintains stable sentiment prediction performance on the training set. Save the model to obtain the trained model.

[0124] Step 4: Test the trained model using the test set and save the experimental results of the model on this part of the test set. Test five times in total, ensuring that the data segments used in the test set do not overlap each time. Then, take the average of the five test results as the final test result of the model. If the test result meets the expected requirements, save the model to obtain the emotion recognition model. If the test result does not meet the expected requirements, adjust the model architecture and parameters, and repeat steps 1 to 3 until the test result meets the expected experimental results. Save the model to obtain the emotion recognition model.

[0125] Step 4: Use the emotion recognition model to test 5 frames of face images and the corresponding 1s EEG signal. The emotion category label Valence is "high".

[0126] The emotion recognition model built in this embodiment processes 5 frames of face images and corresponding 1s EEG signals, and the predicted emotion category label Valence is "high", which is consistent with the data emotion category label Valence. This indicates that the model has successfully achieved the emotion classification task of EEG signals and face images.

[0127] Example 2:

[0128] Compared with Example 1, this example constructs a 4-layer MGA-FDC model for testing. The specific steps are the same as in Example 1. Based on the saved 4-layer MGA-FDC model, the following tests are performed:

[0129] Step 1: Use 1 second of EEG signal data and the corresponding 5 frames of face images as the data input for the MSTFAI-SCFF model, with the emotion category label Valence set to "high";

[0130] Step 2: EEG signal data is used as input for the EEG feature learning part to obtain output features; 5 frames of face images are used as input for the face image feature learning part to obtain output features.

[0131] Step 3: The output features of the EEG feature learning part and the face image feature learning part are successively fed into the multi-granularity attention module and the feature distribution calibration module, and then into the classifier. The predicted sentiment category label Valence is "high", which is consistent with the data sentiment category label Valence, indicating that the model has successfully achieved the sentiment classification task for EEG and face images.

[0132] Example 3:

[0133] Compared with Example 1, this example constructs a 6-layer MGA-FDC model for testing. The specific steps are the same as in Example 1. Based on the saved 6-layer MGA-FDC model, the following tests are performed:

[0134] Step 1: Use 1 second of EEG signal data and the corresponding 5 frames of face images as the data input for the MGA-FDC model, with the sentiment category label Arousal set to "low";

[0135] Step 2: The MGA-FDC model is used to process the input data. The predicted sentiment category label Arousal is "low", which is consistent with the sentiment category label Arousal in the data. This indicates that the model has successfully achieved the sentiment classification task for EEG and face images.

[0136] The table below compares the performance of the MGA-FDC model with other methods (the best results are marked in bold, reproducible results are marked with *, and work using the 10-fold cross-validation method is marked with ●):

[0137] Table 3

[0138]

[0139] The experiments conducted in this invention were performed on the DEAP and MAHNOB-HCI datasets, and two validation methods, cross-trial validation and 5-fold cross-validation, were used to evaluate and analyze the performance of the invention. Table 3 shows the comparison results of the proposed method with other methods on the DEAP and MAHNOB-HCI datasets. Valence and Arousal represent the binary classification results of the model in terms of effectiveness and activation, respectively, while Emotion represents the quadratic classification result of the model in terms of emotion.

[0140] In cross-experimental validation, the MGA-FDC model achieved accuracies of 82.56%, 82.44%, and 66.51% for Valence, Arousal, and Emotion on the DEAP dataset, respectively, and 81.63%, 80.81%, and 65.28% for the same categories on the MAHNOB-HCI dataset. In 5-fold cross-validation, the MGA-FDC model also demonstrated excellent performance in the Valence, Arousal, and Emotion categories, achieving accuracies of 97.48%, 97.96%, and 90.04% on the DEAP dataset, and 98.83%, 99.26%, and 91.89% on the MAHNOB-HCI dataset, respectively, outperforming existing methods.

[0141] Based on the combined performance of the MGA-FDC model on both datasets, it can be seen that the MGA-FDC model has the best performance compared to the current methods. It can more effectively complete the bimodal emotion recognition task based on EEG signals and face images, and has more practical significance for practical applications.

Claims

1. A bimodal emotion recognition method based on multi-granularity attention and feature distribution calibration, characterized in that, The specific steps are as follows: Step 1: Obtain the sample set and divide each sample in the sample set into multiple 1-second data segments to obtain the data segment set. Divide the data segment set into 5 parts, take 1 part as the test set, and the rest as the training set. Step 2: Build the MGA-FDC model; The MGA-FDC model comprises five parts arranged layer by layer from top to bottom: an EEG signal feature learning model, a face image feature learning model, a multi-granularity attention module, a feature distribution calibration module, and a classifier. The construction process of the MGA-FDC model is as follows: Step 2.1: Construct an EEG signal feature learning model to obtain the output features of the EEG modes of the EEG signal samples in the sample set; Step 2.2: Construct a face image feature learning model to obtain the output features of face image modalities from the sample set of face image videos; Step 2.3: The output features of the EEG modality and the output features of the face image modality are fused through a multi-granularity attention module to obtain the processed dual-modality features, EEG modality features, and face image modality features; Step 2.4: Perform feature distribution calibration on the processed bimodal features, EEG modal features, and face image modal features through the feature distribution calibration module to obtain the calibrated EEG modal features and face image modal features; Step 2.5: Perform a concatenation operation on the two calibrated modal features obtained in Step 2.4, and input the result into the classifier to complete the construction of the MGA-FDC model; The multi-granularity attention module includes The specific processing procedure of the layer MGA module is as follows: Step 2.3.1: For the first-level MGA module, through... and Element-level addition operation yields For each subsequent MGA module, through , and Element-level addition operation yields Features, the formula is as follows: (1) in, Indicates the highest level of the MGA. Indicates 1- N The number of layers in any MGA between, This represents element-wise addition. , The first The input EEG modal features and face image modal features of the layer MGA module, To pass , and Element-level addition operations yield bimodal features; Step 2.3.2: Based on EEG modal features Dual-modal features facial image modal features Three types of features are generated respectively. , , Three query vectors, simultaneously, fully capture Internal key feature information, based on Generate key vectors The details are as follows: (2) (3) (4) (5) in, , , , They represent the first The layer MGA module is used to generate query vectors. Query vector Query vector Key vector The parameter matrix; Step 2.3.3, based on Combine query vector , , generate , , Three sets of attention scores, the process is as follows: (6) (7) (8) in, Represents the scaling factor of the dot product. , , They represent Combine query vector , , The generated attention score, Indicates to Perform a transpose operation; Step 2.3.4, based on , , The three attention scores are further used to generate the final attention score. and based on Weighted vector ,get The following formula is used to complete feature learning within the facial image modality: (9) (10) in, For the final attention score, for Weighted vector for n +1 layer of facial image modal features, , , express Combine query vector , , The generated attention score; Step 2.3.5: Repeat steps 2.3.1-2.3.4 to obtain the result. EEG modal characteristics of layer MGA modules Complete feature learning within the EEG modality; The data processing procedure of the feature distribution calibration module is as follows: Step 2.4.1 and Step 2.3.1 generate The method is the same. Depend on , and The element-wise addition is obtained as follows: (11) Step 2.4.2, Generate Corresponding scaling weights : Obtained through adaptive global average pooling exist , Global features in dimensionality, then processed by 1D convolution pairs exist , Dimensionality is reduced by performing global feature reduction; then, batch normalization and ReLU activation function are added to reduce dimensionality. exist , The degree of data divergence in the global feature distribution across dimensions reduces the difficulty of network training and enables non-linear activation; subsequently, dimensionality enhancement of the features is achieved again through 1D convolution, resulting in... exist , More representative features in dimensions , Finally, based on , The dot product operation yields... The formula is as follows: (12) (13) (14) in, Represents a dimensional transformation operation. This represents the global average pooling operation. Represents a 1D convolution operation. This represents a batch normalization operation. Represents the ReLU activation function; Step 2.4.3, Generate Corresponding offset weight First, complete the operation using 1D convolution. The feature dimensionality is reduced, and batch normalization and ReLU activation function are introduced. Then, the feature dimensionality is increased by 1D convolution. Finally, the feature normalization is completed by Sigmoid activation function. The offset weight is implemented by the following formula: (15) Step 2.4.4: Based on scaling weights Offset weight Complete the task The calibration of the feature distribution is achieved using the following formula: (16) in, Represents element-wise multiplication. For the calibrated facial image modal features, For offset weights, To scale the weights; Step 2.4.5: Repeat steps 2.4.1-2.4.4 to obtain... = , For the calibrated EEG modal characteristics, For offset weights, To scale the weights; Step 3: Train the MGA-FDC model using data fragments from the training set to obtain the trained MGA-FDC model; Step 4: Test the trained MGA-FDC model using data fragments from the test set to obtain the emotion recognition model; Step 5: Input the sample to be identified into the above emotion recognition model to perform EEG signal and emotion recognition of the image sample.

2. The dual-modal emotion recognition method based on multi-granularity attention and feature distribution calibration according to claim 1, characterized in that, The sample set was obtained from the publicly available DEAP and MAHNOB-HCI datasets. Each sample in the sample set contains a face image video and its corresponding EEG signal and emotion category label.

3. The dual-modal emotion recognition method based on multi-granularity attention and feature distribution calibration according to claim 2, characterized in that, The EEG signal feature learning model includes a hierarchically configured first two-dimensional convolutional module, a depthwise separable convolutional module, and a second two-dimensional convolutional module. The specific data processing procedure of this model is as follows: Step 2.1.1: Use the EEG signal samples in the sample set as the input to the first two-dimensional convolutional module, and set the convolutional kernel size to the sampling rate. This allows the model to capture frequency information at 2Hz and above, thus obtaining the pre-processed EEG signal characteristics. Step 2.1.2: Use the pre-processed EEG signal features obtained in Step 2.1.1 as the input features of the depthwise separable convolution module. Use depthwise convolution to capture the spatial relationship of the pre-processed EEG signal features and use separable convolution to capture the temporal features of the pre-processed EEG signal features. By combining depthwise convolution and separable convolution, EEG signal features containing spatiotemporal relationships are obtained. Step 2.1.3: Use the EEG signal features containing spatiotemporal relationships as input to the second 2D convolutional module. Further learn the EEG signal features through 2D convolution to obtain the output features of the EEG modes. .

4. The dual-modal emotion recognition method based on multi-granularity attention and feature distribution calibration according to claim 3, characterized in that, The facial image feature learning model includes a convolutional neural network and a convolutional long short-term memory network. The data processing procedure of this model is as follows: Step 2.2.1: Randomly select 5 frames from any 1s data segment as face image data input to the pre-trained convolutional neural network. The pre-trained convolutional neural network processes each face image sample to obtain the face image features corresponding to each face image sample. Step 2.2.2: The face image features obtained in Step 2.2.1 are sequentially fed into the convolutional long short-term memory network, and then the output features of the convolutional long short-term memory network are flattened to capture the temporal features between each face image frame. Step 2.2.3: Feed the face image features obtained in Step 2.2.2 into the convolutional neural network sequentially, adjust the dimensionality of the output features, and obtain the output features of the face image modality. .

5. The dual-modal emotion recognition method based on multi-granularity attention and feature distribution calibration according to claim 4, characterized in that, The specific processing procedure of the classifier is as follows: The data after passing through the feature distribution calibration module... , The concatenation is performed, and the vector is flattened into a one-dimensional vector. This vector is then fed into a classifier to obtain the constructed MGA-FDC model, as follows: (17) in, Used to indicate splicing operations. This is used to represent a classifier, which consists of three parts: a fully connected layer, a Dropout layer, and an activation function. For the calibrated facial image modal features, The calibrated EEG modal characteristics.

6. The dual-modal emotion recognition method based on multi-granularity attention and feature distribution calibration according to claim 1, characterized in that, The specific process of step 3 is as follows: each data segment sample in the training set is fed into the MGA-FDC model built in step 2, and its predicted sentiment category is output respectively. The predicted sentiment category of each data segment sample is compared with its sentiment category label, and the loss between the two is minimized. The network parameters in the model are updated through the backpropagation algorithm until the model maintains stable sentiment prediction performance on the training set. The model is then saved to obtain the trained model.

7. The dual-modal emotion recognition method based on multi-granularity attention and feature distribution calibration according to claim 1, characterized in that, The specific process of step 4 is as follows: Test the trained model with the test set and save the experimental results of the model on this part of the test set. Test five times in total, ensuring that the data segments used in the test set do not overlap each time. Then take the average of the five test results as the final test result of the model. If the test results meet the expected requirements, save the model to obtain the emotion recognition model. If the test results do not meet the expected requirements, adjust the model architecture and parameters, and repeat steps 1 to 4 until the test results meet the expected requirements, save the model, and obtain the emotion recognition model.

Images