A multimodal signal fusion emotion recognition method based on attention mechanism

Abstract

This paper discloses a multimodal signal fusion emotion recognition method based on an attention mechanism. The method comprises the following steps: first, preprocessing physiological signal data; constructing a channel attention module for multi-channel EEG data; using EEGNet to extract EEG and other physiological signal features separately; introducing a shared-private encoder to decouple shared features between multiple modalities and private features within each modality; and finally, proposing a cross-modal cross-attention fusion mechanism to achieve multimodal feature interaction and effective fusion. By combining multimodal signals, the present invention overcomes the limitations of a single modality by introducing a shared-private feature separation mechanism and a cross-modal cross-fusion mechanism to achieve modal collaborative modeling and effective integration, thereby improving the accuracy and generalization of emotion recognition.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and machine learning technology, and in particular relates to a multimodal signal fusion emotion recognition method based on an attention mechanism. Background Art

[0002] Emotion recognition technology has made significant progress in recent years with the development of artificial intelligence and machine learning. It is widely used in a variety of scenarios, including customer service, healthcare, education, human-computer interaction, and smart hardware. In particular, emotion recognition based on multimodal physiological signals, by combining physiological signals such as EEG, heart rate, and respiration, can more accurately capture and understand an individual's emotional state. Identifying emotions from physiological signals can, to a certain extent, overcome cultural differences and individual differences in expression, thereby providing more objective emotional judgments. The integration of multimodal physiological signals can comprehensively describe the different aspects of emotion, as emotions are complex psychophysiological processes involving multiple external and internal activities, and each modality contains complementary information. However, this technology still faces challenges such as low signal-to-noise ratio and individual differences.

[0003] Machine learning and deep learning are two major technical approaches in emotion recognition, each with unique advantages and applicable scenarios. Machine learning methods rely on manual feature extraction and selection, often requiring domain experts to design features based on prior knowledge. For example, they extract frequency-domain features (such as alpha and beta waves) or time-domain features (such as signal mean and variance) from EEG signals. This approach performs well on small datasets and offers high interpretability, but its performance is highly dependent on the quality of feature engineering and struggles to capture complex nonlinear relationships. With the rapid development of deep learning technology, a growing number of studies are focusing on end-to-end deep learning networks. Deep learning can automatically learn multi-level feature representations from raw data without relying on manually designed features, significantly reducing human intervention. For example, models such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been widely used in emotion recognition tasks and are capable of effectively processing high-dimensional, nonlinear physiological signal data. Furthermore, the attention mechanism in deep learning can adaptively learn key EEG channels related to emotion without relying on prior knowledge, further improving recognition performance and generalization.

[0004] Traditional multimodal fusion methods are divided into: feature-level, decision-level, and hybrid fusion. Feature-level fusion is the most common fusion method. It mainly combines the features after extracting the features of each modality. Although direct concatenation does not cause information loss, it will lead to dimensionality explosion and may cause redundant or conflicting features. Decision-level fusion relies on the design of good rules and is integrated in the later stages of the task, resulting in poor learning flexibility. The main problem that deep learning-based fusion methods solve is that the level of traditional fusion methods is too shallow to deeply capture the complementary and redundant relationships between different modalities. Moreover, these methods focus on the surface and analyze the correlation between modalities. They cannot use deep neural networks to deeply capture the complementary and redundant relationships between multiple modalities. Summary of the Invention

[0005] In response to the above problems, the present invention proposes an attention-based multimodal signal fusion emotion recognition method, which effectively mines the potential relationship between EEG signals and emotions through an attention-based multimodal signal fusion end-to-end recognition network.

[0006] The above purpose is achieved through the following technical solutions:

[0007] A multimodal signal fusion emotion recognition method based on an attention mechanism, the method comprising the following steps:

[0008] S1. Perform unified preprocessing on multimodal physiological signals, including EEG and other modal signals EXG. The preprocessing is to first perform baseline calibration on the multimodal physiological signals, then perform segmentation processing, and finally use min-max normalization operation;

[0009] S2. Build an EEG multi-channel attention module to enhance key channel features through the attention mechanism based on the EEG signal obtained after preprocessing in step S1.

[0010] S3. Based on the outputs of steps S1 and S2, a convolutional neural network is used to construct a local temporal filter, a spatial filter, and a depthwise separable spatiotemporal convolution kernel to extract features from the EEG signal and the other modality signal EXG;

[0011] S4. Extract the feature vectors of EEG and EXG according to step S3, and construct a shared-private encoder to explicitly model the shared features between EEG and EXG and the unique features within the modality;

[0012] S5. Based on step S4, the shared features between EEG and EXG and their respective unique features are obtained. Using a cross-modal cross-attention mechanism, with the shared features as the interaction medium, the private features of the EEG and EXG modalities are guided to conduct multiple rounds of information exchange in the deep structure, fusing the bimodal features.

[0013] S6. A joint learning method is used to train the attention-based multimodal signal fusion network composed of S1-S5. The network uses the Adam optimizer, the initial learning rate is set to 0.001, and the cosine annealing is used to dynamically adjust the learning rate.

[0014] Furthermore, the unified preprocessing of the multimodal physiological signals in step S1 specifically includes the following steps:

[0015] S1.1. Baseline calibration of multimodal physiological signals. Baseline calibration involves subtracting the normal state signal value from the emotional arousal signal value to obtain the change value of emotional arousal.

[0016] S1.2. Segment the EEG and EXG signals separately, selecting a 1-s window with a 25% overlap.

[0017] S1.3. Perform min-max normalization on the EEG and EXG signals, using the following formula:

[0018] ,

[0019] Where, represents the normalized signal, represents the original signal, represents the minimum value of the original signal, Indicates the maximum value of the original signal;

[0020] The preprocessed EEG signal is recorded as , where C is the number of electrodes of EEG signal, is the sampling rate of the EEG signal; the other modal signals EXG after preprocessing are recorded as ,in is the number of electrodes of EXG, is the sampling rate of EXG.

[0021] Furthermore, the EEG multi-channel attention module described in step S2 is constructed as follows:

[0022] S2.1. Perform global average pooling on the time dimension to obtain the global feature of each channel with the shape of (C, 1), where C is the number of EEG electrodes;

[0023] S2.2. Use 1x1 point convolution to transform the global features, enhance the global feature expression ability, and flatten the global features to obtain a length of The channel feature tensor of ;

[0024] S2.3. Calculate the attention weights through the Softmax function, and then expand to obtain the relationship matrix between each sample channel. ;

[0025] S2.4. Constructing the identity matrix ,in is the identity matrix; It is a learnable parameter matrix used to dynamically adjust the relationship between channels and then construct the graph association matrix between channels : ;

[0026] S2.5. Use 1D convolution and ReLU activation functions to perform nonlinear changes on global features and enhance global feature expression.

[0027] S2.6. Convert the correlation matrix Perform matrix multiplication with the global features in S2.1 to obtain a channel attention score vector with the same shape as (C, 1);

[0028] S2.7. Use the sigmoid function to constrain the channel attention score vector between 0 and 1. Finally, implement the graph convolution operation through matrix multiplication to perform weighted fusion of channel features.

[0029] Furthermore, the convolutional neural network described in step S3 has the following specific network settings:

[0030] The first layer is a temporal filter: a 2D convolution with a convolution kernel size of (1, Fs / 2) is used, where Fs represents the sampling rate of the EEG signal. According to the Nyquist-Shannon sampling theorem, the convolution kernel length is half of the data sampling rate, capturing frequency information of 2 Hz and above, and extracting signal components of all available frequencies;

[0031] The second layer is a spatial filter: using a depthwise convolution with a convolution kernel size of (C, 1) through a depthwise convolution operation;

[0032] The third layer is a pooling layer with an average size of (1,4);

[0033] The fourth layer uses batch normalization along the feature map dimension, applies the exponential linear unit (ELU) activation function to accelerate convergence, and uses dropout technology with a probability p of 0.25. Specifically, the neurons in this layer have a 25% probability of losing their activation ability and are thus discarded to effectively control overfitting.

[0034] The fifth layer uses a depthwise separable convolution with a convolution kernel size of (1, Fs / 8) to extract deeper spatiotemporal features;

[0035] The sixth layer performs the same batch normalization, exponential linear unit activation, and dropout techniques as the fourth layer, and finally obtains a feature vector of length 128. The feature vector of EEG is recorded as: , the eigenvector of EXG is recorded as: .

[0036] Furthermore, the specific method of step S4 is as follows:

[0037] S4.1. Based on the modality invariance principle, assuming that multimodal signals have both invariant and unique features, we design three encoders: a shared encoder, an EEG modality-specific encoder, and an EXG modality-specific encoder. These encoders extract the shared invariant features of the EEG and EXG modalities, the unique features of the EEG modality, and the unique features of the EXG modality, respectively. The symbols of the three encoders are: 、 、 ;

[0038] S4.2. Concatenate the feature vectors of the EEG modality and the EXG modality and input them into the shared encoder to obtain the shared features of the modalities. S= E s (concat[ z eeg , z exg ] : θ s ) ,in concat[ z eeg , z exg ] Indicates that the feature vectors of the EEG modality and the EXG modality are spliced ​​together, and the EEG feature vector is input into the EEG modality private encoder to obtain the unique features of the EEG modality Similarly, the feature vector of EXG is input into the private encoder of EXG mode to obtain the unique features of EXG mode ;

[0039] S4.3. Construct the reconstruction task, set up a decoder symmetric to the shared encoder, and use shared features The feature vectors of EEG and EXG modalities are reconstructed, and the reconstruction loss uses the MSE loss function; the MSE loss function The calculation formula is as follows: , where n is the dimension size of y, represents the original features before reconstruction, Represents the reconstruction feature;

[0040] S4.4. In order to distinguish shared features from unique features, an orthogonality constraint loss function is used so that the shared features and unique features of the modality are as orthogonal as possible in the feature space; the orthogonality constraint loss function Calculation: ,in is the matrix norm, and the superscript T indicates the transpose.

[0041] Furthermore, the cross-modal attention mechanism in step S5 adopts the basic architecture of Transformer, obtains the Query vector, Key vector, and Value vector through linear transformation, and then calculates the self-attention score based on the Query and Key, multiplies it with the Value, makes a residual connection with the input, and inputs it into the batch normalization and feedforward neural network. Unlike self-attention, the Query vector, Key vector, and Value vector of cross-modal attention come from different sources, as follows:

[0042] S5.1. Shared features Perform linear transformation to obtain the K matrix and V matrix of the two-stream cross attention module:

[0043] ,

[0044] ,

[0045] ,

[0046] ,

[0047] Where K represents the key vector, V represents the value vector, S represents the shared feature, b represents the bias, W represents the linear transformation matrix, Q represents the query vector, superscript e represents the EEG modality, superscript m represents the EXG modality, subscript i represents different tributaries, subscript 0 represents the EEG tributary, and subscript 1 represents the EXG tributary;

[0048] S5.2. Calculating cross-modal attention weights :

[0049] ,

[0050] in, is the total dimension of the Q tensor;

[0051] S5.3. Multiply the calculated attention score by V to get the output of each layer ;

[0052] S5.4. After batch normalization and feed-forward neural network layers, i.e., two fully connected layers, using ReLU activation functions to keep the shape dimensions unchanged for residual connections;

[0053] S5.5. Each branch has three Transformer blocks, each of which performs the computations in S5.2-S5.4. Two branches are designed for EEG and EXG modalities, respectively. Each branch uses shared features as the interaction medium, guiding the unique features of each modality to undergo multiple rounds of information fusion in the Transformer blocks.

[0054] S5.6. Finally, the output vectors of the two tributaries are concatenated and input into the classification network to perform binary or multi-classification tasks.

[0055] Furthermore, in step S6, the total loss function of the joint learning method is: , in, is the loss function of the emotion classification task, and the hyperparameter and It determines the impact of each regularization term on the overall loss The contribution weight parameters are selected based on the grid search hyperparameter tuning method. = =0.2.

[0056] The beneficial effects of the present invention compared to the prior art are:

[0057] 1. This invention is based on an end-to-end emotion recognition method, which reduces manual intervention, makes full use of the data itself, and has good generalization ability.

[0058] 2. This invention is based on the channel attention mechanism of graph convolutional neural network. By adaptively learning the relationship between channels, it effectively replaces the traditional reliance on expert knowledge to extract useful channels, realizes the accurate extraction of key information, and significantly improves the recognition accuracy of the model.

[0059] 3. Compared with relying on single-modal electroencephalogram (EEG) signals for emotion recognition, the present invention integrates multimodal physiological signals (ECG, EOG, etc.) to integrate complementary information and provide a more comprehensive feature representation for the model, thereby significantly improving recognition accuracy and robustness.

[0060] 4. This paper introduces a shared-private encoder to display shared features between modeling modalities and unique representations within the modality, and uses reconstruction loss and orthogonal constraint loss for learning to achieve coordinated modeling and effective integration between modalities, thereby improving the accuracy and generalization ability of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of the method of the present invention;

[0062] Figure 2 is a network diagram of the channel attention of the present invention;

[0063] Figure 3 is a feature extractor network diagram of the present invention;

[0064] Figure 4 This is the visualization result of the channel attention weights of the method of the present invention on the DEAP dataset;

[0065] Figure 5 It is the visualization result of each evaluation index of the method of the present invention on two data sets. DETAILED DESCRIPTION

[0066] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific implementation examples.

[0067] The present invention proposes a multimodal signal fusion emotion recognition method based on an attention mechanism. By constructing a channel attention based on a graph convolutional neural network, it fuses physiological signals from different modalities, performs modal alignment, uses cross-attention for feature fusion, and finally builds a multi-task learning framework, which effectively improves the recognition accuracy and robustness and solves the overfitting problem caused by a small data set.

[0068] To validate the effectiveness of our method, we selected two publicly available datasets for multimodal emotion recognition: Dreamer and DEAP. DEAP is a multimodal emotion recognition dataset that includes EEG signals and other peripheral physiological signals. Based on physiological signals generated by music videos, this dataset records signals from 40 physiological channels collected by 32 subjects while they watched 40 one-minute music videos. The first 32 channels are used to collect EEG signals, following the international 10-20 system, while channels 33 and 34 are used to record electrooculogram (EOG) signals. In the DEAP dataset, EXG refers to EOG signals. Before each video begins, there is a 3-second silence to record baseline signals. After watching the videos, participants rated their emotions on a scale of 1 to 9 based on valence, arousal, dominance, and liking.

[0069] Dreamer is a multimodal emotion recognition dataset that includes both EEG and ECG signals. This dataset records physiological signals generated by 23 subjects (14 men and 9 women) while watching movie clips. Before each clip begins, there is a 3-second silence period to record baseline signals. Each clip lasts 60 seconds. EEG is recorded using 14 electrodes following the international 10-20 system, while electrocardiogram (ECG) signals are recorded using two electrodes. In the Dreamer dataset, EXG represents the ECG signal. After watching each clip, subjects self-assess and rate valence, arousal, and dominance on a scale of 1-5. The following description focuses on the DEAP dataset.

[0070] like Figure 1 The present invention provides a multimodal signal fusion emotion recognition method based on an attention mechanism, comprising the following steps:

[0071] S1. Preprocess the EEG data. First, perform baseline calibration, then split the EEG data with a 1s window and an overlap rate of 0.25, expand the data set, and finally perform normalization. In the DEAP data set, the sampling rate of EEG is 128Hz and the sampling rate of EXG is 128Hz. After preprocessing, the EEG signal is recorded as , where C is the number of electrodes of EEG signal, is the sampling rate of the EEG signal, where C=32 represents the number of EEG electrodes. =128 represents the sampling rate of EEG; the other modal signals EXG after preprocessing are recorded as: ,in =2 represents the number of electrodes of EXG, 128 represents the sampling rate of EXG;

[0072] S2. Figure 2 Construct a channel attention module. The core operations are as follows:

[0073] S2.1. For the input EEG signal: Perform global average pooling on the time dimension of to obtain the global features of each channel with a shape of (C, 1), where C is the number of EEG electrodes;

[0074] S2.2. Use 1x1 point convolution to transform the global features to enhance the feature expression capability. And flatten the channel features to obtain a length of The channel feature tensor of ;

[0075] S2.3. Calculate the attention weights through the Softmax function, and then expand to obtain the relationship matrix between each sample channel. ;

[0076] S2.4. Constructing the identity matrix ,in is the identity matrix; It is a learnable parameter matrix used to dynamically adjust the relationship between channels and then construct the graph association matrix between channels : ;

[0077] S2.5. Use 1D convolution and ReLU activation functions to perform nonlinear changes on global features and enhance global feature expression.

[0078] S2.6. Convert the correlation matrix Perform matrix multiplication with the global features in S2.1 to obtain a channel attention score vector with the same shape as (C, 1);

[0079] S2.7. Use the sigmoid function to constrain the channel attention score vector between 0 and 1. Finally, implement the graph convolution operation through matrix multiplication to perform weighted fusion of channel features.

[0080] S3. Figure 3 Use feature extractors to extract spatiotemporal features of EEG and EXG respectively. The network settings are as follows:

[0081] The first layer is a temporal filter: a 2D convolution with a kernel size of (1, 63) is used. According to the Nyquist-Shannon sampling theorem, the kernel length is half the data sampling rate, which can capture frequency information of 2 Hz and above. Ideally, it can extract signal components of all available frequencies more completely.

[0082] The second layer is the spatial filter: using the depthwise convolution with a convolution kernel size of (32, 1), the depthwise convolution operation improves the calculation speed on the one hand, and on the other hand, a spatial filter is assigned to each feature map extracted by each temporal filter, thereby effectively extracting frequency-specific spatial features;

[0083] The third layer is a pooling layer with an average size of (1,4). Through the pooling operation, the scale invariance of the features is improved and the feature dimension is reduced;

[0084] The fourth layer uses batch normalization along the feature map dimensions, applies the exponential linear unit (ELU) activation function to accelerate convergence, and employs dropout with a probability p of 0.25. This means that neurons in this layer have a 25% probability of losing their activation ability and are therefore discarded to effectively control overfitting.

[0085] The fifth layer uses a depthwise separable convolution with a kernel size of (1, 16) to extract deeper spatiotemporal features. The depthwise separable convolution consists of depthwise convolution and pointwise convolution, which fuses features from different channels through pointwise convolution.

[0086] The sixth layer performs the same batch normalization, exponential linear unit activation, and dropout techniques as the fourth layer, and finally obtains a feature vector of length 128. The feature vector of EEG is recorded as: , the eigenvector of EXG is recorded as: .

[0087] S4. Based on the feature extraction in step S3, obtain the feature vectors of EEG and EXG. Construct a shared-private encoder to explicitly model the shared features between EEG and EXG and the unique features within the modality. The specific implementation method is as follows:

[0088] S4.1. Based on the modality invariance principle, assuming that multimodal signals have both invariant and unique features, we design three encoders: a shared encoder, an EEG modality-specific encoder, and an EXG modality-specific encoder. These encoders respectively pre-determine the shared (i.e., invariant) features of the two modalities, the unique features of the EEG modality, and the unique features of the EXG modality. The symbols for the three encoders are: 、 、 ;

[0089] S4.2. Concatenate the feature vectors of the two modalities and input them into the shared encoder to obtain the shared features of the modalities: S= E s (concat[ z eeg , z exg ] : θ s ) , the EEG feature vector is input into the EEG modality private encoder to obtain the unique features of the EEG modality: , Similarly, the feature vector of EXG is input into the private encoder of EXG modality to obtain the unique features of EXG modality: ;

[0090] S4.3. In order to ensure that there is no information loss after the encoder, the present invention constructs a reconstruction task, sets a decoder symmetrical to the shared encoder, and uses shared features The feature vectors of EEG and EXG modalities are reconstructed, and the reconstruction loss uses the MSE loss function; the MSE loss function The calculation formula is as follows: , where n is the dimension size of y, represents the original features before reconstruction, Represents the reconstruction feature;

[0091] S4.4. In order to distinguish shared features from unique features, an orthogonal constraint loss function is used to make the shared features and unique features of the modality as orthogonal as possible in the feature space, effectively reducing the interference between the two, and each containing different information. The orthogonality constraint loss function Calculation: ,in is the matrix norm, and the superscript T indicates the transpose.

[0092] S5. Use cross-modal attention to achieve cross-modal feature fusion. Using the Transformer infrastructure, a linear transformation is performed to obtain the query vector, key vector, and value vector. A self-attention score is then calculated based on the query and key, multiplied by the value, and then residually connected to the input before being fed into a batch normalization and feedforward neural network. Unlike self-attention, the query vector, key vector, and value vector of cross-modal attention come from different sources. Specifically, the design is as follows:

[0093] S5.1. Shared features Perform linear transformation to obtain the K matrix and V matrix of the two-stream cross attention module:

[0094] ,

[0095] ,

[0096] ,

[0097] ,

[0098] Where K represents the key vector, V represents the value vector, S represents the shared feature, b represents the bias, W represents the linear transformation matrix, Q represents the query vector, superscript e represents the EEG modality, superscript m represents the EXG modality, subscript i represents different tributaries, subscript 0 represents the EEG tributary, and subscript 1 represents the EXG tributary;

[0099] S5.2. Calculate cross-modal attention weights: ,in, is the total dimension of the Q tensor;

[0100] S5.3. Multiply the calculated attention score by V to get the output of each layer: ;

[0101] S5.4. After batch normalization and feed-forward neural network layers, i.e., two fully connected layers with ReLU activation function, the shape dimension is kept unchanged to facilitate residual connections.

[0102] S5.5. Each branch has three Transformer blocks, each performing the computations in S5.2-S5.4. Two branches are designed for EEG and EXG modalities, with each branch using shared features as the interaction medium to guide the unique features of each modality through multiple rounds of information fusion in the Transformer blocks.

[0103] S5.6. Finally, the output vectors of the two tributaries are concatenated and input into the classification network to perform binary or multi-classification tasks.

[0104] S6. A joint learning approach was used to train the attention-based multimodal signal fusion network composed of S1-S5. The training loss function consisted of a cross-entropy loss function, an orthogonal constraint function, and a reconstruction loss. The total number of epochs per fold was 55, and the learning rate was set to 0.001. The Adam optimizer was used for parameter optimization, and the learning rate was dynamically adjusted using cosine annealing. For each subject, a five-fold cross-validation was performed and the average was taken as the final result for that subject. The results of all subjects were then averaged to serve as the overall performance indicator of the method on this dataset. The accuracy, recall, precision, and F1 score were used as the overall performance indicators of the test model.

[0105] A multimodal signal fusion emotion recognition method based on attention mechanism was used to conduct subject-related experiments on DEAP and DREAMER datasets. A 5-fold cross-validation method was used for each subject. The results are as follows: Figure 5 As shown in the results, the overall score of the DEAP dataset is higher than that of the DREAMER dataset, the error is also smaller, and better results are achieved.

[0106] In order to demonstrate the superiority of the method of the present invention, traditional machine learning methods such as SVM, single-modal recognition methods such as DGCNN and LResCapsule, and multimodal recognition methods such as MMResLSTM, DCCA, and Husformer were used to compare with the method of the present invention on two datasets. The results are shown in Table 1. The accuracy of the method of the present invention on both datasets is significantly better than that of the other six most advanced benchmark models.

[0107] Table 1

[0108]

[0109] In order to verify the effectiveness of the EEG multi-channel attention module, Figure 4 This is the attention weighting of each subject's 32 channels in the arousal classification task on the DEAP dataset. A larger weight coefficient indicates a channel's greater importance to the task. As can be seen from the figure, the weight coefficients for channels P3, P7, O1, and FC5 are significantly higher than those for other channels. These corresponding brain regions are the parietal and frontal lobes, respectively, which play a crucial role in emotional arousal.

Claims

1. A multimodal signal fusion emotion recognition method based on attention mechanism, characterized by: The method comprises the following steps: S1. Perform unified preprocessing on multimodal physiological signals, including EEG and other modal signals EXG. The preprocessing is to first perform baseline calibration on the multimodal physiological signals, then perform segmentation processing, and finally use min-max normalization operation; S2. Build an EEG multi-channel attention module to enhance key channel features through the attention mechanism based on the EEG signal obtained after preprocessing in step S1. S3. Based on the outputs of steps S1 and S2, a convolutional neural network is used to construct a local temporal filter, a spatial filter, and a depthwise separable spatiotemporal convolution kernel to extract features from the EEG signal and the other modality signal EXG; S4. Extract the feature vectors of EEG and EXG according to step S3, and construct a shared-private encoder to explicitly model the shared features between EEG and EXG and the unique features within the modality; S5. Based on step S4, the shared features between EEG and EXG and their respective unique features are obtained. Using a cross-modal cross-attention mechanism, with the shared features as the interaction medium, the private features of the EEG and EXG modalities are guided to conduct multiple rounds of information exchange in the deep structure, fusing the bimodal features. S6. A joint learning approach was used to train the attention-based multimodal signal fusion network composed of S1-S5. The network used the Adam optimizer with an initial learning rate of 0.001 and cosine annealing to dynamically adjust the learning rate. The method for constructing the EEG multi-channel attention module described in step S2 is as follows: S2.

1. Perform global average pooling on the time dimension to obtain the global feature of each channel with the shape of (C, 1), where C is the number of EEG electrodes; S2.

2. Use 1x1 point convolution to transform the global features, enhance the global feature expression ability, and flatten the global features to obtain a length of The channel feature tensor of ; S2.

3. Calculate the attention weights through the Softmax function, and then expand to obtain the relationship matrix between each sample channel. ; S2.

4. Constructing the identity matrix ,in is the identity matrix; It is a learnable parameter matrix used to dynamically adjust the relationship between channels and then construct the graph association matrix between channels : ; S2.

5. Use 1D convolution and ReLU activation functions to perform nonlinear changes on global features and enhance global feature expression; S2.

6. Convert the correlation matrix Perform matrix multiplication with the global features in S2.1 to obtain a channel attention score vector with the same shape as (C, 1); S2.

7. Use the sigmoid function to constrain the channel attention score vector to between 0 and 1. Finally, perform a graph convolution operation through matrix multiplication to perform weighted fusion of channel features. The specific method of step S4 is as follows: S4.

1. Based on the modality invariance principle, assuming that multimodal signals have both invariant and unique features, we design three encoders: a shared encoder, an EEG modality-specific encoder, and an EXG modality-specific encoder. These encoders extract the shared invariant features of the EEG and EXG modalities, the unique features of the EEG modality, and the unique features of the EXG modality, respectively. The symbols of the three encoders are: 、 、 ; S4.

2. Concatenate the feature vectors of the EEG modality and the EXG modality and input them into the shared encoder to obtain the shared features of the modalities. ,in Indicates that the feature vectors of the EEG modality and the EXG modality are spliced ​​together, and the EEG feature vector is input into the EEG modality private encoder to obtain the unique features of the EEG modality Similarly, the feature vector of EXG is input into the private encoder of EXG mode to obtain the unique features of EXG mode ; S4.

3. Construct the reconstruction task, set up a decoder that is symmetric to the shared encoder, and use shared features The feature vectors of EEG and EXG modalities are reconstructed, and the reconstruction loss uses the MSE loss function; the MSE loss function The calculation formula is as follows: , where n is the dimension size of y, represents the original features before reconstruction, Represents the reconstruction feature; S4.

4. In order to distinguish shared features from unique features, an orthogonality constraint loss function is used so that the shared features and unique features of the modality are as orthogonal as possible in the feature space; the orthogonality constraint loss function Calculation: ,in is the matrix norm, and the superscript T indicates the transpose.

2. The method for emotion recognition based on multimodal signal fusion using an attention mechanism according to claim 1, characterized in that: The unified preprocessing of multimodal physiological signals described in step S1 specifically includes the following steps: S1.

1. Baseline calibration of multimodal physiological signals. Baseline calibration involves subtracting the normal state signal value from the emotional arousal signal value to obtain the change value of emotional arousal. S1.

2. Segment the EEG and EXG signals separately, selecting a 1-s window with a 25% overlap. S1.

3. Perform min-max normalization on the EEG and EXG signals, respectively, using the following formula: ， Where, represents the normalized signal, represents the original signal, represents the minimum value of the original signal, Indicates the maximum value of the original signal; The preprocessed EEG signal is recorded as , where C is the number of electrodes of EEG signal, is the sampling rate of the EEG signal; the other modal signals EXG after preprocessing are recorded as ,in is the number of electrodes of EXG, is the sampling rate of EXG.

3. The method for emotion recognition based on multimodal signal fusion using an attention mechanism according to claim 1, wherein: The convolutional neural network described in step S3 has the following specific network settings: The first layer is a temporal filter: a 2D convolution with a convolution kernel size of (1, Fs / 2) is used, where Fs represents the sampling rate of the EEG signal. According to the Nyquist-Shannon sampling theorem, the convolution kernel length is half of the data sampling rate, capturing frequency information of 2 Hz and above, and extracting signal components of all available frequencies; The second layer is a spatial filter: using a depthwise convolution with a kernel size of (C, 1) through a depthwise convolution operation; The third layer is a pooling layer with an average size of (1,4); The fourth layer uses batch normalization along the feature map dimension, applies the exponential linear unit (ELU) activation function to accelerate convergence, and uses dropout technology with a probability p of 0.

25. Specifically, the neurons in this layer have a 25% probability of losing their activation ability and are thus discarded to effectively control overfitting. The fifth layer uses a depth-separable convolution with a kernel size of (1, Fs / 8) to extract deeper spatiotemporal features; The sixth layer performs the same batch normalization, exponential linear unit activation, and dropout techniques as the fourth layer, and finally obtains a feature vector of length 128. The feature vector of EEG is recorded as: , the eigenvector of EXG is recorded as: .

4. The method for emotion recognition based on multimodal signal fusion using an attention mechanism according to claim 1, wherein: The cross-modal attention mechanism in step S5 uses the Transformer infrastructure to obtain the Query vector, Key vector, and Value vector through linear transformation. Then, the self-attention score is calculated based on the Query and Key, multiplied by the Value, and connected to the input with a residual connection. The result is input into the batch normalization and feedforward neural network. Unlike self-attention, the Query vector, Key vector, and Value vector of cross-modal attention come from different sources, as follows: S5.

1. Shared features Perform linear transformation to obtain the K matrix and V matrix of the two-stream cross attention module: ， ， ， ， Where K represents the key vector, V represents the value vector, S represents the shared feature, b represents the bias, W represents the linear transformation matrix, Q represents the query vector, superscript e represents the EEG modality, superscript m represents the EXG modality, subscript i represents different tributaries, subscript 0 represents the EEG tributary, and subscript 1 represents the EXG tributary; S5.

2. Calculating cross-modal attention weights : ， in, is the total dimension of the Q tensor; S5.

3. Multiply the calculated attention score by V to get the output of each layer ; S5.

4. After batch normalization and feed-forward neural network layers, i.e., two fully connected layers, using ReLU activation functions to keep the shape dimensions unchanged for residual connections; S5.

5. Each branch has three Transformer blocks, each of which performs the computations in S5.2-S5.

4. Two branches are designed for EEG and EXG modalities, respectively. Each branch uses shared features as the interaction medium, guiding the unique features of each modality to undergo multiple rounds of information fusion in the Transformer blocks. S5.

6. Finally, the output vectors of the two tributaries are concatenated and input into the classification network to perform binary or multi-classification tasks.

5. The method for emotion recognition based on multimodal signal fusion using an attention mechanism according to claim 1, wherein: Step S6. The total loss function of the joint learning method training is: ,in, is the loss function of the emotion classification task, and the hyperparameter and It determines the impact of each regularization term on the overall loss The contribution weight parameters are selected based on the grid search hyperparameter tuning method. = =0.2.

Images