Multi-modal emotion recognition method and system based on cross-modal attention mechanism

Abstract

The invention provides a multi-modal emotion recognition method and system based on a cross-modal attention mechanism, and the method comprises the steps: 1, carrying out the preprocessing of an EEG signal and an fMRI signal which are collected synchronously, and obtaining a preprocessed one-dimensional time sequence signal; 2, converting the preprocessed one-dimensional time sequence signal into a multi-dimensional data structure more suitable for deep learning model processing; 3, a dual-channel parallel hybrid fusion network model is constructed, emotion recognition based on the hybrid fusion network model is carried out, and the hybrid fusion network model comprises a cross-modal feature layer fusion module and a multi-modal decision-making layer fusion module; and step 4, features from the self-cross-modal feature layer fusion module and the multi-modal decision-making layer fusion module are spliced, and a classifier is adopted to obtain an emotion judgment result. By means of the scheme, the emotion state can be analyzed from multiple angles, and the accuracy of emotion recognition is improved by integrating the electroencephalogram signals and the fMRI signals.

Description

Technical Field

[0001] This invention relates to the field of biomedical engineering technology, and in particular to a multimodal emotion recognition method and system based on a cross-modal attention mechanism. Background Technology

[0002] With the rapid development of affective computing and brain-computer interface technologies, emotion recognition based on physiological signals has become a core enabling technology for scenarios such as smart healthcare, intelligent cockpits, and virtual reality interaction. Currently, common physiological modalities mainly include electroencephalography (EEG) and functional magnetic resonance imaging (fMRI).

[0003] EEG records potential changes generated by the collective firing of neurons using a high-density electrode array on the scalp. It has millisecond-level temporal resolution and can capture the dynamic neural oscillation characteristics induced by emotions in real time. However, its spatial resolution is only at the centimeter level, making it difficult to accurately locate emotion-related brain regions.

[0004] fMRI utilizes blood oxygen level-dependent (BOLD) signals to reflect metabolic activity in brain regions, achieving spatial resolution down to the millimeter level. It can clearly visualize the spatial distribution of emotional processing networks. However, its temporal resolution is limited by hemodynamic response functions, making it unable to track rapid emotional fluctuations. Therefore, the single-modal approach suffers from a "spatiotemporal trade-off" bottleneck.

[0005] EEG-only models are prone to misjudgment in dynamic emotion detection tasks due to insufficient spatial information;

[0006] While fMRI-only models can characterize emotional networks, they struggle to capture transient emotional state changes and are limited by expensive equipment and specific scenarios.

[0007] Existing technologies introduce a "multimodal physiological signal emotion recognition" framework, fusing EEG and fMRI to achieve both high temporal and spatial accuracy. However, this fusion method has the following main drawbacks:

[0008] Early feature-level stitching directly concatenates the EEG band energy map with the fMRI activation map, ignoring the differences in sampling rates and physical meanings between the two modalities, resulting in high-dimensional redundant features drowning out complementary information;

[0009] Late fusion voting simply weights the independent classification results of two modalities, fails to uncover cross-modal interactions, and the weights are set based on experience, lacking adaptability.

[0010] Existing deep learning models generally use CNN, LSTM or graph convolutional networks to process EEG and fMRI respectively, but they lack an explicit "cross-modal attention" mechanism, making it difficult to align the channel-band features of EEG with the voxel-time features of fMRI, resulting in unstable fusion results.

[0011] The model is severely black-boxed and cannot explain the specific contributions of different frequency bands (Delta, Theta, Alpha, Beta, Gamma) or brain regions (frontal lobe, temporal lobe, cingulate gyrus, etc.) to emotion recognition, which reduces the credibility of clinical and commercial applications.

[0012] The large number of algorithm parameters and high computational complexity make it difficult to deploy in real time on wearable or edge devices, which limits its widespread use in scenarios such as in-vehicle emotion monitoring and mobile mental health screening.

[0013] To address this, we have invented a novel multimodal emotion recognition method and system based on a cross-modal attention mechanism, which solves the aforementioned technical problems. Summary of the Invention

[0014] The purpose of this invention is to provide a multimodal emotion recognition method and system based on cross-modal attention mechanism that can fully utilize the advantages of EEG's high temporal resolution and fMRI's high spatial resolution, explicitly model cross-modal interaction, and is both interpretable and lightweight, in order to overcome the problems of spatiotemporal trade-offs, information redundancy, black box uninterpretability, and deployment difficulties in existing technologies.

[0015] The objective of this invention can be achieved through the following technical measures: a multimodal emotion recognition method based on a cross-modal attention mechanism, the method comprising:

[0016] Step 1: Preprocess the synchronously acquired EEG and fMRI signals to obtain a preprocessed one-dimensional time-series signal;

[0017] Step 2: Convert the preprocessed one-dimensional time series signal into a multi-dimensional data structure that is more suitable for deep learning model processing;

[0018] Step 3: Construct a dual-channel parallel hybrid fusion network model and perform emotion recognition based on the hybrid fusion network model. The hybrid fusion network model includes a cross-modal feature layer fusion module and a multimodal decision layer fusion module.

[0019] Step 4: The features from the cross-modal feature layer fusion module and the multimodal decision layer fusion module are concatenated, and a classifier is used to obtain the emotion judgment result.

[0020] In step 1, the preprocessing of the EEG signal includes:

[0021] Step 1a: For the problematic lead signal, reconstruct the bad channel signal and take the average of the EEG data around the bad channel as the reconstruction value of the bad channel.

[0022] Step 2a: Use a bandpass filter to remove high-frequency noise;

[0023] Step 3a: Reduce the sampling rate and the amount of data to improve the speed of data processing;

[0024] Step 4a: Set an electrode point with a known potential as a reference electrode to reduce the influence of noise on the signal;

[0025] Step 5a: Extract the time points when emotions are induced during the signal acquisition process, and perform data segmentation and baseline correction;

[0026] Step 6a: ICA artifact identification is performed on the EEG signal. The time and frequency characteristics of each component of ICA are observed. The artifact components of electrooculography and electrocardiography are removed, and the remaining ICA components are used to reconstruct the EEG signal.

[0027] In step 1, the preprocessing of the fMRI signal includes:

[0028] Step 1b: Remove non-stationary data, specifically the image frame data acquired in the first few seconds;

[0029] Step 2b: Perform layer time correction, determine the reference layer, calculate the time offset of other layers relative to the reference layer, and resample according to the obtained time offset to correct the time point consistency.

[0030] Step 3b: Perform head motion correction by registering brain images at different scanning time points to reference frames through rigid body changes. The reference frames are selected as the first or middle frames in the sequence, and image frames with large head motion ranges are removed.

[0031] Step 4b involves image registration. First, a high-resolution MRI structural image is selected as the reference image. The low-resolution fMRI image is registered to the selected reference structural image using an image transformation method that employs radiometric transformation or non-rigid transformation. Then, the T1 structural image data is registered to the standard space using radiometric transformation. Finally, the two transformation matrices are combined and applied to the fMRI data to register it to the standard space.

[0032] Step 5b involves spatial smoothing to reduce noise in the fMRI data.

[0033] In step 2, the one-dimensional time-series signal includes a one-dimensional EEG time-series signal and a one-dimensional fMRI time-series signal. Based on the spatial distribution map of the EEG electrodes on the scalp, the one-dimensional EEG time-series signal is mapped into a two-dimensional grid matrix. Then, based on the different emotional expression characteristics in different frequency bands, the two-dimensional grid matrix is ​​expanded into a three-dimensional EEG data sequence containing spatial, temporal, and frequency band information. The blood oxygenation level-dependent signal of specific brain regions related to the emotion task is extracted from the one-dimensional fMRI time-series signal to form fMRI multi-channel time-series data.

[0034] In step 3, the cross-modal feature layer fusion module first inputs the EEG 3D data sequence and the fMRI multi-channel temporal data into a gated recurrent unit (GRU) network to effectively capture the temporal dependencies within each modality and outputs EEG feature sequences and fMRI feature sequences containing temporal information. Then, the EEG feature sequence is used as a query for the target modality, and the fMRI feature sequence is used as a key and value for the source modality. The attention weight of the fMRI features on the EEG features is calculated to generate a first fusion feature, representing how fMRI information supplements and enhances the EEG features. Next, the fMRI feature sequence is used as a query for the target modality, and the EEG feature sequence is used as a key and value for the source modality. The attention weight of the EEG features on the fMRI features is calculated to generate a second fusion feature, representing how EEG information supplements and enhances the fMRI features. Finally, the first fusion feature and the second fusion feature are concatenated to obtain a deep fusion feature, which is then passed through a global average pooling layer and a fully connected layer to obtain the emotion classification result of the cross-modal feature layer fusion module.

[0035] In step 3, the multimodal decision layer fusion module constructs two independent Transformer-based models: the EEG-Transformer model and the fMRI-Transformer model. The EEG 3D data sequence is input into the EEG-Transformer model for deep feature extraction and emotion classification, yielding EEG deep features and EGG decision output results. The fMRI multi-channel time-series data is input into the fMRI-Transformer model for deep feature extraction and emotion classification, yielding fMRI deep features and fMRI decision output results. The EGG decision output results and the fMRI decision output results are then weighted and fused using a fusion strategy to obtain the fused emotion classification result of the multimodal decision layer fusion module.

[0036] In step 4, the deep fusion features output by the cross-modal feature layer fusion module are concatenated with the EEG deep features and the fMRI deep features output by the multimodal decision layer fusion module to form a combined feature vector containing feature layer interaction information and single-modal independent information; the combined feature vector is input into one or more fully connected layers, and the emotion determination result is obtained through the Softmax activation function.

[0037] The objective of this invention can also be achieved through the following technical measures: a multimodal emotion recognition system based on a cross-modal attention mechanism, the system comprising:

[0038] The data preprocessing unit preprocesses the synchronously acquired EEG and fMRI signals to obtain preprocessed one-dimensional time-series signals.

[0039] The multidimensional data construction unit converts the preprocessed one-dimensional time-series signal into a multidimensional data structure that is more suitable for deep learning models.

[0040] The feature extraction unit constructs a dual-channel parallel hybrid fusion network model to perform emotion recognition based on the hybrid fusion network model. The hybrid fusion network model includes a cross-modal feature layer fusion module and a multimodal decision layer fusion module.

[0041] The emotion recognition unit concatenates features from the cross-modal feature layer fusion module and the multimodal decision layer fusion module, and uses a classifier to obtain the emotion determination result.

[0042] The cross-modal feature layer fusion module first inputs the EEG 3D data sequence and the fMRI multi-channel temporal data into a gated recurrent unit (GRU) network to effectively capture the temporal dependencies within each modality and outputs EEG feature sequences and fMRI feature sequences containing temporal information. Then, using the EEG feature sequence as a query for the target modality and the fMRI feature sequence as a key and value for the source modality, the attention weight of the fMRI features on the EEG features is calculated to generate a first fusion feature, characterizing how fMRI information supplements and enhances the EEG features. Next, using the fMRI feature sequence as a query for the target modality and the EEG feature sequence as a key and value for the source modality, the attention weight of the EEG features on the fMRI features is calculated to generate a second fusion feature, characterizing how EEG information supplements and enhances the fMRI features. Finally, the first and second fusion features are concatenated to obtain a deep fusion feature, which is then passed through a global average pooling layer and a fully connected layer to obtain the emotion classification result of the cross-modal feature layer fusion module.

[0043] The multimodal decision-making layer fusion module constructs two independent Transformer-based models: the EEG-Transformer model and the fMRI-Transformer model. Preprocessed EEG time-series data is input into the EEG-Transformer model for deep feature extraction and emotion classification, yielding EEG deep features and EEG decision outputs. Similarly, fMRI time-series data is input into the fMRI-Transformer model for deep feature extraction and emotion classification, yielding fMRI deep features and fMRI decision outputs. The EEG and fMRI decision outputs are then weighted and fused using a fusion strategy to obtain the fused emotion classification result from the multimodal decision-making layer fusion module.

[0044] This invention presents a multimodal emotion recognition method and system based on a cross-modal attention mechanism, belonging to the branch of affective computing and neural information processing in the field of biomedical engineering technology. It fuses EEG signals from a cross-modal attention mechanism with fMRI signals for multimodal emotion recognition, and can be engineered for use in wearable devices, smart cockpits, virtual reality interaction, and remote mental health monitoring. This invention also covers key aspects such as signal preprocessing, spatial-frequency three-dimensional data construction, deep neural network architecture design, hybrid-level fusion strategies, interpretability analysis, and integrated hardware and software deployment. It targets applications in various scenarios including smart healthcare, brain-computer interfaces, emotion disorder screening, and human-computer interaction. It overcomes the limitations of single-modal physiological signal expression in conveying emotional information and the difficulty in recognizing dynamically changing emotions. By using multimodal emotion research, it compensates for the deficiencies of single-modal information sources, facilitating the analysis of emotional states from multiple perspectives, and improving the accuracy of emotion recognition by integrating EEG and fMRI signals. This invention also verifies the performance of the designed model by comparing it with several common algorithms; ablation experiments further verify the effectiveness of the hybrid fusion approach, demonstrating that the cross-modal attention mechanism can capture useful information between different modalities. These findings not only lay a solid theoretical foundation for future multimodal emotion research but also provide practical pathways for real-world applications. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a multimodal emotion recognition method based on a cross-modal attention mechanism in a specific embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram of the EEG signal preprocessing flow in a specific embodiment of the present invention;

[0047] Figure 3 This is a flowchart of fMRI data preprocessing in a specific embodiment of the present invention;

[0048] Figure 4 This is a structural diagram of a cross-modal feature layer fusion module in a specific embodiment of the present invention;

[0049] Figure 5 This is a diagram of the internal structure of the GRU model in a specific embodiment of the present invention;

[0050] Figure 6 This is a structural diagram of a cross-modal attention module in a specific embodiment of the present invention;

[0051] Figure 7 This is a schematic diagram of a multimodal decision-making layer fusion mechanism in a specific embodiment of the present invention;

[0052] Figure 8 This is a flowchart of the algorithm model of a multimodal fusion recognition model based on spatial correlation in a specific embodiment of the present invention;

[0053] Figure 9 This is a model training accuracy curve in a specific embodiment of the present invention;

[0054] Figure 10 This is a model training loss curve in a specific embodiment of the present invention;

[0055] Figure 11 This is a schematic diagram of a multi-class confusion matrix in a specific embodiment of the present invention;

[0056] Figure 12 The flowchart below shows the multimodal emotion recognition method based on cross-modal attention mechanism of the present invention.

[0057] Figure 13 This is a structural diagram of the multimodal emotion recognition system based on cross-modal attention mechanism of the present invention. Detailed Implementation

[0058] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0059] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, and / or combinations thereof.

[0060] fMRI data has high spatial resolution but low temporal resolution. Conversely, EEG data has high temporal resolution but lower spatial resolution compared to fMRI. For emotion classification tasks, both temporal and spatial information of brain activity are crucial. Using only single-modality data prevents these methods from fully utilizing the rich temporal and spatial information, thus limiting their ability to improve learning performance. Therefore, using data from both modalities simultaneously can synthesize the temporal and spatial information of brain activity. Updating parameters through deep learning neural network training incentivizes the network to model the interaction patterns of multimodal information, enabling the network to adaptively fuse multimodal data. Therefore, this invention uses deep learning to learn features from both fMRI and EEG modalities, helping to alleviate the problem that single-modality data cannot provide sufficient information, thereby further improving the accuracy of dynamic brain effect connectivity learning.

[0061] This invention achieves deep fusion and efficient recognition of EEG and fMRI signals by constructing a dual-channel parallel network architecture that includes feature layer fusion and decision layer fusion. Figure 12 As shown, Figure 12 This is a flowchart of the multimodal emotion recognition method based on a cross-modal attention mechanism according to the present invention. The multimodal emotion recognition method based on a cross-modal attention mechanism includes:

[0062] Step 101: Preprocessing of multimodal physiological signals. The synchronously acquired EEG and fMRI signals are preprocessed to obtain preprocessed one-dimensional time-series signals.

[0063] The preprocessing steps for EEG signals include, but are not limited to, denoising (such as filtering), removing baseline drift, and normalization, in order to eliminate interference from environmental and physiological artifacts.

[0064] Preprocessing of fMRI signals includes, but is not limited to, time-to-time correction, head motion correction, spatial normalization, and smoothing, in order to improve signal quality and comparability among different subjects.

[0065] Step 102: Construction of multi-channel and multi-dimensional datasets, which transforms the preprocessed one-dimensional time series signal into a multi-dimensional data structure more suitable for deep learning model processing.

[0066] The preprocessed one-dimensional time-series multimodal signals are converted into a multi-dimensional data structure more suitable for deep learning models, in preparation for the construction of multi-channel and multi-dimensional datasets. Based on the spatial distribution map of EEG electrodes on the scalp, the one-dimensional multi-channel EEG time-series data is mapped into a two-dimensional grid matrix. Combining features of different frequency bands, the two-dimensional grid matrix is ​​expanded into a three-dimensional EEG data sequence containing spatial, temporal, and frequency band information. Blood oxygen level dependent (BOLD) signals from specific brain regions related to the emotion task are extracted from fMRI signals to form fMRI multi-channel time-series data.

[0067] Step 103: Based on the emotion recognition of the hybrid fusion model, construct a dual-channel parallel hybrid fusion network model.

[0068] The multimodal dataset is input into a dual-channel parallel hybrid fusion network model, and the output is the sentiment classification result. This model includes a cross-modal feature layer fusion module and a multimodal decision layer fusion module.

[0069] The cross-modal feature layer fusion module aims to deeply mine and fuse the interactive information of EEG and fMRI at the feature level. The EEG 3D data sequence and the fMRI multi-channel temporal data are respectively input into a gated recurrent unit (GRU) network to effectively capture the temporal dependencies within each modality and output feature sequences S_E (EEG features) and S_F (fMRI features) containing temporal information.

[0070] The Cross-Modal Attention (CMA) fusion module, also known as the cross-modal feature layer fusion module, is designed with two CMA units connected in parallel.

[0071] The first CMA unit: Using S_E as the query of the target modality and S_F as the key and value of the source modality, it calculates the attention weight of fMRI features on EEG features and generates the fusion feature A_E←F, which characterizes how fMRI information supplements and enhances EEG features;

[0072] The second CMA unit uses S_F as the query for the target modality and S_E as the key and value for the source modality. It calculates the attention weights of EEG features on fMRI features and generates the fusion feature A_F←E, which characterizes how EEG information supplements and enhances fMRI features.

[0073] The feature integration and output module concatenates the fused features A_E←F and A_F←E output by the two CMA units, and then passes them through a global average pooling layer and a fully connected layer to obtain the sentiment classification result of the module.

[0074] The multimodal decision-making fusion module, as a parallel branch, retains the independent recognition capabilities of each modality and performs high-level information fusion. Independent Transformer-based models are constructed separately. Preprocessed EEG signals are input into the EEG-Transformer model, and fMRI signals are input into the fMRI-Transformer model, where deep feature extraction and emotion classification are performed respectively, yielding their respective decision outputs.

[0075] Step 104: Final feature concatenation and emotion classification. Integrate deep interactive features from the feature layer fusion module and independent modal features from the decision layer fusion module to form a more comprehensive and robust combined feature representation, which is then handed over to the final classifier to complete the emotion determination.

[0076] The decision results output by the two Transformer models are weighted and summed or other decision fusion strategies are used to obtain the fused sentiment classification result of this module. The deep fusion features output by the cross-modal feature layer fusion module before the fully connected layer are concatenated with the single-modal deep features extracted by the two Transformer models in the multimodal decision layer fusion module to form a final combined feature vector containing feature layer interaction information and single-modal independent information.

[0077] The combined feature vector is input into one or more fully connected layers, and finally passed through a Softmax activation function to output the final emotion category with the highest probability as the recognition result.

[0078] In a specific embodiment of the present invention, a hybrid fusion method was studied to address how to more efficiently select features with strong emotional state representation capabilities from single-modal EEG and fMRI signals, how to optimize the fusion of these two modal emotional features, and how to maximize the mining of cross-modal interaction information. The method involves three steps: first, utilizing the independence between features of each modality and selecting the optimal model for each modality based on its characteristics; second, further analyzing cross-modal feature information; and third, fusing the results of these two parts to output the final classification result. The hybrid fusion algorithm includes feature-level fusion and decision-level fusion, utilizing both the independence of single-modality and the complementarity between multimodalities, combining the advantages of both to further improve the generalization and robustness of the emotion recognition model. Specifically, as shown... Figure 1 As shown, the multimodal emotion recognition method based on cross-modal attention mechanism in this embodiment includes the following steps:

[0079] Step 1, data preprocessing: preprocess the EEG and fMRI signals separately, including noise reduction, baseline drift removal, filtering and frequency division, and standardization, to eliminate the adverse effects of physical factors as much as possible.

[0080] Electroencephalogram (EEG) signals are collected by wearing a non-invasive EEG cap on the scalp. These signals are highly random and weak. Because they are easily contaminated by irrelevant noise or other physiological activities during the collection process, various artifacts are formed, including ECG artifacts, EMG artifacts, and EEG artifacts.

[0081] Independent Component Analysis (ICA) is currently considered a highly effective algorithm for identifying and removing artifact signals. Its core idea is to maximize independence by finding a separation matrix W that decomposes the observed signal into multiple independent component signals. Common EEG signal preprocessing procedures include... Figure 2 As shown. The specific steps of preprocessing are as follows:

[0082] (1a) Remove bad channels / average of surrounding areas. During the early stage of EEG signal acquisition, there may be some poor contact between individual electrodes and the scalp, resulting in abnormal signals in this part. Also, remove the electrooculography electrodes "HEO" and "VEO" and the reference electrodes "M1" and "M2". For problematic lead signals, in order to ensure the integrity of the EEG signal, it is necessary to reconstruct the bad channel signal and take the average of the EEG data around it as the reconstruction value of the bad channel.

[0083] (2a) Bandpass filtering: The main frequency range of EEG signals is about 0.5-50 Hz. Bandpass filtering means that only signals in this frequency range can pass through. Using a bandpass filter can effectively remove high-frequency noise. Generally, Butterworth filters are used to process the signals.

[0084] (3a) Downsampling: By reducing the sampling rate, the amount of data is reduced, thereby increasing the speed of data processing.

[0085] (4a) Re-reference: EEG reflects the potential difference between two electrodes, and this value is too small. By setting an electrode point with a known potential as a reference electrode, the influence of noise on the signal can be reduced. Generally, a position with a potential of 0 or no change is selected, such as bilateral mastoid processes.

[0086] (5a) Data segmentation / baseline correction: Based on the experimental paradigm design, the time points during signal acquisition that evoked emotions were extracted, while other time points were considered irrelevant. After data segmentation, baseline correction was required to avoid the influence of resting-state EEG and eliminate baseline drift.

[0087] (6a) ICA algorithm / artifact removal: ICA artifact recognition is performed on the EEG signal using the EEGLAB toolbox in MATLAB. The time and frequency characteristics of each component of ICA are observed, artifact components such as electrooculogram and electrocardiogram are removed, and finally the remaining ICA components are used to reconstruct the EEG signal, thus obtaining a clean EEG signal.

[0088] fMRI data is a crucial tool for studying brain function and neural network connectivity, measuring the spontaneous fluctuations of the Boltzmann-Oxley (BOLD) signal associated with neuronal activity. During imaging, due to the natural physiological activities of the participants and the influence of the scanning equipment, the obtained BOLD signal inevitably contains not only components reflecting neuronal activity but also elements not contributed by neurons. Therefore, rigorous preprocessing of fMRI data is essential to ensure the accuracy and reliability of the obtained data. The preprocessing steps are as follows: Figure 3 As shown. Specifically includes:

[0089] (1b) Remove non-steady-state data. When a magnetic field is applied to a brain region, hydrogen molecules align themselves along the direction of the magnetic field, and these molecules take 5 to 6 seconds to reach a stable state. Therefore, image frame data acquired in the first few seconds (typically 10 seconds) must be removed before analysis.

[0090] (2b) Inter-slice temporal correction: During fMRI data acquisition, MRI scans are performed slice by slice, and the data acquisition time for each slice varies slightly, resulting in time delays between different slices. This temporal inconsistency can affect data analysis, thus requiring slice temporal correction. Slice temporal correction involves determining a reference slice, calculating the time offset of other slices relative to the reference slice, and resampling based on the obtained time offsets to correct for time point consistency.

[0091] (3b) Head motion correction: Brain images at different scan times are registered to a reference frame using rigid body transformation. The reference frame is usually the first or middle frame in the sequence. Furthermore, after head motion correction, image frames with large head motion ranges should be removed. When using rigid body transformation, three translation parameters (in millimeters) and three rotation parameters are calculated. These six motion parameters are used to calculate the head movement deviation (FD), which is a measure of the degree of head movement. Generally, image frames with an FD exceeding 0.5 mm should be removed.

[0092] (4b) Image registration: Image registration is used to align MRI images scanned during the same subject's participation in the task, ensuring that all image sequences have consistent geometric shape and positional information in space. First, a high-resolution MRI (magnetic resonance imaging) structural image is selected as a reference image. The low-resolution fMRI image is registered to the selected reference structural image using radiometric transformation or non-rigid transformation image transformation. Then, T1 structural image data is registered to the standard space using radiometric transformation. T1 structure refers to MRI images obtained through T1-weighted scanning sequences. Finally, the two transformation matrices are combined and applied to the fMRI data to register it to the standard space.

[0093] (5b) Spatial smoothing is an important preprocessing step used to reduce noise in fMRI data. In practical studies, the most commonly used method is Gaussian smoothing. As a linear filtering method, it uses a Gaussian kernel to calculate the weighted average of adjacent voxels to replace the existing values, thereby reducing noise and enhancing the MRI signal so that activation regions can be detected more accurately in subsequent analysis.

[0094] Step 2, constructing a multi-channel dataset: After preprocessing, in order to fully consider the spatial and temporal correlations of EEG and fMRI signals in different modalities, these one-dimensional temporal data are converted into a two-dimensional grid matrix based on the location of the EEG electrodes of the sensors. Then, based on the different emotional characteristics in different frequency bands, the two-dimensional grid matrix is ​​converted into a three-dimensional data structure containing space and frequency bands, considering the correlation between channels while studying the correlations in different spaces and temporal sequences.

[0095] Step 3 involves embedding a parallel spatial-frequency attention mechanism, designed to adaptively capture valuable spatial and frequency information. This is achieved using a cross-modal feature layer fusion module and a multimodal decision layer fusion method.

[0096] Different modalities have varying degrees of influence on emotion classification, and information from different modalities at the same stage also exhibits certain correlations and complementarities. To further capture these potential values, a cross-modal multi-head attention mechanism (hereinafter referred to as cross-modal attention) is integrated into the cross-modal feature fusion module. Compared to self-attention mechanisms, multi-head attention mechanisms emphasize using the same attention mechanism to learn features from different parts, and then combining these different features as knowledge. By automatically capturing temporal interaction features between physiological signals through gated recurrent units (GRUs) and cross-modal attention, the feature layer fusion recognition effect is improved, such as... Figure 4 As shown.

[0097] like Figure 5 The image shows a specific GRU model, obtained by changing the dimensions of the input. Where C represents the number of channels, and T represents the sample length of each channel. As the input to the gated cyclic unit, it is used to capture the time-series signal of the signal, where R represents a C×T dimensional real matrix.

[0098] After inputting into the GRU layer, the following calculation formula is obtained. First, the reset gate is obtained based on the information passed down from the previous state. and Update Gate The calculation formula is as follows:

[0099]

[0100]

[0101] in, and These correspond to the weight parameters of the two gates from the input to the hidden layer. , The weights from the hidden layer to the output layer corresponding to the two gates. b represents the activation function. r To reset the gate's bias term, increasing the model's expressive power, it forms a complete gating computation unit together with the weight matrix, controlling the changes in information over time. (Reset gate) Its function is to control the candidate state Does it depend on the previous moment? , update door Its function is to control how much information the current state retains from historical information and to determine the candidate states. The amount of new information received. Candidate states. Calculation formula:

[0102]

[0103] In the formula, h t It is the candidate hidden state at the current time step, and has the same dimension as the hidden state;

[0104] tanh is the hyperbolic tangent activation function, which compresses the output value to the range [-1, 1].

[0105] W xh It is the weight matrix input to the hidden state, which will change the current input x. t Mapped to the hidden state space;

[0106] x t This is the input for the current time step;

[0107] W hh It is the weight matrix from hidden state to hidden state, which processes the information of the previous hidden state;

[0108] r t It is a reset gate unit that controls the previous state h. t-1 How much information needs to be "forgotten"?

[0109] h t-1 It is the hidden state of the previous time step;

[0110] b h It is the bias term of the hidden state, a learnable bias parameter.

[0111] This formula calculates the candidate hidden state, the possible new memory state under the current input and reset gate control. This candidate state is then compared with the update gate z. t Combined, they form the final hidden state.

[0112] In summary, network status updates, The closer it is to 1, the more it means At this point, the state is equal to the state at the previous time step; less data has been forgotten, and the correlation with information from the previous time step is strong; conversely, the same applies. The specific calculation formula is as follows:

[0113]

[0114] in, This indicates that you are choosing to remember the data from the previous moment. It indicates that information at the current moment has been forgotten.

[0115] The cross-modal feature fusion module utilizes GRU to effectively capture temporal relationship features of signals and aggregate information, while mitigating issues such as vanishing or exploding gradients. A cross-modal attention mechanism further captures the interaction information between physiological signals, enhancing the representation of related features in one modality through another. The fused features are then sequentially processed through a global average pooling layer and a fully connected layer, finally outputting the emotion recognition result via a Softmax function.

[0116] Cross-modal attention (CMA) enables information alignment and interaction between different modalities. In this module, S_e is a pre-processed EEG feature representation, containing emotion-related spatiotemporal features extracted from the raw EEG signal; S_p is a pre-processed fMRI feature representation, containing emotion-related brain region activation features extracted from the BOLD signal. Cross-modal information interaction is achieved by adapting the source modality to the target modality, where the source modality is... and The target mode is To avoid missing modal information, a parallel source mode and target mode combination method is used to construct two parallel CMA modules. The first CMA module outputs... With the output of the second CMA module Both represent common features between EEG and fMRI signals, but the specific characteristics they represent are not entirely the same. This represents the emotional features that are of interest in the fMRI signal P (source modality) and the corresponding EEG signal E (target modality). This indicates important emotional features obtained by adapting the electroencephalogram (EEG) signal E (source modality) to the fMRI signal P (target modality). For example... Figure 6As shown, the output features of parallel cross-modal attention are concatenated and fused, and global average pooling is used to further filter features. Different weights of emotion features are adaptively learned through the cross-modal attention mechanism. The output features of two CMA units are fused through concatenation to form a complementary cross-modal emotion representation. The fused features are then passed through a global average pooling layer for feature compression and filtering, and a fully connected layer is used to map the feature space. Finally, the emotion classification probability distribution is output through a Softmax function.

[0117] The multimodal decision layer fusion module uses Transformer-based single-modal emotion recognition as a parallel branch, fully considering the effective emotion recognition of different modalities.

[0118] In decision-level fusion, features are extracted from spatially correlated data across multiple modalities. Multiple features from each channel separated from two modalities are then used for emotion classification and recognition within their respective network models, yielding emotion recognition results for each modality. These results are then weighted and summed to obtain the final decision fusion result. The principle and mechanism of multimodal physiological signal emotion recognition in decision-level fusion are as follows: Figure 7 As shown.

[0119] This invention preprocesses two physiological signals separately, including three steps: filtering and denoising, eliminating baseline drift, and standardization. Next, it performs multi-dimensional feature extraction and constructs an effective feature set. Then, it conducts simulation experiments on a single modality, selects the most suitable classifier based on the characteristics of each modality, and finally weights and fuses the recognition results of the two single modalities to obtain the decision-level fusion recognition effect.

[0120] Step 4, Feature Fusion and Recognition: After extracting local features between channels and temporal features within channels through a dual-channel parallel network, Concat is used to fuse the two high-quality and high-importance features. Finally, a fully connected network and the Softmax function are used to achieve emotion classification.

[0121] In one embodiment, experimental tests and results were conducted on a multimodal fusion recognition model based on spatial correlation. The algorithm block diagram is as follows: Figure 8As shown, the complete process of emotion recognition based on EEG and fMRI multimodal data is as follows: First, EEG and fMRI multimodal datasets are collected and multi-channel spatial data is constructed. After preprocessing such as baseline drift removal, denoising, and normalization, spatial 3D data and multi-channel temporal data are constructed respectively, and then the training set and test set are divided at a ratio of 5:1. Next, a multimodal fusion model is constructed with EEG signal input and fMRI data input, through cross-modal feature layer fusion, multimodal decision layer fusion, fully connected layer, and output layer. Then, the model is initialized and trained using the training set, and the learning rate is dynamically adjusted to update the model. Convergence is continuously judged. If convergence is not achieved, iterative process continues. If convergence is achieved, the best model and parameters are saved. Finally, the trained model is inferred using the test set, and the emotion recognition classification result is output. To verify the superiority of the designed multimodal model, the hyperparameters and model parameters are continuously adjusted through experiments, and different indicators are used to reflect the model performance. The performance and classification effect of the multimodal emotion model are evaluated by training accuracy and validation accuracy. The specific parameter settings are shown in Table 1.

[0122] Table 1 Model training parameter settings Parameter name value Single sample length 128 Batch size 64 Training times 100 Learning rate 0.001 Optimizer Adam loss function Focal Loss

[0123] To verify the performance of the proposed model, this invention compares it with several common algorithms, including baseline convolutional neural networks (CNN), deep belief networks (DBN), graph convolutional networks (GCN), long short-term memory networks (LSTM) and GCNN (ECLGCNN), dense networks with spatiotemporal spectral attention (SSTEmotionNet), and dynamic graph convolutional neural networks (DGGCN). Simultaneously, analyses were performed on unimodal, bimodal, and multimodal emotion recognition, and the superiority of the proposed model was verified through comparison.

[0124] To provide more details regarding model recognition performance, Figure 9 The accuracy curves of the multimodal model during the training process are shown.

[0125] With each training iteration, the model demonstrates strong learning ability on the training set, achieving 95% accuracy. However, it exhibits significant fluctuations on the test set, but overall, the curve maintains an upward or downward trend. Figure 10 As shown, the accuracy ultimately reached 85%. To prevent overfitting, an early stopping mechanism was implemented, such as... Figure 9 The training ends around the 83rd epoch, at the location indicated by the middle green line.

[0126] To verify the generalization ability of the EF_CN model on emotion recognition tasks, this section conducts a multi-class classification comparison experiment on the dataset. Analysis of the experimental results in Table 2 reveals that the accuracy of other models in the comparison tasks decreased, further demonstrating the superiority of this model.

[0127] Table 2 Comparison of Multi-Class Recognition Performance of Different Network Models Model Accuracy (%) F1 CNN+LSTM 74 70.6 DBN 73.2 76.5 DGGCN 81.3 - SST-EmotionNet 84 - EF_CNT 85 81.2

[0128] To analyze the model's classification performance on specific emotional features, Figure 11 The model's multi-class confusion matrices on the training and validation sets are presented, reflecting the model's recognition results for different emotional states and the corresponding misclassification rates. By comparing these two confusion matrices, the model's classification performance on specific emotional features can be analyzed in depth. In the confusion matrix of the training set, the model performs best in recognizing the fear emotion, with 3262 correctly classified samples and a relatively small number of misclassified samples. This indicates that the model can capture the features of the fear emotion well. In contrast, the model performs poorly in recognizing the surprised emotion, with 595 correctly classified samples and a relatively large number of misclassified samples. This may be related to the small number of surprised emotion samples in the training set, making it difficult for the model to fully learn the features of this emotion. In the confusion matrix of the validation set, the model still performs well in recognizing the fear emotion, with 2974 correctly classified samples and a relatively small number of misclassified samples. However, the model's performance in recognizing the surprised emotion is still poor, with 666 correctly classified samples and a relatively large number of misclassified samples. This further verifies the impact of the number of samples in the training set on the model's recognition performance.

[0129] In another embodiment, to verify the effectiveness of the hybrid fusion approach, different multimodal fusion methods were studied. Multiple sets of comparative experiments were set up, where G represents the GRU module, Concat represents direct feature fusion, CMA represents fusion based on cross-modal attention mechanisms, H represents the proposed hybrid fusion, and SA represents self-attention mechanism fusion. Analysis of the experimental results in Table 3 shows that multimodal fusion can improve the performance of the emotion recognition model. By comparing self-attention mechanisms and cross-modal attention mechanisms, it was verified that the proposed CMA module can capture useful information between EEG and fMRI signals. The hyphen (-) indicates the removal of the corresponding module.

[0130] Table 3 Comparison of Recognition Performance of Different Module Combinations Model ACC (%) F1 -G 81.53 83.50 -CMA 75.12 73.20 -H 82.27 84.31 -SA 81.87 84.99 EF_CNT 85.00 81.20

[0131] Compared with the prior art, the technical solution proposed in this invention has the following significant advantages:

[0132] Significantly improved recognition accuracy and robustness: This invention utilizes a unique hybrid fusion architecture, leveraging both the deep interactive advantages of feature layer fusion and the stability advantages of decision layer fusion. Experimental results demonstrate that the proposed EF_CNT model outperforms several existing baseline models in both accuracy and F1 score, proving its superior performance in emotion recognition tasks.

[0133] Efficient multimodal information complementarity is achieved: the core cross-modal attention mechanism adaptively learns and captures the complex dependencies between the high temporal dynamics of EEG and the high spatial localization of fMRI, achieving effective alignment and complementarity of the two modalities and solving the problem of deep fusion of heterogeneous data. Ablation experiments demonstrate that, compared with traditional feature splicing or self-attention mechanisms, the CMA module can more effectively capture useful cross-modal information, significantly improving model performance.

[0134] The model has stronger generalization ability: the hybrid fusion strategy avoids the information loss or model bias that may be caused by a single fusion method, integrates information from different levels, and makes the model more adaptable to data changes and noise, thereby improving the model's generalization ability.

[0135] Accordingly, embodiments of the present invention also provide a multimodal emotion recognition system based on a cross-modal attention mechanism, such as... Figure 13 The diagram shown is a structural schematic of the system. This multimodal emotion recognition system based on a cross-modal attention mechanism includes the following modules:

[0136] The data preprocessing unit 201 preprocesses the synchronously acquired EEG and fMRI signals to obtain a preprocessed one-dimensional time-series signal.

[0137] Multidimensional data construction unit 202 constructs a multi-channel and multi-dimensional dataset, converting the preprocessed one-dimensional time-series signal into a multidimensional data structure that is more suitable for deep learning model processing.

[0138] The feature extraction unit 203 constructs a dual-channel parallel hybrid fusion network model to perform emotion recognition based on the hybrid fusion network model. The hybrid fusion network model includes a cross-modal feature layer fusion module and a multimodal decision layer fusion module.

[0139] The emotion recognition unit 204 concatenates the features from the cross-modal feature layer fusion module and the multimodal decision layer fusion module, and uses a classifier to obtain the emotion determination result.

[0140] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0141] Except for the technical features described in the specification, all other technologies are known to those skilled in the art.

Claims

1. A multi-modal emotion recognition method based on cross-modal attention mechanism, characterized in that, The method comprises: Step 1, pre-processing the EEG signal and the fMRI signal collected synchronously to obtain a pre-processed one-dimensional time sequence signal; Step 2, converting the pre-processed one-dimensional time sequence signal into a multi-dimensional data structure more suitable for processing by a deep learning model; Step 3, constructing a dual-channel parallel hybrid fusion network model, and performing emotion recognition based on the hybrid fusion network model, wherein the hybrid fusion network model comprises a cross-modal feature layer fusion module and a multi-modal decision layer fusion module; Step 4, splicing the features from the cross-modal feature layer fusion module and the multi-modal decision layer fusion module, and using a classifier to obtain an emotion determination result.

2. The multi-modal emotion recognition method based on cross-modal attention mechanism according to claim 1, characterized in that, In step 1, the pre-processing of the EEG signal comprises: Step 1a, for problematic lead signals, reconstructing the bad channel signal by averaging the EEG data around the bad channel as the reconstructed value of the bad channel; Step 2a, using a band-pass filter to remove high-frequency noise; Step 3a, reducing the sampling rate to reduce the amount of data and improve the speed of processing data; Step 4a, setting an electrode point with a known potential as a reference electrode to reduce the impact of noise on the signal; Step 5a, extracting the time point at which an emotion is induced during the collection of the signal, segmenting the data, and correcting the baseline; Step 6a, performing ICA artifact recognition on the EEG signal, observing the time and frequency characteristics of each component of ICA, removing the electrooculogram and electrocardiogram artifact components, and reconstructing the EEG signal using the remaining ICA components.

3. The multi-modal emotion recognition method based on cross-modal attention mechanism according to claim 1, characterized in that, In step 1, the pre-processing of the fMRI signal comprises: Step 1b, removing non-steady-state data to remove image frame data collected in the first few seconds; Step 2b, performing layer time correction, determining a reference layer, calculating the time offset of other layers relative to the reference layer, and resampling according to the obtained time offset to correct the time point consistency; Step 3b, performing head motion correction, registering the brain images at different scan time points to the reference frame by rigid body transformation, selecting the first frame or the middle frame in the sequence as the reference frame, and removing image frames with large head motion ranges; Step 4b, performing image registration, first, selecting a higher resolution MRI structural image as a reference image, and registering the low-resolution fMRI image to the selected reference structural image through radiometric transformation or non-rigid transformation image transformation; then, registering the T1 structural image data to the standard space through radiometric transformation; finally, combining the two transformation matrices and applying them to the fMRI data to register them to the standard space; Step 5b, performing spatial smoothing to reduce noise in the fMRI data.

4. The multi-modal emotion recognition method based on cross-modal attention mechanism according to claim 1, characterized in that, In step 2, the one-dimensional time sequence signal includes a one-dimensional EEG time sequence signal and a one-dimensional fMRI time sequence signal, the one-dimensional EEG time sequence signal is mapped into a two-dimensional grid matrix according to a spatial distribution map of a brain electrical electrode on a scalp, and then the two-dimensional grid matrix is extended into an EEG three-dimensional data sequence containing spatial, temporal and frequency band information according to different characteristics of emotions in different frequency bands; the one-dimensional fMRI time sequence signal is extracted to form blood oxygen level dependent signals of specific brain regions related to emotional tasks, thereby forming fMRI multi-channel time sequence data.

5. The multi-modal emotion recognition method based on cross-modal attention mechanism according to claim 4, characterized in that, In step 3, the cross-modal feature layer fusion module first inputs the EEG three-dimensional data sequence and the fMRI multi-channel time sequence data into a gated recurrent unit (GRU) network respectively to effectively capture the time dependence within each modality and output an EEG feature sequence and an fMRI feature sequence containing time sequence information; then, the EEG feature sequence is taken as a query of a target modality, the fMRI feature sequence is taken as a key and a value of a source modality, an attention weight of the fMRI feature to the EEG feature is calculated, a first fusion feature is generated to represent how the fMRI information supplements and enhances the EEG feature; the fMRI feature sequence is taken as a query of a target modality, the EEG feature sequence is taken as a key and a value of a source modality, an attention weight of the EEG feature to the fMRI feature is calculated, a second fusion feature is generated to represent how the EEG information supplements and enhances the fMRI feature; finally, the first fusion feature and the second fusion feature are spliced to obtain a deep fusion feature, and then a global average pooling layer and a fully connected layer are used to obtain an emotional classification result of the cross-modal feature layer fusion module.

6. The multi-modal emotion recognition method based on cross-modal attention mechanism according to claim 5, characterized in that, In step 3, the multi-modal decision layer fusion module constructs two independent Transformer-based models, namely an EEG-Transformer model and an fMRI-Transformer model, the EEG three-dimensional data sequence is input into the EEG-Transformer model for deep feature extraction and emotional classification to obtain an EEG deep feature and an EGG decision output result, and the fMRI multi-channel time sequence data is input into the fMRI-Transformer model for deep feature extraction and emotional classification to obtain an fMRI deep feature and an fMRI decision output result; the EGG decision output result and the fMRI decision output result are fused by using a weighted summation fusion strategy to obtain a fusion emotional classification result of the multi-modal decision layer fusion module.

7. The multi-modal emotion recognition method based on cross-modal attention mechanism according to claim 6, characterized in that, In step 4, the deep fusion feature output by the cross-modal feature layer fusion module is spliced with the EEG deep feature and the fMRI deep feature output by the multi-modal decision layer fusion module to form a combined feature vector containing feature layer interaction information and single modality independent information; the combined feature vector is input into one or more fully connected layers, and an emotional determination result is obtained through a Softmax activation function.

8. A multi-modal emotion recognition system based on cross-modal attention mechanism characterized in that, The system comprises: The data preprocessing unit pre-processes the synchronously collected EEG signal and fMRI signal to obtain a pre-processed one-dimensional time sequence signal; The multi-dimensional data construction unit converts the pre-processed one-dimensional time sequence signal into a multi-dimensional data structure more suitable for deep learning model processing; The feature extraction unit constructs a double-channel parallel hybrid fusion network model and performs emotion recognition based on the hybrid fusion network model, wherein the hybrid fusion network model includes a cross-modal feature layer fusion module and a multi-modal decision layer fusion module. The emotion recognition unit splices the features from the cross-modal feature layer fusion module and the multi-modal decision layer fusion module and obtains an emotion determination result using a classifier.

9. The multi-modal emotion recognition method based on cross-modal attention mechanism according to claim 8, characterized in that, The cross-modal feature layer fusion module first inputs the EEG three-dimensional data sequence and the fMRI multi-channel time sequence data into a gated recurrent unit (GRU) network respectively to effectively capture the time dependence relationship within each modality and output an EEG feature sequence and an fMRI feature sequence containing time sequence information; then, the EEG feature sequence is taken as a target modality query, the fMRI feature sequence is taken as a source modality key and value, the attention weight of the fMRI feature to the EEG feature is calculated, a first fusion feature is generated, which represents how the fMRI information supplements and enhances the EEG feature; the fMRI feature sequence is taken as a target modality query, and the EEG feature sequence is taken as a source modality key and value, the attention weight of the EEG feature to the fMRI feature is calculated, a second fusion feature is generated, which represents how the EEG information supplements and enhances the fMRI feature; finally, the first fusion feature and the second fusion feature are spliced to obtain a deep fusion feature, and then a global average pooling layer and a full connection layer are used to obtain an emotion classification result of the cross-modal feature layer fusion module.

10. The multi-modal emotion recognition method based on cross-modal attention mechanism according to claim 8, characterized in that, The multi-modal decision layer fusion module constructs two independent Transformer-based models, namely an EEG-Transformer model and an fMRI-Transformer model. The pre-processed EEG time sequence data is input into the EEG-Transformer model for deep feature extraction and emotion classification to obtain an EEG deep feature and an EGG decision output result. The fMRI time sequence data is input into the fMRI-Transformer model for deep feature extraction and emotion classification to obtain an fMRI deep feature and an fMRI decision output result. The EGG decision output result and the fMRI decision output result are weighted and fused to obtain a fusion emotion classification result of the multi-modal decision layer fusion module.

Images