Emotion recognition method and system based on bidirectional mamba and cross-modal fusion transformer

Abstract

The present application relates to the technical field of artificial intelligence and machine learning, in particular to an emotion recognition method and system based on bidirectional Mamba and cross-modal fusion Transformer, comprising: preprocessing multi-modal original physiological signals to obtain preprocessed multi-modal physiological signals; inputting the preprocessed multi-modal physiological signals into corresponding encoders respectively, each encoder performing long-range time series modeling through a built-in bidirectional Mamba module, and outputting multi-modal time series token sequences of a unified specification respectively; concatenating the multi-modal time series token sequences in a sequence dimension into a joint token sequence, inputting the joint token sequence into a cross-modal Transformer fusion module for sequence-level global self-attention interaction, and obtaining a fusion feature vector; inputting the fusion feature vector into a task perception layer for processing, and outputting an emotion dimension recognition result through a classification head; the present application improves the accuracy of emotion recognition.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and machine learning technology, and in particular to an emotion recognition method and system based on bidirectional Mamba and cross-modal fusion Transformer. Background Technology

[0002] Emotion recognition is a core task in affective computing, with significant applications in healthcare, human-computer interaction, and driver monitoring. Physiological signal-based emotion recognition has garnered widespread attention due to its difficulty in being subjectively forged. Among these, electroencephalography (EEG) signals, which directly reflect the state of brain neural activity, are considered one of the most reliable sources of emotion recognition signals.

[0003] Early approaches to emotion recognition primarily relied on external behavioral features such as facial expressions and speech rhythm, combining handcrafted features with traditional machine learning classifiers. However, these methods were susceptible to interference from subjective masking and feigned expressions by participants, resulting in limited reliability. Subsequently, research shifted towards physiological signals such as EEG, ECG, GSR, and EOG, which are less susceptible to subjective control or faking, leading to higher reliability.

[0004] In recent years, deep models such as CNN, RNN / LSTM, Graph Neural Networks (GNN), and Transformer have gradually replaced the two-stage paradigm of handcrafted features and traditional classifiers, achieving end-to-end feature learning and classification. Multimodal fusion, cross-subject domain adaptation, and efficient long sequence modeling architectures represented by Mamba have become cutting-edge research directions in this field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an emotion recognition method based on bidirectional Mamba and cross-modal fusion Transformer. This method captures global emotion-related patterns while maintaining linear computational complexity, achieving cross-modal sequence-level attention fusion and avoiding information loss associated with traditional vector-level fusion.

[0006] On the one hand, an emotion recognition method based on bidirectional Mamba and cross-modal fusion Transformer is provided, including: The original multimodal physiological signals were preprocessed to obtain the preprocessed multimodal physiological signals. The preprocessed multimodal physiological signals are input into the corresponding encoders. In each encoder, feature extraction is performed first, and then long-range temporal modeling is performed through the built-in bidirectional Mamba module to output multimodal temporal token sequences of uniform specifications. The multimodal temporal token sequence is concatenated into a joint token sequence along the sequence dimension. After superimposing modality type embedding and position embedding, it is input into the cross-modal Transformer fusion module for sequence-level global self-attention interaction to obtain the fused feature vector. The fused feature vector is input into the task perception layer for processing to obtain shared features. After the shared features are processed by the task-specific projection layers corresponding to each emotion dimension, the emotion dimension recognition results are output through the classification head.

[0007] Furthermore, it also includes: during the training phase, the fused feature vector is input into the subject discriminator via a gradient inversion layer, and cross-subject general emotion representations are learned through domain adversarial training, jointly optimizing the emotion classification loss and the domain discrimination loss.

[0008] Furthermore, the multimodal temporal token sequences are concatenated into a joint token sequence along the sequence dimension. After overlaying modality type embeddings and position embeddings, the sequence is input into the cross-modal Transformer fusion module for sequence-level global self-attention interaction to obtain a fused feature vector. Specifically: First, the three multimodal temporal token sequences are overlaid with their respective learnable modality type embedding vectors, enabling the Transformer to distinguish tokens from different modalities. Next, the three tokens with overlaid embeddings are concatenated along the sequence dimension to construct a joint token sequence of length 3N. Simultaneously, learnable position embeddings are overlaid at the 3N positions of the joint sequence to provide position information for the Transformer. Then, the joint token sequence is input normalized and then input into a multi-layer Transformer encoder for global self-attention calculation, realizing fine-grained information interaction between cross-modal tokens. Finally, the joint token sequence output by the Transformer is subjected to gated average pooling and then passed through an output projection layer to obtain the final fused feature vector.

[0009] Furthermore, the bidirectional Mamba module specifically involves: firstly, performing a bidirectional selective state space scan on the features; for the input sequence, using a forward selective state space scan branch and a reverse selective state space scan branch, simultaneously running a forward SSM scan and a reverse SSM scan to obtain a forward output and a reversed output; the two branches each use independent state space model parameters, and the two sets of parameters are updated independently and not shared during training. Next, the forward output and the inverted output are concatenated along the channel dimension, and a channel-wise gated weight is generated by linear transformation and Sigmoid activation. The forward output and the inverted output are then weighted and fused using the gated weight to obtain the fused feature. The fused feature is then transformed by the output projection layer and used as the output sequence of the bidirectional Mamba module.

[0010] Furthermore, the reversed output after inversion specifically involves: in the reverse branch, first performing a flip operation along the time axis on the input feature sequence to obtain a flipped input sequence; then performing an independent reverse causal one-dimensional convolution operation on the flipped input sequence, and processing it with the SiLU activation function to obtain the reverse convolution features; The inverse convolutional features are input into the projection layer for parameter decomposition to obtain the inverse adaptive stride, the inverse input projection matrix, and the inverse output projection matrix. The inverse parameters, the inverse state transition matrix, and the inverse direct connection parameters are substituted into the recursive equation of the state space model to perform a scanning operation. The scanning operation result is multiplied element-wise with the result of the gated input after SiLU activation, which has been flipped, to obtain the output features in the inverse temporal sequence. The inverse temporal output features are flipped back along the time axis to the original sequence to obtain the inverse output features aligned with the original temporal sequence.

[0011] Furthermore, the multimodal raw physiological signals include: electroencephalography (EEG), electrooculography (EOG), and electromyography (EMG).

[0012] Furthermore, the task awareness layer includes gated fusion, a hybrid expert network, and a task-specific projection layer.

[0013] On the other hand, an emotion recognition system based on bidirectional Mamba and cross-modal fusion Transformer is provided, including: a preprocessing module, which preprocesses the original multimodal physiological signals to obtain preprocessed multimodal physiological signals; The modality-specific encoder module inputs the preprocessed multimodal physiological signals into the corresponding encoders. In each encoder, feature extraction is performed first, and then long-range temporal modeling is performed through the built-in bidirectional Mamba module, outputting a multimodal temporal token sequence of uniform specifications. The cross-modal Transformer fusion module concatenates the multimodal temporal token sequence into a joint token sequence along the sequence dimension. After superimposing modality type embedding and position embedding, the sequence-level global self-attention interaction is performed by the cross-modal Transformer fusion module to obtain the fused feature vector. The classification module inputs the fused feature vector into the task perception layer for processing to obtain shared features. After the shared features are processed by the task-specific projection layers corresponding to each emotion dimension, the emotion dimension recognition results are output through the classification head.

[0014] Furthermore, an electronic device is also provided, including: Memory, used for non-transitory storage of computer-readable instructions; and Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0015] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method described in the first aspect is performed.

[0016] The above technical solution has the following advantages or beneficial effects: This invention preserves the features of each modality as an N-dimensional token sequence (instead of compressing them into a single vector) before performing Transformer fusion, enabling cross-modal attention to capture the correspondence and complementary patterns of different modalities in the time dimension, effectively avoiding the problem of attention degenerating into a linear transformation in traditional vector-level fusion.

[0017] This invention introduces DANN during the training phase to improve cross-subject generalization ability, and domain adversarial training forces the feature extractor to learn emotional features irrelevant to the subjects, thereby enhancing the system's practicality in real-world scenarios. Attached Figure Description

[0018] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0019] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a diagram of the overall architecture of the present invention; Figure 3 This is a structural diagram of the bidirectional Mamba bidirectional state-space model of the present invention; Figure 4 This is a structural diagram of the EEG encoder of the present invention; Figure 5 This is a structural diagram of the cross-modal Transformer fusion module of the present invention; Figure 6 This is a structural diagram of the task perception layer of the present invention; Figure 7 This is a schematic diagram of the domain adversarial neural network module of the present invention; Figure 8 This is a flowchart of the system training process of the present invention. Detailed Implementation

[0020] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the invention. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] In this embodiment of the invention, "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of this invention, "multiple" refers to two or more.

[0023] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0024] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0025] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0026] Example 1 Terminology Explanation: Electroencephalogram (EEG) is a physiological signal that records the electrical activity of neuronal populations in the cerebral cortex using electrodes. In this invention, a 32-lead EEG acquisition is used.

[0027] Electrooculogram (EOG) records the electrical potential changes caused by eye movements and consists of two channels: horizontal electrooculogram (hEOG) and vertical electrooculogram (vEOG).

[0028] Electromyography (EMG) records the electrical activity signals of muscles. This invention uses two channels: the zygomaticus major (zEMG) and the trapezius (tEMG).

[0029] Mamba: A deep learning architecture based on the Selective State-Space Model (SSM) that can perform long-range modeling of sequential data with linear complexity.

[0030] Bi-directional Mamba: BiMamba is an improved architecture that runs Mamba scans in both the forward and reverse directions and merges bi-directional features through a gating mechanism.

[0031] VAD Emotion Model: A three-dimensional emotion description framework consisting of Valence, Arousal, and Dominance.

[0032] Domain Adversarial Neural Network (DANN) achieves cross-subject generalization through gradient inversion.

[0033] Mixture of Experts (MoE) model: This model dynamically selects expert networks to process different samples through a routing mechanism.

[0034] DEAP Dataset: Database for Emotion Analysis using Physiological Signals, a publicly available emotion recognition dataset containing multimodal physiological signals from 32 participants and 40 music video stimuli.

[0035] Baseline period: refers to the time window for collecting signals in a resting state before the presentation of emotional stimuli, used to record the baseline level of physiological signals of subjects under conditions without emotional stimuli.

[0036] In emotion recognition tasks based on electroencephalogram (EEG) signals, existing technologies have the following problems: First, there is insufficient utilization of single-modal information. Most existing studies only use EEG signals, neglecting the emotion-related information carried by peripheral physiological signals such as EOG (electroophthalmos) and EMG (electromymos). Peripheral physiological signals and central nervous system activity are complementary to a certain extent. Using EEG alone cannot fully capture the multidimensional physiological manifestations of emotions, resulting in limited recognition accuracy.

[0037] Second, the temporal modeling capability is insufficient. Traditional convolutional neural networks (CNNs) and standard recurrent neural networks (RNNs / LSTMs) have significant limitations in capturing long-range temporal dependencies in EEG. CNNs have limited receptive fields, and LSTMs are computationally inefficient and the gradient vanishing problem is difficult to completely solve, resulting in insufficient modeling accuracy for emotion-related neural oscillation patterns (such as α, β, and γ frequency band activities).

[0038] Third, the cross-modal fusion strategy is crude. Most existing multimodal fusion methods use simple feature concatenation or vector-level weighted summation, which cannot fully capture the temporal cross-relationships between different modalities. Compressing the features of each modality into a single vector before fusion causes cross-modal attention to severely degenerate into a linear transformation, losing the core advantages of the Transformer attention mechanism.

[0039] Fourth, it suffers from poor cross-subject generalization ability. EEG signals exhibit strong individual variability, and model performance significantly degrades when training and test data come from different subjects. Existing methods lack effective domain adaptation mechanisms, making it difficult to learn universal emotional representations across subjects.

[0040] Fifth, the class imbalance problem is not handled properly. Emotion recognition datasets often suffer from class imbalance. The standard cross-entropy loss function pays too much attention to easily classified samples, resulting in poor recognition performance of the model in the minority classes and unstable overall performance.

[0041] like Figure 1-2 The emotion recognition method based on bidirectional Mamba and cross-modal fusion Transformer of the present invention includes the following steps: S1: Preprocess the original multimodal physiological signals to obtain preprocessed multimodal physiological signals.

[0042] Multimodal raw physiological signals include: electroencephalography (EEG), electrooculography (EOG), and electromyography (EMG).

[0043] Preprocessing of multimodal raw physiological signals is performed as follows: S101: A modal adaptive filtering strategy is used to perform bandpass filtering on the original multimodal physiological signals to obtain the filtered original multimodal physiological signals. Specifically: For EEG signals, since the data_preprocessed_python version of the DEAP dataset has already performed 4-45Hz bandpass filtering, this invention does not repeat the filtering to avoid signal distortion. A 1-45Hz bandpass filter is applied to EOG signals to preserve eye-movement related components; a 10-45Hz bandpass filter is applied to EMG signals to highlight muscle contraction characteristics.

[0044] S102: Baseline correction and trial-level normalization are performed on the filtered multimodal raw physiological signals. After removing the baseline period (3 seconds), z-score normalization is performed on the time axis for each trial signal, so that the normalization is completed before segmentation, thereby preserving the relative amplitude information between different time periods within the same trial and avoiding semantic fragmentation caused by segment-level normalization.

[0045] (1) in, and These are the mean and standard deviation of this channel in this trial, respectively. It is a numerically stable term; This represents the amplitude of a single sample of the input signal.

[0046] Furthermore, baseline removal includes two steps: truncation and baseline correction, specifically: Let the original trial signal be... (In DEAP, T=8064, corresponding to a sampling rate of 63 seconds × 128Hz, and C is the number of channels), the baseline period length is 3 seconds, corresponding to the number of sampling points. .

[0047] Step 1 (Segmentation): Divide the signal along the time axis into baseline segments. With the test section The baseline segment itself is discarded after this step and does not participate in subsequent segmentation, normalization, and model training.

[0048] Step 2 (baseline correction): Calculate the channel-by-channel mean of the baseline segment on the time axis, and subtract the mean from each time point of the trial segment.

[0049] S103: A sliding window strategy is used to segment each 60-second trial into multiple samples, achieving sliding window segmentation. A sliding window strategy with a 4-second window length and 50% overlap is used to segment each 60-second trial into multiple samples, increasing the training sample size by approximately 2 times compared to non-overlapping segmentation.

[0050] S104: Fuzzy label exclusion mechanism is used to remove fuzzy samples. For continuous emotion scores (1-9 points), a dual threshold strategy is used to convert them into binary classification labels: scores below 4.0 points are classified as "low" (0), and scores above or equal to 6.0 points are classified as "high" (1). Fuzzy samples in the range [4.0, 6.0) are labeled and excluded from training, reducing the interference of annotation noise on the model. Among them, the continuous emotion score is the subjective self-evaluation rating of the participants in the DEAP dataset after watching each approximately 1-minute music video segment, based on the Self-Assessment Manikin (SAM) scale, with a rating range of 1 to 9.

[0051] This invention effectively compensates for the information limitations of single-modal methods by integrating three complementary physiological signals: EEG, EOG, and EMG, and improves the average recognition accuracy of three-dimensional emotion indicators (valence / arousal / dominance).

[0052] This invention employs fuzzy label exclusion to ensure data quality. The dual-threshold fuzzy label exclusion mechanism reduces the interference of subjective labeling ambiguity on the model, thereby improving the model's learning efficiency and generalization ability from the perspective of data quality.

[0053] S2: The preprocessed multimodal physiological signals are input into the corresponding encoders. In each encoder, feature extraction is performed first, and then long-range time series modeling is performed through the built-in bidirectional Mamba module to output multiple time series token sequences of the same specification.

[0054] Encoders corresponding to multimodal physiological signals include: EEG encoders, EOG encoders, and EMG encoders, such as... Figure 4 As shown.

[0055] The EEG encoder is designed for the spatial topology and multi-band characteristics of electroencephalogram (EEG) signals, and includes: First, a spatial convolutional layer is performed: using a two-dimensional convolutional kernel of the form (32, 1), cross-electrode spatial filtering is performed along the electrode dimension, compressing the 32-lead signal into 16-channel spatial features, combined with batch normalization and ReLU activation. This operation simulates spatial filters (such as CSP, xDAWN) in traditional EEG analysis, automatically learning the optimal spatial weights under data-driven conditions.

[0056] Next, the time-frequency decomposition module is used: five one-dimensional convolutions with different kernel sizes (64, 32, 16, 8, and 4 sampling points) are used to extract features from five frequency bands: δ (1-4Hz), θ (4-8Hz), α (8-14Hz), β (14-31Hz), and γ (31-45Hz). The five outputs are stacked along the frequency band dimension to obtain a time-frequency feature tensor of shape (B, 5, C, T), effectively capturing the temporal activity of each emotion-related frequency band.

[0057] Finally, temporal tokenization is performed. The time-frequency feature tensor is reshaped and mapped to a linear projection layer. Dimension, to obtain The token sequence is then processed through three stacked bidirectional Mamba (BiMamba) layers for long-range time-series modeling. Finally, adaptive average pooling is used to compress the time dimension T into N fixed tokens, outputting the result. The token sequence.

[0058] Furthermore, the EOG encoder and EMG encoder are specifically designed for the characteristics of peripheral physiological signals, specifically as follows: The EOG encoder uses two layers of one-dimensional convolutions to extract eye-tracking pattern features. These features are then input into a two-layer bidirectional Mamba (BiMamba) convolutional algorithm for temporal modeling, and N fixed tokens are output through adaptive average pooling. The EOG signal is sensitive to attention switching and emotion-related eye-tracking patterns, serving as a complement to EEG. The kernel sizes of the two one-dimensional convolutional layers are 15 and 7.

[0059] The EMG encoder uses two layers of one-dimensional convolution and max pooling to extract facial muscle activity features. These features are then input into a two-layer BiMamba algorithm for temporal modeling, and N fixed tokens are output through adaptive average pooling. The EMG signal is sensitive to the movement of facial muscles and can supplement the expression-related emotional information that EEG cannot capture. The kernel sizes of the two one-dimensional convolution layers are 25 and 15, respectively; the max pooling downsampling is 4 times.

[0060] Finally, standardize the output specifications. Ensure all encoders output a token sequence of the same specification. This facilitates subsequent cross-modal fusion.

[0061] Furthermore, the bidirectional Mamba module built into EEG encoders, EOG encoders, and EMG encoders, such as Figure 3 As shown, specifically: First, a bidirectional selective state-space scan is performed on the features. For the input sequence, a forward selective state-space scan branch and a backward selective state-space scan branch are used to run the forward SSM scan and the backward SSM scan simultaneously to obtain the forward output and the reversed output. The two branches use independent state-space model parameters, and the two sets of parameters are updated independently and do not share each other during training. For an input sequence of length L, the bidirectional Mamba performs a forward SSM scan (from t=0 to t=L) and a backward SSM scan (from t=L to t=0) simultaneously.

[0062] It is important to note that forward branches and reverse branches use different parameters; the parameters used for forward branches include: ,conv1d, , The reverse branch uses independent parameters, including: , _, , , These two sets of parameters are optimized separately, enabling forward and backward scans to learn different timing patterns.

[0063] SSM models the sequence using the following discrete recursive equation, resulting in the state-space model recursive equation: (2) (3) in, It is in a hidden state; , This is the transition matrix obtained through zero-order preservation (ZOH) discretization; The input is an adaptive step size, which is generated by a linear projection activated by softplus; , To output projection parameters; This is the model output at time t; It is the input feature vector at time t.

[0064] In the forward branch, the input sequence x undergoes causal one-dimensional convolution and SiLU activation to obtain... ; Step size decomposed by projection layer Input matrix With output matrix Substitute it into the SSM recursive equation for scanning, and then add gating. Activating the SiLU yields the forward output: (4) in, Here is the state transition matrix. Forward input projection matrix, For the forward output projection matrix, For direct connection parameters, For adaptive step size, For element-wise multiplication, The SiLU activation function is used. For gating branch input.

[0065] In the reverse branch, the reversed output includes: first, the input feature sequence is flipped along the time axis to obtain a flipped input sequence; then, a reverse causal one-dimensional convolution operation is performed on the flipped input sequence, and processed by the SiLU activation function to obtain reverse convolution features; the reverse convolution features are parametrically decomposed to obtain reverse parameters; the reverse parameters are substituted into the recursive equation of the state space model to perform a scan operation, and the scan operation result is multiplied element-wise with the result of the gated input after flipping and SiLU activation to obtain the output features under reverse time sequence; finally, the reverse time sequence output features are flipped back along the time axis to obtain the reversed output features aligned with the original time sequence.

[0066] Specifically: First, flip the input sequence x along the time axis to obtain... ; Through independent causal convolution Activated with SiLU, we obtain Similarly, decompose into , , Substitute into formula (2) (using the reverse branch parameter) , Scan and overlay the flipped gating Activation of SiLU yields: (5) in, This is the reverse state transition matrix. For the reverse input projection matrix, To output the projection matrix in reverse, For reverse direct connection parameters, For reverse adaptive step size, This is a sequence reversal operation. It is a function symbol representing the sequential scan computation process of a selective state-space model; Indicates parameters ( The selective state-space scan operation is performed by recursively expanding the input sequence x according to equation (2) time step by time, and its output sequence is the same length as the input sequence.

[0067] because It was calculated in a coordinate system where the time axis was flipped. It needs to be flipped back along the time axis to the original timing sequence in order to be consistent with... Alignment and fusion at the same time coordinate: (6) in, This is the output sequence of the reverse SSM. This is the inverted output after the flip.

[0068] Next, the channel-level gating fusion mechanism is introduced. This mechanism differs from the naive fusion method of direct summation or concatenation; it is a bidirectional fusion mechanism based on input adaptive gating. Specifically, it includes: gating the forward output... and reverse flip output After concatenation along the channel dimension, a channel-wise gated weight g is generated through a linear layer and a Sigmoid activation function. The gated weight is initialized to 0 (Sigmoid output 0.5) to ensure equal fusion of forward and backward information in the early stages of training, and is adaptively adjusted as training progresses.

[0069]

[0070] (8) in, For the Sigmoid function, For element-wise multiplication, For channel splicing, These are the learnable weight parameters of the gated projection layer (a fully connected linear layer). Gated weights Initialize to near zero to ensure equal fusion of forward and backward information in the early stages of training, and adaptively adjust as training progresses; It is a forward output; It is the reverse SSM output sequence; It is a channel-by-channel gating weight; It is a fused output, that is, a fused feature.

[0071] Finally, the fused features are transformed by the output projection layer and used as the output sequence of the bidirectional Mamba module. Each bidirectional Mamba layer adopts a structure combining pre-norm (normalization before computation) and residual connections, which effectively alleviates the gradient vanishing problem in deep networks.

[0072] Furthermore, to address the overflow issue that occurs during model training, this invention also enhances numerical stability during the forward SSM scan branch. This involves the logarithmic parameters of the state matrix A. Perform a truncation operation (clamp to) (range), to avoid exponential overflow; truncate the output (clamp to) This prevents NaN from propagating in the network. Specifically, it involves adjusting the logarithmic parameters of the state matrix A. Perform a truncation operation to make scope , acting on state matrix A and The computational stage applies to both the forward and backward branches, not just the forward branch, because during training... The gradient update may cause its value to become extremely large. If no truncation is performed, the exp() operation will cause numerical overflow, resulting in inf / NaN in A and polluting the entire SSM recursion process.

[0073] The output is truncated, and its range is... The final output of the forward propagation of the BiMamba module (i.e., the result after gating fusion according to formula (6) and then projecting the output) is intended to prevent abnormally large values ​​from accumulating and amplifying layer by layer through residual connections, which could lead to NaN in the downstream layer.

[0074] Wherein, the state matrix A consists of trainable parameters learned by the model. During the forward computation, it is obtained using the following formula: (9) in, It is an element-wise numerical pruning operation that prunes tensors. Each element is restricted to a closed interval. Inside: (10).

[0075] The proposed bidirectional Mamba architecture captures the global temporal dependencies of EEG signals in linear time complexity O(L) (where L is the sequence length). Compared to the O(L) serial computation of LSTM, BiMamba supports fully parallel training; and compared to the O(L²) attention of Transformer, it is more computationally efficient for long sequences. Channel-level gated fusion further enhances the adaptive integration capability of bidirectional information.

[0076] S3: The multimodal temporal token sequence is concatenated into a joint token sequence along the sequence dimension. After superimposing modality type embedding and position embedding, it is input into the cross-modal Transformer fusion module for sequence-level global self-attention interaction to obtain the fused feature vector.

[0077] This invention proposes a sequence-level cross-modal fusion mechanism to address the problem of cross-modal attention degradation in traditional vector-level fusion, such as... Figure 5 As shown.

[0078] First, a joint token sequence is constructed. The token sequences of the three modalities (N tokens per modality) are concatenated along the sequence dimension to obtain a joint token sequence of length 3N. The concatenation operation along the sequence dimension enables the Transformer attention mechanism to perform global interactions at 3N temporal positions, thereby capturing fine-grained temporal correspondences across modalities.

[0079] First, the three multimodal temporal token sequences are superimposed with their respective learnable modality type embedding vectors, enabling the Transformer to distinguish tokens from different modal sources; the modality type embedding vectors can adopt a token type embedding similar to BERT.

[0080] Next, the three embedded tokens are concatenated along the sequence dimension to construct a joint token sequence of length 3N. Simultaneously, learnable position embeddings are superimposed on the 3N positions of the joint sequence to provide positional information for the Transformer. Then, the joint token sequence is normalized and input into a multi-layer Transformer encoder for global self-attention calculation, enabling fine-grained information interaction between cross-modal tokens. Finally, the joint token sequence output by the Transformer is subjected to gated average pooling and then passed through an output projection layer (linear transformation + layer normalization) to obtain the final fused feature vector.

[0081] The Transformer encoder can be used in a Pre-Norm style, including a 2-layer encoder and 8 attention heads.

[0082] The joint token sequence with 3N positions is first processed through a linear layer and a sigmoid function to generate a gating weight for each position. Then, a weighted average is applied to obtain the fused feature vector. Specifically: (11) in, For gated projection weights, For the first Each token represents... For the Sigmoid function, To fuse feature vectors, the size is .

[0083] This operation enables the model to adaptively focus on more discriminative token positions.

[0084] This invention preserves the features of each modality as an N-dimensional token sequence (instead of compressing them into a single vector) before performing Transformer fusion, enabling cross-modal attention to capture the correspondence and complementary patterns of different modalities in the time dimension, effectively avoiding the problem of attention degenerating into a linear transformation in traditional vector-level fusion.

[0085] S4: The fused feature vector is input into the task perception layer for processing to obtain shared features. After the shared features are processed by the task-specific projection layers corresponding to each emotion dimension, the emotion dimension recognition results are output through the classification head.

[0086] The task awareness layer includes gated fusion, hybrid expert networks, and a task-specific projection layer. For example... Figure 6 As shown, specifically: Gated fusion: Gated fusion operates on the fused feature vector obtained after cross-modal fusion. The fused feature vector is gated dimension-wise to obtain emotion-discriminating features; linear layers and sigmoid activation are applied to this... The feature channel dimension of the feature vector (i.e.) This dimension), keeping the input and output dimensions unchanged, learns an independent gating weight for each feature dimension, and then multiplies it element-wise with the original feature: (12) (13) in, For input features, It is the Sigmoid activation function. For the gated weight matrix, For gated bias terms, To integrate gating weights, This is a gated modulation output.

[0087] Gated fusion can dynamically suppress irrelevant feature dimensions and highlight emotion-discriminating features.

[0088] Hybrid Expert Network (MoELayer): Four expert networks are set up, each expert is a two-layer MLP, and the router adaptively assigns weights to each expert based on the input (soft routing). Different experts can specialize in processing different emotion patterns, thereby improving representation capacity.

[0089] Task-Specific Projection Layers: Valence, Arousal, and Dominance each have independent projection layers. These layers learn unique representations for each dimension based on shared hybrid expert network features, and maintain information flow through residual connections. In this embodiment, the projection layers are LayerNorm, Linear, and GELU.

[0090] The classification head is a two-layer MLP classification head. The classification head for each task is a two-layer MLP, namely: Linear, GELU, Dropout, and Linear. This structure has stronger non-linear expressive power compared to a single-layer linear classification head.

[0091] Furthermore, to enhance cross-subject generalization ability, this invention introduces a Domain Adversarial Neural Network (DANN) mechanism, such as... Figure 7 As shown.

[0092] During the training phase, the fused feature vectors are input into the subject discriminator via a gradient inversion layer. Domain adversarial training is used to learn a cross-subject general emotion representation, jointly optimizing the emotion classification loss and the domain discrimination loss. Specifically: Gradient Reversal Layer (GRL): This layer performs an identity transformation during forward propagation and multiplies the gradient by a negative scalar during backward propagation. This forces the upstream feature extractor to learn emotional representations independent of the subject's identity; among them, According to the formula, p linearly anneals from 0 to 1 as the training progresses.

[0093] (14) Subject discriminator: The two-layer MLP in the middle layer is responsible for predicting the subject ID (32 categories) from the fused features. Through adversarial training with GRL, the feature extractor is forced to confuse the differences between subjects, thereby forcing the learning of cross-subject general emotional features.

[0094] Joint loss optimization, the total joint training loss is: (15) in, The weighted sum of the three-dimensional emotion classification loss. To determine the cross-entropy loss for the subjects, Control the intensity of the confrontation.

[0095] Through this mechanism, the feature extractor is forced to learn general emotion representations that are irrelevant to the subjects while improving the accuracy of emotion recognition.

[0096] This invention introduces a Domain Adversarial Neural Network (DANN) mechanism to improve cross-subject generalization ability. Domain adversarial training forces the feature extractor to learn emotional features that are irrelevant to the subjects, thereby enhancing the practicality of the system in real-world scenarios.

[0097] To improve the robustness of the model, this invention proposes a combined training strategy, such as... Figure 8 As shown, it includes: Focal Loss: Uses Focal Loss instead of standard cross-entropy to focus on samples that are difficult to classify. (16) in, This represents the probability that the model predicts the true class of the sample.

[0098] pass modulation factor Reduce the weight of easily classified samples so that the model focuses on learning samples that are difficult to classify; at the same time, set differentiated class weights for the class distribution of each dimension (valence weight [2.0:1.0], arousal [1.43:1.0], dominance [1.64:1.0]), combined with label smoothing ( Prevent the model from becoming overconfident.

[0099] Mixup Data Augmentation: distributed( Sampling mixing ratio Linear interpolation is performed on the input features and labels of two random samples within the same batch to generate virtual training samples, thereby improving the model's robustness to input perturbations. (17) (18) in, This represents the virtual mixed input samples (EEG / EOG / EMG) generated by Mixup. This represents the input signal of the i-th original training sample in the current batch; This represents the input signal of the j-th original training sample randomly selected in the current batch; Indicates and The corresponding virtual hybrid tag; Indicates sample The corresponding real tags; Indicates sample The corresponding real tags.

[0100] EEG signal enhancement: Gaussian noise is injected into the EEG / EOG / EMG signals during the training phase. The model robustness is improved by using random time shift (±10 sampling points), amplitude scaling (0.9-1.1 times), random time mask (maximum 10 points, probability 20%), and random channel dropping (probability 0.2).

[0101] Warmup + Cosine Annealing Learning Rate Scheduling: The first 5 epochs use linear warm-up (increasing from 0.3 times the initial learning rate to the target learning rate 1e-4), followed by cosine annealing to ensure stability in the early stages of training and sufficient convergence in the later stages.

[0102] Random Weighted Average (SWA): SWA is enabled from the 80th epoch, periodically averaging model weights and updating batch normalized statistics to improve the generalization and stability of the final model.

[0103] Early stopping mechanism: The overall score (accuracy + F1) after smoothing (moving average over 3 epochs) is used as the indicator. If there is no improvement after 20 consecutive epochs, early stopping is triggered to prevent overfitting.

[0104] This invention employs a combination of training strategies including Focal Loss, Mixup, SWA, and label smoothing, which effectively solves practical problems such as class imbalance, overfitting, and training instability, enabling the model to converge stably even on small-sample, high-noise physiological signal data.

[0105] Example 2 This embodiment provides an emotion recognition system based on bidirectional Mamba and cross-modal fusion Transformer, including: The preprocessing module preprocesses the raw multimodal physiological signals to obtain preprocessed multimodal physiological signals. The modality-specific encoder module inputs the preprocessed multimodal physiological signals into the corresponding encoders. In each encoder, feature extraction is performed first, and then long-range temporal modeling is performed through the built-in bidirectional Mamba module, outputting a multimodal temporal token sequence of uniform specifications. The cross-modal Transformer fusion module concatenates the multimodal temporal token sequence into a joint token sequence along the sequence dimension. After superimposing modality type embedding and position embedding, the sequence-level global self-attention interaction is performed by the cross-modal Transformer fusion module to obtain the fused feature vector. The classification module inputs the fused feature vector into the task perception layer for processing to obtain shared features. After the shared features are processed by the task-specific projection layers corresponding to each emotion dimension, the emotion dimension recognition results are output through the classification head.

[0106] Example 3 This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0107] Example 4 This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An emotion recognition method based on bidirectional Mamba and cross-modal fusion Transformer, characterized in that, include: The original multimodal physiological signals were preprocessed to obtain the preprocessed multimodal physiological signals. The preprocessed multimodal physiological signals are input into the corresponding encoders. In each encoder, feature extraction is performed first, and then long-range temporal modeling is performed through the built-in bidirectional Mamba module to output multimodal temporal token sequences of uniform specifications. The multimodal temporal token sequences are concatenated into a joint token sequence along the sequence dimension. After overlaying modality type embeddings and position embeddings, the sequence is input into a cross-modal Transformer fusion module for sequence-level global self-attention interaction to obtain a fused feature vector. Specifically: First, the three multimodal temporal token sequences are each overlaid with their respective learnable modality type embedding vectors, enabling the Transformer to distinguish tokens from different modalities. Next, the three token sequences with overlaid embeddings are concatenated along the sequence dimension to construct a joint token sequence of length 3N. Simultaneously, learnable position embeddings are overlaid at the 3N positions of the joint sequence to provide positional information for the Transformer. Then, the joint token sequence is input normalized and then input into a multi-layer Transformer encoder for global self-attention calculation, realizing fine-grained information interaction between cross-modal tokens. Finally, the joint token sequence output by the Transformer is subjected to gated average pooling and then passed through an output projection layer to obtain the final fused feature vector. The fused feature vector is input into the task perception layer for processing to obtain shared features; after the shared features are processed by the task-specific projection layers corresponding to each emotion dimension, the emotion dimension recognition results are output through the classification head.

2. The emotion recognition method based on bidirectional Mamba and cross-modal fusion Transformer according to claim 1, characterized in that, Also includes: During the training phase, the fused feature vector is input into the subject discriminator via a gradient inversion layer. Through domain adversarial training, it learns a general emotion representation across subjects and jointly optimizes the emotion classification loss and the domain discrimination loss.

3. The emotion recognition method based on bidirectional Mamba and cross-modal fusion Transformer according to claim 1, characterized in that, The bidirectional Mamba module is specifically described as follows: First, a bidirectional selective state space scan of the features is performed; for the input sequence, the forward selective state space scan branch and the reverse selective state space scan branch are used to run the forward SSM scan and the reverse SSM scan simultaneously to obtain the forward output and the reversed output; the two branches use independent state space model parameters, and the two sets of parameters are updated independently and do not share each other during the training process. Next, the forward output and the inverted output are concatenated along the channel dimension, and a channel-wise gated weight is generated by linear transformation and Sigmoid activation. The forward output and the inverted output are then weighted and fused using the gated weight to obtain the fused feature. The fused feature is then transformed by the output projection layer and used as the output sequence of the bidirectional Mamba module.

4. The emotion recognition method based on bidirectional Mamba and cross-modal fusion Transformer according to claim 3, characterized in that, The reversed output after inversion is specifically as follows: In the reverse branch, the input feature sequence is first flipped along the time axis to obtain the flipped input sequence; the flipped input sequence is then subjected to a reverse causal one-dimensional convolution operation and processed by the SiLU activation function to obtain the reverse convolution features; The inverse convolution features are decomposed into inverse parameters; the inverse parameters are substituted into the recursive equation of the state space model to perform a scanning operation; the scanning operation result is multiplied element by element with the result of the gated input after SiLU activation after inversion processing to obtain the output features under inverse time series. The inverted time series output features are flipped back along the time axis to the original sequence, resulting in inverted output features aligned with the original time series.

5. The emotion recognition method based on bidirectional Mamba and cross-modal fusion Transformer according to claim 1, characterized in that, The multimodal raw physiological signals include: electroencephalography (EEG), electrooculography (EOG), and electromyography (EMG).

6. The emotion recognition method based on bidirectional Mamba and cross-modal fusion Transformer according to claim 1, characterized in that, The task perception layer includes gated fusion, hybrid expert network, and task-specific projection layer.

7. An emotion recognition system based on bidirectional Mamba and cross-modal fusion Transformer, characterized in that, include: The preprocessing module preprocesses the raw multimodal physiological signals to obtain preprocessed multimodal physiological signals. The modality-specific encoder module inputs the preprocessed multimodal physiological signals into the corresponding encoders. In each encoder, feature extraction is performed first, and then long-range temporal modeling is performed through the built-in bidirectional Mamba module, outputting a multimodal temporal token sequence of uniform specifications. The cross-modal Transformer fusion module concatenates the multimodal temporal token sequences into a joint token sequence along the sequence dimension. After superimposing modality type embeddings and position embeddings, the sequence-level global self-attention interaction is performed by the cross-modal Transformer fusion module to obtain a fused feature vector. Specifically: First, the learnable modality type embedding vectors of each of the three multimodal temporal token sequences are superimposed, enabling the Transformer to distinguish tokens from different modalities. Next, the three token sequences with superimposed embeddings are concatenated along the sequence dimension to construct a joint token sequence of length 3N. Simultaneously, learnable position embeddings are superimposed on the 3N positions of the joint sequence to provide positional information for the Transformer. Then, the joint token sequence is input normalized and then input into a multi-layer Transformer encoder for global self-attention calculation, realizing fine-grained information interaction between cross-modal tokens. Finally, the joint token sequence output by the Transformer is subjected to gated average pooling and then passed through an output projection layer to obtain the final fused feature vector. The classification module inputs the fused feature vector into the task perception layer for processing to obtain shared features. After the shared features are processed by the task-specific projection layers corresponding to each emotion dimension, the emotion dimension recognition results are output through the classification head.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the emotion recognition method based on bidirectional Mamba and cross-modal fusion Transformer as described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the emotion recognition method based on bidirectional Mamba and cross-modal fusion Transformer as described in any one of claims 1 to 6.

Images