A multi-modal emotion recognition method based on decoupled enhancement and global modeling

Abstract

This invention discloses a multimodal emotion recognition method based on decoupling enhancement and global modeling, belonging to the field of multimodal emotion recognition technology. The method includes: first, acquiring features from three modalities—text, speech, and vision—and mapping them to a unified latent space; after the initial modal input and basic interaction representations are constructed, a bidirectional state space global temporal modeling branch and a causal-inspired decoupling enhancement branch are set up in parallel; then, an adaptive gating mechanism is used to dynamically fuse the global temporal representation and the causal-inspired decoupling enhancement representation to form a unified decision vector; finally, cross-entropy loss, Soft-HGR constraints, matrix-based dual-weighted focus contrast loss, and orthogonal constraint loss are used for joint training to achieve emotion category prediction. This invention can improve the accuracy, robustness, and generalization ability of multimodal emotion recognition in complex long dialogue scenarios, and is applicable to applications such as intelligent customer service, human-computer interaction, educational assistance, and mental health monitoring.

Description

Technical Field

[0001] This invention belongs to the field of multimodal emotion recognition technology, specifically relating to a multimodal emotion recognition method based on decoupling enhancement and global modeling. Its research object is the text, speech, and visual multimodal emotion analysis task in complex long dialogue scenarios. Background Technology

[0002] Multimodal emotion recognition, as an important research direction in the field of affective computing, aims to comprehensively utilize information from multiple modalities such as text, speech, and vision to accurately understand and judge human emotional states. It has significant application value in scenarios such as intelligent customer service, mental health monitoring, medical assistance, and educational interaction. Compared to unimodal emotion recognition methods, multimodal emotion recognition can more fully explore the complementary relationships between different modalities and exhibits stronger robustness and adaptability in complex dialogue environments.

[0003] Currently, researchers have conducted research on multimodal emotion recognition tasks from multiple perspectives, including feature extraction for different modalities such as text, speech, and vision, designing modality fusion strategies at different levels, and constructing more robust and discriminative deep learning models. In existing multimodal fusion research, Zadeh et al. constructed a tensor fusion network (TFN) to explicitly model high-order interactions between different modalities; Liu et al. proposed a low-rank multimodal fusion method to reduce computational complexity during the fusion process; Majumder et al. designed a hierarchical feature fusion strategy to progressively learn interactions between modalities during the fusion process; Mai et al. modeled unimodal, bimodal, and trimodal interactions using a hierarchical graph fusion network; and Hazarika et al. balanced modality invariance with the learning of modality-specific representations to enhance multimodal representation capabilities.

[0004] Existing multimodal sentiment recognition methods still have certain shortcomings in complex long-dialogue scenarios. Sentiment expression in multi-turn dialogues typically exhibits strong long-range dependencies, and relying solely on local contextual information is insufficient to fully characterize its overall evolution. The high-level representations formed through cross-modal interactions often simultaneously contain core components closely related to sentiment judgment as well as spurious associations with only superficial statistical correlations. When the model lacks effective differentiation and constraints, its robustness and generalization ability will be affected. Furthermore, the contributions of different modalities to sentiment judgment dynamically fluctuate with changes in context. Using static concatenation or fixed-weight fusion methods makes it difficult to accurately reflect the true contributions of each modality in the current semantic environment. Therefore, it is necessary to design a multimodal sentiment recognition method based on decoupling enhancement and global modeling to address these issues. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings and deficiencies of the existing technologies mentioned above by proposing a multimodal emotion recognition method based on decoupling enhancement and global modeling. By setting up a bidirectional state space global temporal modeling branch and a causal heuristic decoupling enhancement branch in parallel, global temporal representation and causal heuristic decoupling enhancement representation are generated respectively. Then, an adaptive heterogeneous feature fusion module is used to realize the dynamic fusion of the two types of high-level information, thereby improving the accuracy, robustness and generalization ability of multimodal emotion recognition in complex long dialogue scenarios.

[0006] The technical solution adopted by this invention to solve its technical problem is: to provide a multimodal emotion recognition method based on decoupling enhancement and global modeling, the method comprising the following steps:

[0007] Step 1: Obtain the text sequence, speech sequence, and video sequence from the dialogue sample to be identified, extract text modal features, speech modal features, and visual modal features respectively, and map the three modal features to a unified latent space through a feature projection layer to obtain a multimodal input representation; based on the multimodal input representation, construct a contextual modeling unit to model the semantic dependency, emotional continuity, and cross-discourse interaction information between the current utterance and its preceding and following utterances, generating a multimodal contextual representation that integrates local context and global contextual information; the multimodal contextual representation serves as the input to the subsequent parallel dual-branch modeling structure;

[0008] Step 2: After the multimodal input representation or basic cross-modal interaction representation is constructed, a parallel two-branch modeling structure is built, synchronously executing the bidirectional state-space global temporal modeling branch and the causal heuristic decoupling enhancement branch, wherein:

[0009] The bidirectional state space global temporal modeling branch, based on the selective state space model, performs forward temporal modeling and backward temporal modeling on each modal sequence, and fuses the forward temporal representation and backward temporal representation to obtain a global temporal representation covering historical semantic information and future semantic information.

[0010] The causal-inspired decoupling enhancement branch first constructs a multimodal interaction representation through a cross-modal attention mechanism, then projects it onto a unified representation space through a shared mapping, and introduces a two-branch decoupling structure in this representation space to decompose the interaction representation into core components related to sentiment discrimination and potential false association components. The core component representation is strengthened through a residual enhancement mechanism, while the feature coupling between the core components and potential false association components is suppressed through orthogonal constraints, thereby obtaining a causal-inspired decoupling enhancement representation.

[0011] Step 3: Input the global temporal representation and the causal heuristic decoupled enhanced representation into the adaptive heterogeneous feature fusion module. Through the gating mechanism, the fusion weights are dynamically generated according to the semantic features of the input samples. The two types of representations are weighted and fused to obtain the fused representation. The fused representations of text modality, speech modality and visual modality are mapped into a unified decision vector, and the sentiment classification result is output.

[0012] Step 4: Construct a multi-objective joint optimization function to perform end-to-end training on the multimodal emotion recognition model, wherein the joint optimization function includes:

[0013] Classification loss used to constrain the consistency between predicted results and true labels. ;

[0014] Collaborative constraint loss used to enhance the statistical correlation between fused representations of different modalities ;

[0015] Contrastive learning loss used to improve the ability to distinguish imbalanced and hard-to-class samples. ;

[0016] Orthogonal constraint loss used to ensure that core components and potentially spuriously related components are independent. ;

[0017] The total loss function is expressed as ,

[0018] in, , and These represent the weight coefficients of the Soft-HGR constraint loss, the matrix-weighted focus contrast loss, and the orthogonal constraint loss, respectively.

[0019] Step 5: Evaluate the performance of the proposed multimodal emotion recognition method based on decoupling enhancement and global modeling;

[0020] Furthermore, the specific steps of step 1 are as follows:

[0021] Step 1-1: Obtain the text sequence, speech sequence and video sequence from the dialogue sample, wherein the video sequence is processed by face detection and alignment to obtain a facial expression frame sequence;

[0022] Steps 1-2: Encode the text sequence using the RoBERTa pre-trained language model and extract sentence-level semantic feature representations of the text modality;

[0023] Steps 1-3: Use the OpenSmile acoustic feature extraction tool to process the speech sequence and extract fine-grained acoustic feature representations of the speech modalities;

[0024] Steps 1-4: Use the ResNet-101 visual feature extraction network pre-trained on the ImageNet and VGGFace2 datasets to extract features from the facial expression frame sequence, obtaining the high-level spatial features of the visual modality. .

[0025] Steps 1-5: Align the dimensions of text modal features, speech modal features, and visual modal features through a linear mapping layer, and map them to a unified latent space to obtain text input representation, speech input representation, and visual input representation.

[0026] Furthermore, the specific steps of the bidirectional state-space global temporal modeling branch in step 2 are as follows:

[0027] Step 2A-1: Input the text input representation, speech input representation, and visual input representation into the bidirectional Mamba temporal modeling structure constructed based on selective state space units, respectively;

[0028] Step 2A-2: In each selective state space unit, the current modal input representation is first normalized, and the normalized input is divided into the main modeling branch and the gated modulation branch;

[0029] Steps 2A-3: In the main modeling branch, extract the temporal dynamic features of the current modality in sequence through linear mapping, one-dimensional convolution and state space modeling operations;

[0030] Step 2A-4: In the gated modulation branch, gate weights related to the current input are generated through linear mapping and nonlinear activation;

[0031] Step 2A-5: Modulate the output of the main modeling branch and the output of the gated modulation branch element by element, and output a unidirectional timing representation through linear mapping;

[0032] Step 2A-6: Perform a forward scan along the original time sequence to obtain the forward time sequence representation;

[0033] Step 2A-7: Perform reverse processing on the input sequence and then perform a reverse scan to obtain the reverse time sequence representation;

[0034] Step 2A-8: Fuse the forward and backward timing representations to obtain the global timing representation of the current mode.

[0035] The bidirectional scan and global timing representation in steps 2A-6 to 2A-8 can be expressed as follows:

[0036]

[0037]

[0038]

[0039] in, This represents the forward temporal representation of the m-th mode. This represents the reverse timing representation of the m-th mode. Let H_m represent the global temporal representation of the m-th mode, and let H_m represent the input representation of the m-th mode mapped to the unified latent space. Presentation layer normalization operation, and These represent the forward and backward selective state-space modeling units, respectively. This indicates a time reversal operation. This represents a linear mapping layer.

[0040] Furthermore, the specific steps of the causal-inspired decoupling enhancement branch in step 2 are as follows:

[0041] Step 2B-1: Using the target modality as the query vector and the other modalities as the source of key and value vectors, construct a cross-modal attention interaction path to obtain the intermediate interaction representation of the target modality;

[0042] Step 2B-2: Input the intermediate interaction representations of each modality into the shared mapping layer, project them onto the unified basic representation space, and obtain the common representation;

[0043] Step 2B-3: Construct two parallel branches based on the common representation. One branch is used to extract the core components related to sentiment discrimination, and the other branch is used to extract potential false association components.

[0044] Step 2B-4: Selectively decompose the common representation using a dual-branch separation mechanism to obtain the core component representation and the potential spurious association component representation, respectively;

[0045] Step 2B-5: Introduce a residual enhancement mechanism in the core component branch to supplement the core component representation with effective discriminative information from the common representation;

[0046] Step 2B-6: Suppress the overlap between core component representations and potentially spurious association component representations using orthogonal constraint loss;

[0047] Step 2B-7: Output the core component representation after residual enhancement and orthogonal constraints as the causal-inspired decoupling enhancement representation of the current mode.

[0048] The shared mapping, bi-branch separation, and residual enhancement processes in steps 2B-2 to 2B-7 can be represented as follows:

[0049] ,

[0050] ,

[0051] ,

[0052] in, Represents the common representation of the m-th mode. This represents the hyperbolic tangent activation function. and These represent the weight parameters and bias parameters of the common representation layer, respectively. This represents the intermediate interaction representation of the m-th modality. This represents the mask corresponding to the core component branch. and These represent the weight parameters and bias parameters of the core component branches, respectively. The mask representing the branch corresponding to potentially spurious related components. and These represent the weight and bias parameters of the spurious association component branch, respectively. The core component representation of the m-th mode. This represents the latent spurious association component of the m-th mode. This represents the core component of the m-th mode after residual enhancement. The causal-inspired decoupling enhancement representation for the m-th modality. This represents the Sigmoid activation function. This represents element-wise multiplication. Presentation layer normalization operation.

[0053] Furthermore, the specific steps of step 3 are as follows:

[0054] Step 3-1: Jointly input the global temporal representation of each modality with the corresponding causal heuristic decoupled enhancement representation to generate the fusion input required for gating weight calculation;

[0055] Step 3-2: Generate adaptive gating weights based on the fused input through a gating mapping layer;

[0056] Step 3-3: Use adaptive gating weights to weightedly combine the global temporal representation and the decoupled enhanced representation of causal heuristics to obtain the fused representation of the current modality;

[0057] Steps 3-4: After obtaining the text modality fusion representation, speech modality fusion representation, and visual modality fusion representation respectively, the three are concatenated and mapped to a unified decision space to obtain the final decision vector;

[0058] Steps 3-5: Input the final decision vector into the classifier and output the corresponding sentiment category prediction result.

[0059] Furthermore, the specific steps of step 4 are as follows:

[0060] Step 4-1: Construct a cross-entropy classification loss function based on the prediction distribution to constrain the consistency between the sentiment category prediction results and the true labels;

[0061] Step 4-2: Construct the Soft-HGR constrained loss to enhance the statistical synergy between the final fused representations of different modalities;

[0062] Step 4-3: Construct a matrix-weighted, dual-focus contrastive loss that acts on the final decision vector. To enhance the model's discriminative ability in scenarios with class imbalance and hard-to-distinguish samples, the matrix-weighted dual-focus contrastive loss is expressed as:

[0063] ,

[0064] Where N represents the number of samples in the training batch, i, j, and k represent the sample numbers in the training batch, and P(i) represents the set of positive samples of the same class as sample i. Let i represent the set of negative samples corresponding to sample i. This represents the class balance weights corresponding to sample i. This represents the predicted probability that sample i belongs to the true sentiment category. Indicates the focus adjustment parameter. This represents the sample-level weights between sample i and positive sample j. and Let represent the similarity between sample i and sample j, and between sample i and sample k, respectively. Represents an exponential function. Represents a logarithmic function.

[0065] Step 4-4: Construct orthogonal constraint loss for decoupling enhancement branches targeting potential spurious association components and causal heuristics. The orthogonal constraint loss is used to separate the representations of the core components after residual augmentation from potentially spurious correlation components.

[0066] ;

[0067] Where m represents text, speech, or visual modality, This represents the potential spurious association component of the m-th mode. This represents the core component of the m-th mode after residual enhancement. Indicates matrix transpose. Let t represent the Frobenius norm, and a, v represent the text, speech, and visual modal, respectively.

[0068] Steps 4-5: Apply cross-entropy classification loss Soft-HGR constraint loss Matrix-based dual-weighted focal contrast loss and orthogonal constraint loss We perform a weighted summation to form the total loss function L of the model, which is expressed as:

[0069] ;

[0070] in, , and These represent the weight coefficients of the Soft-HGR constraint loss, the matrix-weighted focus contrast loss, and the orthogonal constraint loss, respectively.

[0071] Steps 4-6: Perform end-to-end parameter updates for the entire multimodal emotion recognition network based on the total loss function.

[0072] Furthermore, the specific steps of step 5 are as follows:

[0073] Step 5-1: Perform an ablation experiment on the method described above and the method for removing the global temporal modeling branch in the bidirectional state space to verify the effectiveness of the global temporal modeling branch;

[0074] Step 5-2: Perform an ablation experiment on the method described above and the method for removing the causal-inspired decoupling enhancement branch to verify the effectiveness of the causal-inspired decoupling enhancement branch;

[0075] Step 5-3: Perform ablation experiments on the method described above and the method for removing the adaptive heterogeneous feature fusion mechanism to verify the effectiveness of the adaptive heterogeneous feature fusion module;

[0076] Step 5-4: Discuss the specific values ​​of each loss weight in the joint optimization and explore the optimal parameter configuration.

[0077] Beneficial effects:

[0078] 1. This invention uses a bidirectional state-space model to jointly model historical and future semantics, which can better represent the global emotion evolution process in complex long dialogues and improve the model's ability to model long-range dependencies.

[0079] 2. This invention uses a causal-inspired decoupling mechanism to structurally separate core emotional components from potential false association components, and uses orthogonal constraints to suppress feature overlap between the two types of components, thereby improving the robustness and interpretability of the model.

[0080] 3. This invention dynamically fuses global temporal information and causal heuristic decoupling enhancement information through an adaptive heterogeneous feature fusion module. It can automatically adjust the contribution ratio of the two types of high-level information according to the semantic features of different samples, thereby improving the model's adaptability to different scenarios and samples.

[0081] 4. This invention improves the overall performance of multimodal emotion recognition in complex long dialogue scenarios by jointly optimizing cross-entropy classification loss, soft-HGR constraint loss, matrix-based dual-weight focus contrast loss and orthogonal constraint loss, thus taking into account classification performance, modal synergy, class balance and feature separability. Attached Figure Description

[0082] Figure 1 This is a schematic diagram of the operation process of the present invention;

[0083] Figure 2 Here is a diagram of the Mamba module structure;

[0084] Figure 3 A parallel fusion structure that enhances the decoupling of global temporal modeling and causal heuristics (taking text modality as an example). Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be noted that, to avoid obscuring the invention with unnecessary details, only structures and / or processing steps closely related to the present invention are shown in the drawings, while other details not directly related to the invention are omitted.

[0086] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0087] like Figures 1 to 3 As shown, this invention proposes a multimodal emotion recognition method based on decoupling enhancement and global modeling. This method targets text, speech, and visual multimodal emotion recognition tasks in complex long-dialogue scenarios. The overall approach consists of a multimodal feature input module, a bidirectional state-space global temporal modeling module, a causal-heuristic decoupling enhancement module, an adaptive heterogeneous feature fusion module, and a multi-objective joint optimization module. The bidirectional state-space global temporal modeling module can be denoted as BiMamba, and the selective state-space model can be denoted as SSM.

[0088] In this embodiment, let a dialogue sample to be identified be... Where u_i represents the i-th utterance. Indicates the length of the dialogue. For each utterance... Obtain the text modality sequence respectively Speech modal sequences and visual modal sequences After feature extraction, the text modal features are denoted as follows: Speech modal features are denoted as Visual modal features are denoted as The three modes are collectively denoted as ,in .

[0089] Step 1: Obtain the text modality features, speech modality features, and visual modality features from the dialogue sample to be recognized, and map the three modalities to a unified latent space through a feature projection layer to obtain the corresponding multimodal input representation.

[0090] Step 1-1: Obtain the text sequence from the dialogue sample Speech sequences and video sequences Among them, video sequences After face detection and alignment, a sequence of facial expression frames is obtained.

[0091] Steps 1-2: Use the RoBERTa pre-trained language model to process the text sequence Encode and extract sentence-level semantic features of the text modality. .

[0092] Steps 1-3: Use the OpenSmile acoustic feature extraction tool to process the speech sequence. Processing is performed to extract fine-grained acoustic features of the speech modalities. .

[0093] Steps 1-4: Use the ResNet-101 visual feature extraction network pre-trained on the ImageNet and VGGFace2 datasets to extract features from the facial expression frame sequence and obtain the high-level spatial features x_v of the visual modality.

[0094] Steps 1-5: Pass through the linear mapping layer respectively , and Will , and Mapping to the unified latent space yields the text input representation. Voice input representation and visual input representation This process can be represented as ,in, , and These represent the linear mapping layers corresponding to text, speech, and visual modalities, respectively. , and These represent the input representations of text, speech, and visual modalities mapped to the unified latent space, respectively. Let m be the unified latent space input representation of the m-th mode. This represents the feature projection layer corresponding to the m-th mode.

[0095] Step 2: After the modal input and basic interaction representation are constructed in the early stage, execute the bidirectional state space global temporal modeling branch and the causal heuristic decoupling enhancement branch in parallel.

[0096] Step 2-1: Input the text to represent Voice input representation and visual input representation Input the BiMamba modules constructed based on selective state space units.

[0097] Step 2-2: In each selective state-space unit, represent the current modal input. Perform layer normalization to obtain the normalized representation. and will It is divided into a main modeling branch and a gating modulation branch, among which... The m-th modal input represents the result after layer normalization.

[0098] Steps 2-3: In the main modeling branch, Temporal dynamic modeling is performed sequentially using linear mapping, one-dimensional convolution, and SSM to obtain the main branch output. ,in, This represents the temporal dynamic characteristics obtained by the main modeling branch of the m-th mode.

[0099] Steps 2-4: In the gated modulation branch, Gating weights related to the current input are generated through linear mapping and nonlinear activation functions. ,in, This represents the gated modulation weight for the m-th mode.

[0100] Steps 2-5: Output the main branch With gate weights Element-wise modulation is performed, and a unidirectional time-series representation is obtained through linear mapping. ,in, This represents the timing representation of the m-th mode in a single scanning direction.

[0101] Steps 2-6: Perform a forward scan along the original time sequence to obtain the forward time sequence representation. ,in, This represents the temporal representation of the m-th mode obtained by modeling from the historical direction to the current direction.

[0102] Steps 2-7: After reversing the input sequence, perform a reverse scan to restore the original timing sequence and obtain the reverse timing representation. ,in, This represents the temporal representation of the m-th mode obtained by modeling from the direction the m-th mode has never arrived.

[0103] Steps 2-8: [Regarding...] and By performing fusion, a global temporal representation of the current modality is obtained. This global temporal representation is used to characterize the trajectory of emotional evolution in complex, long dialogues;

[0104] in, , Represents a linear mapping layer. This indicates a feature concatenation operation. This represents the input representation of the m-th mode after mapping to the unified latent space.

[0105] Steps 2-9: In the causal-inspired decoupling enhancement branch, using the target modality as the query vector Q and the other modalities as the key vector K and value vector V sources, a cross-modal attention interaction path is constructed to obtain the intermediate interaction representation of the target modality. ,in, This represents the intermediate interaction representation after the m-th modality is fused with supplementary information from other modalities.

[0106] Step 2-10: Taking text modality as an example, represent the text input. as query vector Voice input represents and visual input representation Using key vectors and value vectors respectively, text-speech attention paths and text-visual attention paths are established to obtain supplementary speech information. and visual supplementary information ;Will , and The data is then jointly integrated and processed through a feedforward network and layer normalization to obtain an intermediate interactive representation of the text modality. The speech modality and the visual modality are obtained in the same way. and ,in, This indicates supplementary information obtained by the text modality from the speech modality. This represents supplementary information obtained by the text modality from the visual modality.

[0107] Step 2-11: Represent the intermediate interactions of each modality Input shared mapping layer Projecting onto a unified fundamental representation space yields a common representation. ,in , Indicates a shared mapping layer. and These represent the weight parameters and bias parameters of the shared mapping layer, respectively. This represents the hyperbolic tangent activation function.

[0108] Step 2-12: Based on common representation Construct two parallel branches, where the core component branch is used to extract core components related to sentiment discrimination. The potential spurious association branch is used to extract components with potential spurious associations caused by modal co-occurrence, environmental noise, or data bias. .

[0109] Step 2-13: Using a two-branch separation mechanism Selective decomposition was performed to obtain the core component representations. and potential spurious association components ,in , , and These represent the masks corresponding to the core component branches and the potential spurious association branches, respectively. This indicates element-wise multiplication.

[0110] Step 2-14: Introduce a residual enhancement mechanism in the core component branch, and... The effective discrimination information in the middle is supplemented. In the middle, the enhanced core components and through orthogonal constraint loss inhibition and The degree of overlap between them, This represents the orthogonal constraint loss.

[0111] Step 2-15: The core components after residual enhancement and orthogonal constraints... Decoupling Enhancement Representation Inspired by Causality as the Current Modality Output, where , Presentation layer normalization operation.

[0112] It should be noted that the causal heuristic decoupling enhancement in this embodiment does not rely on explicit causal graphs, intervention variables, or strict causal effect estimation of counterfactual samples. Instead, it draws on the idea of ​​distinguishing between effective factors and unstable related factors from the perspective of stable representation learning, and reduces the degree of aliasing between sentiment discrimination core components and potential false association components in the representation space through bi-branch separation, residual enhancement, and orthogonal constraints.

[0113] Step 3: Input the global temporal representation obtained in Step 2 and the causal heuristic decoupled enhanced representation into the adaptive heterogeneous feature fusion module, perform dynamic gating weighted fusion on the two types of representations, map the fusion result into a unified decision representation, and output the sentiment classification result.

[0114] Step 3-1: For each mode m, represent the global timing sequence. Decoupling Enhancement Representation with Causal Heuristics Perform joint input to obtain fused input. ,in, This represents the fusion input corresponding to the m-th mode.

[0115] Step 3-2: Merge the input Input to an adaptive gating network to generate gating weights ,in , This represents the Sigmoid activation function. This represents a multilayer perceptron.

[0116] Step 3-3: Utilizing Gating Weights right and By performing weighted combination, a fused representation of the current modality is obtained. This process can be represented as The adaptive heterogeneous feature fusion module is used to dynamically adjust the contribution ratio of global temporal information and causal heuristic decoupling enhancement information according to the semantic environment of different samples.

[0117] Steps 3-4: Obtain the text modality fusion representations respectively Speech modality fusion representation Visual modality fusion representation Then, the three are concatenated and projected onto a unified decision space via linear mapping to obtain the final decision vector Y, where... , This represents a linear mapping layer.

[0118] Steps 3-5: Input the final decision vector Y into the classifier and obtain the sentiment category prediction distribution P through the Softmax function. The category with the highest probability in P is the sentiment classification result of the current dialogue sample, where P represents the predicted probability distribution of each sentiment category.

[0119] Step 4: Based on the constructed joint loss function, train and optimize the multimodal emotion recognition model to improve the accuracy, robustness and generalization ability of the model in complex long dialogue scenarios.

[0120] Step 4-1: Construct the cross-entropy classification loss based on the prediction distribution P , is used to constrain the consistency between the predicted sentiment category P and the true label y, where y represents the true sentiment category label of the current sample.

[0121] Step 4-2: Construct the Soft-HGR constrained loss to enhance the statistical synergy among the final fused representations of different modalities. ,in, Used to constrain statistical synergies between text, speech, and visual modal fusion representations.

[0122] Step 4-3: Construct a matrix-weighted, dual-focus contrastive loss that acts on the final decision vector Y. To enhance the model's discriminative ability in scenarios with class imbalance and hard-to-distinguish samples, the matrix-weighted dual-focus contrastive loss is expressed as:

[0123] ;

[0124] Where N represents the number of samples in the training batch, i, j, and k represent the sample numbers in the training batch, and P(i) represents the set of positive samples of the same class as sample i. Let i represent the set of negative samples corresponding to sample i. This represents the class balance weights corresponding to sample i. This represents the predicted probability that sample i belongs to the true sentiment category. Indicates the focus adjustment parameter. This represents the sample-level weights between sample i and positive sample j. and Let represent the similarity between sample i and sample j, and between sample i and sample k, respectively. Represents an exponential function. Represents a logarithmic function.

[0125] Step 4-4: Construct orthogonal constraint loss for decoupling enhancement branches targeting potential spurious association components and causal heuristics. The orthogonal constraint loss is used to constrain the representation separation between the core components after residual enhancement and the potentially spurious correlation components.

[0126] ;

[0127] Where m represents text, speech, or visual modality, This represents the potential spurious association component of the m-th mode. This represents the core component of the m-th mode after residual enhancement. Indicates matrix transpose. This represents the Frobenius norm.

[0128] Steps 4-5: Apply cross-entropy classification loss Soft-HGR constraint loss Matrix-based dual-weighted focal contrast loss and orthogonal constraint loss We perform a weighted summation to form the total loss function L of the model, which is expressed as:

[0129] ;

[0130] in, , and These represent the weight coefficients of the Soft-HGR constraint loss, the matrix-weighted focus contrast loss, and the orthogonal constraint loss, respectively.

[0131] Steps 4-6: Perform end-to-end parameter updates for the entire multimodal emotion recognition network based on the total loss function.

[0132] Step 5: Evaluate the performance of the proposed multimodal emotion recognition method based on decoupling enhancement and global modeling.

[0133] Step 5-1: Conduct experimental validation on the IEMOCAP dataset, and use the weighted F1 score W-F1 and accuracy Acc as evaluation metrics.

[0134] Step 5-2: In one embodiment, the model is implemented based on the Python 3.8 and PyTorch 1.12 deep learning framework. The unified latent space dimension is set to 256, the internal state dimension of the BiMamba module is set to 16, the convolutional kernel size is set to 4, the multilayer perceptron in the causal heuristic decoupling enhancement module adopts a two-layer structure and the hidden layer dimension is set to 128, the optimizer adopts AdamW, and the initial learning rate is set to 0.0001.

[0135] Step 5-3: Perform ablation experiments on the BiMamba module, the causal-inspired decoupled attention mechanism, and the orthogonal constraint loss. Construct experimental configurations for removing the BiMamba module, removing the causal-inspired decoupled attention mechanism, replacing the causal-inspired decoupled attention mechanism with standard attention, removing the orthogonal constraint loss, and retaining all modules.

[0136] Step 5-4: Compare the ablation experiments under the same dataset, evaluation metrics, and training conditions. Analyze the impact of the BiMamba module, the causal-inspired decoupling attention mechanism, and the orthogonal constraint loss on model performance based on the changes in W-F1 and Acc, in order to verify the effectiveness of the global temporal modeling, causal-inspired decoupling enhancement, and orthogonal constraint optimization.

[0137] Ablation experiments were conducted on BiMamba, causal-inspired decoupled attention, and orthogonal constraint loss, and the results are shown in Table 1:

[0138] Table 1 - Ablation Experiment Results on the IEMOCAP Dataset

[0139]

[0140] Using the complete model (No. 5, W-F1 at 71.13%, Acc at 70.92%) as a baseline, removing the bidirectional state-space model (BiMamba) resulted in the most significant performance decline (No. 2), with W-F1 and Acc dropping to 69.08% and 68.88% respectively, a decrease of 2.05% and 2.04%. This confirms BiMamba's core supporting role in capturing the dynamic evolution of long-sequence dialogues and providing a global macro-context foundation. Validation of the causal-inspired decoupled attention mechanism shows that removing this module (No. 1) reduces W-F1 and Acc by 1.13% and 0.99% respectively. Further replacing the causal-inspired decoupled attention with standard attention (No. 4) degrades performance to 69.69% and 69.44%. This comparison vividly reveals the generalization bottleneck of traditional attention due to its difficulty in distinguishing stable discriminative cues from spurious relevance, highlighting the superiority of causal-inspired decoupled attention in targeted enhancement of core emotional cues. Furthermore, the removal of orthogonal constraint loss (Section 3) led to a decrease in W-F1 and Acc by 0.55% and 0.19%, respectively, indicating that this mechanism, by explicitly introducing orthogonality constraints into the optimization system, ensures the purity of feature decoupling and the robustness of decision-making. The global modeling undertaken by BiMamba and the decoupling enhancement achieved through causal-inspired decoupling attention together constitute the dominant factors for performance improvement. Combined with the underlying guarantee of orthogonal constraints, these three elements achieve effective synergistic gains within a unified architecture.

[0141] In summary, this embodiment obtains the global timing representation through the BiMamba module. A causal-inspired decoupling enhancement representation is obtained through a causal-inspired decoupling enhancement module. A fused representation is generated through an adaptive heterogeneous feature fusion module. and the final decision vector Y, and through , , and The model is trained using a joint loss function, which enables it to take into account long-range dependency modeling, extraction of core sentiment cues, dynamic fusion of heterogeneous features, and robust decision optimization.

[0142] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-modal emotion recognition method based on decoupled augmentation and global modeling, characterized in that, The method includes the following steps: Step 1: Obtain the text sequence, speech sequence and video sequence in the dialogue sample to be identified, extract the text modal features, speech modal features and visual modal features respectively, and map the three modal features to a unified latent space through a feature projection layer to obtain a multimodal input representation; Step 2: Construct a parallel dual-branch modeling structure, synchronously executing the bidirectional state-space global temporal modeling branch and the causal heuristic decoupling enhancement branch, wherein: The bidirectional state-space global temporal modeling branch is based on the selective state-space model and performs bidirectional temporal modeling on each modal sequence to obtain a global temporal representation; The causal-inspired decoupling enhancement branch constructs a multimodal interaction representation through a cross-modal attention mechanism, and uses a dual-branch decoupling structure to separate core components from potential spurious association components in a unified representation space. The causal-inspired decoupling enhancement representation is obtained through residual enhancement and orthogonal constraints. Step 3: Input the global temporal representation and the decoupled enhanced representation of causal heuristic into the adaptive heterogeneous feature fusion module. Through the gating mechanism, the fusion weights are dynamically generated according to the semantic features of the input samples. The two types of representations are weighted and fused to obtain the fused representation. The fused representations of text modality, speech modality and visual modality are mapped into a unified decision vector and the sentiment classification result is output. Step 4: Construct a multi-objective joint optimization function to perform end-to-end training on the multimodal emotion recognition model; Step 5: Perform a performance evaluation on the above steps.

2. The method of claim 1, wherein, Step 1 includes the following steps: Step 1-1: Obtain the text sequence, speech sequence and video sequence from the dialogue sample. The video sequence is processed by face detection and alignment to obtain the facial expression frame sequence. Steps 1-2: Encode the text sequence using the RoBERTa pre-trained language model and extract sentence-level semantic feature representations of the text modality; Steps 1-3: Use the OpenSmile acoustic feature extraction tool to process the speech sequence and extract fine-grained acoustic feature representations of the speech modalities; Step 1-4: Feature extraction on facial expression frame sequences using a ResNet-101 visual feature extraction network pre-trained on ImageNet and VGGFace2 dataset, resulting in high-level spatial features of visual modality ; Steps 1-5: Align the dimensions of text modal features, speech modal features, and visual modal features through a linear mapping layer, and map them to a unified latent space to obtain text input representation, speech input representation, and visual input representation.

3. The method of claim 1, wherein, The specific steps for implementing the bidirectional state-space global temporal modeling branch in step 2 include: Step 2A-1: Input the text input representation, speech input representation, and visual input representation into the bidirectional Mamba temporal modeling structure constructed based on selective state space units, respectively; Step 2A-2: In each selective state space unit, the current modal input representation is first normalized, and the normalized input is divided into the main modeling branch and the gated modulation branch; Steps 2A-3: In the main modeling branch, extract the temporal dynamic features of the current modality in sequence through linear mapping, one-dimensional convolution and state space modeling operations; Step 2A-4: In the gated modulation branch, gate weights related to the current input are generated through linear mapping and nonlinear activation; Step 2A-5: Modulate the output of the main modeling branch and the output of the gated modulation branch element by element, and output a unidirectional timing representation through linear mapping; Step 2A-6: Perform a forward scan along the original time sequence to obtain the forward time sequence representation; Step 2A-7: Perform reverse processing on the input sequence and then perform a reverse scan to obtain the reverse time sequence representation; Step 2A-8: Fuse the forward and backward timing representations to obtain the global timing representation of the current mode; The bidirectional scan and global timing representation in steps 2A-6 to 2A-8 are expressed as follows: ， ， ， wherein, represents a forward temporal representation of the m-th modality, represents a backward temporal representation of the m-th modality, represents a global temporal representation of the m-th modality, represents an input representation of the m-th modality mapped to a unified latent space, represents a layer normalization operation, and represent forward and backward selective state space modeling units, respectively, represents a time reversal operation, represents a linear mapping layer.

4. The method according to claim 1, characterized in that, The specific steps for implementing the causal heuristic decoupling enhancement branch in step 2 include: Step 2B-1: Using the target modality as the query vector and the other modalities as the source of key and value vectors, construct a cross-modal attention interaction path to obtain the intermediate interaction representation of the target modality; Step 2B-2: Input the intermediate interaction representations of each modality into the shared mapping layer, project them onto the unified basic representation space, and obtain the common representation; Step 2B-3: Construct two parallel branches based on the common representation. One branch is used to extract the core components related to sentiment discrimination, and the other branch is used to extract potential false association components. Step 2B-4: Selectively decompose the common representation using a dual-branch separation mechanism to obtain the core component representation and the potential spurious association component representation, respectively; Step 2B-5: Introduce a residual enhancement mechanism in the core component branch to supplement the core component representation with effective discriminative information from the common representation; Step 2B-6: Suppress the overlap between core component representations and potentially spurious association component representations using orthogonal constraint loss; Step 2B-7: Output the core component representation after residual enhancement and orthogonal constraint as the causal-inspired decoupling enhancement representation of the current mode; The shared mapping, bi-branch separation, and residual enhancement processes in steps 2B-2 to 2B-7 are represented as follows: ， ， ， in, Represents the common representation of the m-th mode. This represents the hyperbolic tangent activation function. and These represent the weight parameters and bias parameters of the common representation layer, respectively. This represents the intermediate interaction representation of the m-th modality. This represents the mask corresponding to the core component branch. and These represent the weight parameters and bias parameters of the core component branches, respectively. The mask representing the branch corresponding to potentially spurious related components. and These represent the weight and bias parameters of the spurious association component branch, respectively. The core component representation of the m-th mode. This represents the latent spurious association component of the m-th mode. This represents the core component of the m-th mode after residual enhancement. The causal-inspired decoupling enhancement representation for the m-th modality. This represents the Sigmoid activation function. This represents element-wise multiplication. Presentation layer normalization operation.

5. The method according to claim 1, characterized in that, Step 3 includes the following specific steps: Step 3-1: Jointly input the global temporal representation of each modality with the corresponding causal heuristic decoupled enhancement representation to generate the fusion input required for gating weight calculation; Step 3-2: Generate adaptive gating weights based on the fused input through a gating mapping layer; Step 3-3: Use adaptive gating weights to weightedly combine the global temporal representation and the decoupled enhanced representation of causal heuristics to obtain the fused representation of the current modality; Steps 3-4: After obtaining the text modality fusion representation, speech modality fusion representation, and visual modality fusion representation respectively, the three are concatenated and mapped to a unified decision space to obtain the final decision vector; Steps 3-5: Input the final decision vector into the classifier and output the corresponding sentiment category prediction result.

6. The method according to claim 1, characterized in that, Step 4 includes the following specific steps: Step 4-1: Construct a cross-entropy classification loss function based on the prediction distribution to constrain the consistency between the sentiment category prediction results and the true labels; Step 4-2: Construct the Soft-HGR constrained loss to enhance the statistical synergy between the final fused representations of different modalities; Step 4-3: Construct a matrix-weighted, dual-focus contrastive loss that acts on the final decision vector. To enhance the model's discriminative ability in scenarios with class imbalance and hard-to-distinguish samples, the matrix-weighted dual-focus contrastive loss is expressed as: ， Where N represents the number of samples in the training batch, i, j, and k represent the sample numbers in the training batch, and P(i) represents the set of positive samples of the same class as sample i. Let i represent the set of negative samples corresponding to sample i. This represents the class balance weights corresponding to sample i. This represents the predicted probability that sample i belongs to the true sentiment category. Indicates the focus adjustment parameter. This represents the sample-level weights between sample i and positive sample j. and Let represent the similarity between sample i and sample j, and between sample i and sample k, respectively. Represents an exponential function. Represents a logarithmic function; Step 4-4: Construct orthogonal constraint loss for decoupling enhancement branches targeting potential spurious association components and causal heuristics. The orthogonal constraint loss is used to separate the representations of the core components after residual enhancement from potentially spurious correlation components. ； Where m represents text, speech, or visual modality, This represents the potential spurious association component of the m-th mode. This represents the core component of the m-th mode after residual enhancement. Indicates matrix transpose. Let t represent the Frobenius norm, and a, v correspond to text, speech, and visual modalities, respectively. Steps 4-5: Apply cross-entropy classification loss Soft-HGR constraint loss Matrix-based dual-weighted focal contrast loss and orthogonal constraint loss We perform a weighted summation to form the total loss function L of the model, which is expressed as: ； in, , and These represent the weight coefficients of the Soft-HGR constraint loss, the matrix-weighted focus contrast loss, and the orthogonal constraint loss, respectively. Steps 4-6: Perform end-to-end parameter updates for the entire multimodal emotion recognition network based on the total loss function.

7. The method according to claim 1, characterized in that, The bidirectional state-space global temporal modeling branch and the causal heuristic decoupling enhancement branch take the same set of multimodal input representations or basic interaction representations as parallel inputs, and output the global temporal representation and the causal heuristic decoupling enhancement representation respectively. The contribution ratio of the two types of representations in the final decision representation is determined by adaptive gating weights.

Images