Multi-modal emotion recognition method with affective feature reversible decoupling and adaptive fusion

Abstract

This invention discloses a multimodal emotion recognition method with reversible decoupling and adaptive fusion of emotion features. Specifically, it involves: acquiring multimodal dataset samples, dividing them into training, validation, and test sets; constructing an emotion recognition model, which includes a multimodal feature encoding module, a reversible attention mask decoupling module, a mutual information constraint mechanism module, and a semantically guided adaptive feature fusion module; training the emotion recognition model using the training set, evaluating the model on the validation set, calculating corresponding performance metrics, and saving model parameters whose validation performance surpasses the historical best results; and testing the optimal model on the test set to output the corresponding emotion recognition results. This method can explicitly separate emotional semantics from modal redundancy noise, effectively improving the interpretability of the decision-making process, and can fully capture fine-grained complementary information across modalities, exhibiting higher accuracy and robustness in complex scenarios.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition technology, specifically involving a multimodal emotion recognition method with reversible decoupling and adaptive fusion of emotion features. Background Technology

[0002] With the development of social media and human-computer interaction technologies, multimodal emotion recognition has become a research hotspot and has been widely applied in various fields such as safe driving, distance education, and healthcare. This technology aims to identify the emotional state of a specific object by fusing information from multiple modalities, including text, vision, and speech.

[0003] However, existing multimodal emotion recognition technologies face significant challenges. First, inherent heterogeneity exists between modalities, with data from different modalities exhibiting significant differences in representation and distribution patterns. This leads to emotional semantic information often becoming entangled with modality-specific non-emotional noise (such as background noise and irrelevant visual textures) during the feature learning stage. Existing methods struggle to effectively separate emotional semantics from noise, limiting the interpretability and generalization ability of the model. Second, in the feature fusion stage, existing methods typically employ simple concatenation operations or coarse-grained attention mechanisms. These methods fail to fully capture fine-grained emotional semantic relationships between modalities within complex cross-modal contexts, resulting in insufficient discriminative power in the fused emotion representation and impacting the final emotion recognition accuracy.

[0004] As can be seen from the above, there are two main challenges in this field: (1) how to design an effective feature learning mechanism to achieve explicit separation of emotional semantic information and modality-specific noise from heterogeneous data; and (2) how to construct a more discriminative fusion strategy to fully realize fine-grained emotional semantic interaction between modalities. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal emotion recognition method with reversible decoupling and adaptive fusion of emotion features, which effectively reduces the interference of modal redundancy noise on emotion recognition and improves the decoupling effect of features.

[0006] The technical solution adopted in this invention is a multimodal emotion recognition method with reversible decoupling and adaptive fusion of emotion features, which is implemented according to the following steps: Step 1: Obtain multimodal dataset samples, which are divided into training set, validation set and test set; Step 2: Construct an emotion recognition model, which includes a multimodal feature encoding module, a reversible attention mask decoupling module, a mutual information constraint mechanism module, and a semantically guided adaptive feature fusion module. Step 3: Train the emotion recognition model using the training set. After each round of training, evaluate the model on the validation set, calculate the corresponding performance indicators, and save the model parameters that have better validation performance than the historical best results. Use the best emotion recognition model to test on the test set and output the corresponding emotion recognition results.

[0007] The invention is further characterized in that, In step 1, the multimodal datasets include the CMU-MOSI dataset, the CMU-MOSEI dataset, and the UR-FUNNY dataset.

[0008] Step 2 specifically involves: Step 2.1: Use the multimodal feature encoding module to preprocess the multimodal dataset to obtain the original data of each modality, and then perform multimodal feature encoding. Step 2.2: Construct a reversible attention mask decoupling module for arbitrary single-modal features. We utilize invertible neural networks to construct a bidirectional invertible mapping relationship between implicit representations and sentiment semantic factors; combined with an attention masking mechanism, we explicitly decouple the implicit features into two parts through a forward transformation function, and then perform inverse reconstruction. Step 2.3: Construct a mutual information constraint mechanism module; use relevant mutual information and conditional mutual information to constrain the decoupled features; Step 2.4: Build a semantically guided adaptive feature fusion module; utilize the shared features obtained from decoupling. As contextual information, the unique features obtained through the residual structure are... Semantic guidance and correction are performed; a dual-branch prediction structure is constructed, which makes predictions based on the enhanced shared features and the corrected specific features respectively, and weights are assigned to the two branches through a gating mechanism, and the final sentiment prediction result is obtained by fusion. Step 2.5: Construct the overall optimization objective function.

[0009] Step 2.1 specifically involves: Step 2.11: Obtain sample sets using the CMU-MOSI, CMU-MOSEI, and UR-FUNNY datasets. For each video segment, extract the raw features of the three modalities, and use Librosa to extract Mel spectrograms to obtain acoustic features. Visual features are extracted using a pre-trained EfficientNet. Use BERT pre-trained models to obtain text features ; Step 2.12: Design unified feature encoding for acoustic feature mode a, text feature mode t, and visual feature mode v; specifically: The input sequences of each modality are projected to a unified dimension through one-dimensional convolution; learnable [CLS] tags and positional encodings are added and input into a multi-layer Transformer encoder. For text feature modalities, BERT is used to obtain word vector representations; for visual and audio feature modalities, frame-level features are directly input. The temporal context features of each modality are obtained after passing through the Transformer encoder. As shown in equation (1): (1) in, The features of the initial input for each modality, For convolution projection, For position encoding; Temporal context features of each modality The Transformer encoder is used again for joint modeling to obtain the final encoded features of each modality. .

[0010] Step 2.2 involves the following specific process: Step 2.21: The encoded features of each modality are processed through a two-layer MLP network to obtain modality-related features and task-related features, which are then concatenated to form implicit features. Introduce a set of learnable reflection vectors. Construct orthogonal matrices Achieve orthogonal blending along the channel dimension to obtain features. As shown in equation (2): (2) in, For the first One reflection vector, For feature dimension, It is the identity matrix; Step 2.22: After orthogonal mixing, adjust the output features. After layer normalization, we get and Subsequently, through reversible forward transformation and channel rearrangement operations, the following was obtained: The forward and reverse transformations and channel rearrangement process are expressed as equations (3)-(5): (3) (4) (5) Equation (3) represents the forward transformation, and equation (4) represents the inverse transformation. Implemented by two layers of MLP, and Orthogonal characteristics After layer normalization, the two features are divided according to the channel dimension. Forward transformation output characteristics and The splicing characteristics are represented as , The channel permutation matrix is ​​fixed but randomly initialized. For batch size, for The transpose of the matrix; Step 2.23: Repeat steps 2.21 and 2.22 four times; Step 2.24, following step 2.23, obtains the emotional identity modality. The query is obtained by constructing linear projections of the features respectively. ,key ,value Then calculate right Explanation weight as well as right Explanation weight Furthermore, information fusion is performed across each channel dimension to achieve complementary updates between features, thereby obtaining shared features of cross-modal consistent sentiment information. and the unique characteristics of each modality's unique attributes The process is represented by equations (6)-(8): (6) (7) (8) in, They are respectively The query, key, and value vectors after projection through the linear layer. Corresponding to The weight matrix of the query, key, and value vectors projected through the linear layer. They are respectively The query, key, and value vectors after projection through the linear layer. They are respectively The weight matrix of the query, key, and value vectors projected through the linear layer. for right diagonal weights, for right diagonal weights, for The transpose matrix of the key vectors. for The transpose matrix of the key vectors. The dimension of the key vector. This indicates the extraction of the main diagonal elements from the attention weight matrix; Step 2.25: Design four loss function constraints during the decoupling process, including reconstruction constraints. Reversible constraints Similarity constraints and independence constraints , expressed as equations (9)-(14): (9) (10) (11) (12) (13) (14) in, These are the implicit features of each modality. To obtain the temporal context features of each modality after passing through the Transformer encoder, It is the inverse transformation function. Represented as The square of the norm, For in the interval Random samples on for Norm, Expressing expectations, Let and represent two sets of random variables whose statistical independence is to be measured. Indicates the first The central moment of the first order, The Gram matrix between samples, For batch size, For a centered matrix, and Representing different modes.

[0011] Step 2.3 specifically involves: Step 2.31: Apply constraints to the shared features of each modality, as well as to the shared features and labels of each modality, and define the relevant mutual information loss. For equation (15): (15) in, , For scoring networks of MLP, Embedding emotional tags, ; Step 2.32, Design Conditions and Mutual Information Constraints A binary discriminant is constructed using the discriminant variational approximation. To distinguish between joint samples and conditional multiplication integral distribution samples, conditional mutual information can be expressed as equation (16): (16) in, Representing the unique characteristics of each modality, It is an embedded representation of sentiment tags. These are features shared across modalities. Let represent the expectation of a sample under the true joint distribution. This represents the expectation of a sample under a conditionally independent negative sample distribution. Indicates the same condition The variables are obtained by selecting hard negative samples in the neighborhood of the target neighborhood using KNN sampling; Step 2.33: Minimize the conditional mutual information loss. , expressed as equation (17): (17) CMI loss for and The sum of losses is shown in equation (18): (18).

[0012] Step 2.4 specifically involves: Step 2.41: Utilize shared features As a guide for the following text, the unique features are analyzed through residual structure. Make corrections; firstly, Enhancement Then, through interaction using a linear residual structure, unique features guided by shared semantics are obtained. , expressed as equation (19): (19) in, These are learnable parameters; Step 2.42: Design a two-branch prediction structure, sharing branches. Based on enhanced shared features Predicting sentiment tendencies, a unique branch guided by shared semantics. Based on the modified unique features Capture complementary details; finally, calculate weighting coefficients using a gating mechanism. Adaptive weighted ensemble of two prediction results The process is expressed as equation (20): (20) in, It is the Sigmoid activation function. These are the weighting coefficients. For MLP prediction heads with shared branches, This is an MLP prediction head with a unique branch guided by shared semantics.

[0013] Step 2.5 specifically involves: Construct the overall optimization objective function Introducing task loss As the primary supervisor, mean squared error is used as the task loss in regression tasks, while cross-entropy loss is used in classification tasks. The overall optimization objective is to combine the loss terms from steps 2.25 and 2.33 with the task loss into a joint optimization objective function, expressed as equation (21): (twenty one) in, For reversible constraint loss hyperparameters, For similarity loss hyperparameters, Loss of independence hyperparameters, To reconstruct the loss hyperparameters, For mutual information loss Hyperparameters.

[0014] The beneficial effects of this invention are: (1) The method of this invention designs a reversible attention mask decoupling module (IAMD), which constructs a bidirectional mapping between implicit features and semantic factors based on a reversible neural network. This mechanism overcomes the problem that traditional decoupling methods (such as feature projection) are prone to causing the loss of original emotional information, and can explicitly decouple implicit features into cross-modal consistent shared features and unique features that retain the unique attributes of each modality. Through this structured explicit separation, the loss of emotional semantic information is reduced, making the decision basis of the model in the feature space clearer, and significantly enhancing the interpretability of the model.

[0015] (2) The method of this invention constructs a mutual information constraint mechanism (MIC) to further optimize feature representation from an information theory perspective. By maximizing the relevance mutual information (RMI) between shared features and between shared features and sentiment labels, the model is forced to aggregate shared features of each modality to the same sentiment semantic center, thereby enhancing semantic consistency modeling. At the same time, by minimizing the conditional mutual information (CMI) between specific features and sentiment labels, the model is forced to strip away modality-specific attributes (such as background noise) that are irrelevant to the sentiment task, effectively reducing the interference of modality redundant noise on sentiment recognition and improving the decoupling effect of features.

[0016] (3) To address the insufficient interaction of existing fusion methods, this invention proposes a semantically guided adaptive feature fusion module (SGAFF). This module utilizes the pure shared sentiment semantics obtained through decoupling as contextual cues to perform fine-grained semantic correction and guidance on modality-specific features, achieving deep-level cross-modal interaction. Furthermore, through a dual-branch gating mechanism, the model can adaptively allocate decision weights between the shared branch and the specific guidance branch according to the characteristics of different samples, thereby dynamically balancing the contributions of different features in complex scenarios and significantly improving the discriminativeness of the fused representation and the robustness of the model.

[0017] (4) Experimental results on the CMU-MOSI, CMU-MOSEI and UR-FUNNY datasets show that the present invention has significant advantages over many classical methods, further confirming the feasibility and effectiveness of the proposed method. Attached Figure Description

[0018] Figure 1 This is a structural diagram of the emotion recognition model of the present invention; Figure 2 This is a schematic diagram of the reversible attention mask decoupling module of the present invention; Figure 3 This is a schematic diagram of the mutual information constraint mechanism of the present invention; Figure 4 This is a schematic diagram of the semantically guided adaptive feature fusion module of the present invention; Figure 5 This is a trend chart of the training loss on the CMU-MOSI dataset. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0020] Example 1 This invention relates to a multimodal emotion recognition method that features reversible decoupling and adaptive fusion of emotion features, such as... Figure 1 As shown, please follow these steps: Step 1: Obtain multimodal dataset samples, which are divided into training set, validation set and test set; Multimodal datasets include the CMU-MOSI dataset, the CMU-MOSEI dataset, and the UR-FUNNY dataset; Step 2: Construct an emotion recognition model, which includes a multimodal feature encoding module, a reversible attention mask decoupling module, a mutual information constraint mechanism module, and a semantically guided adaptive feature fusion module; specifically: Step 2.1: Preprocess the multimodal dataset using the multimodal feature encoding module to obtain the original data for each modality, and then perform multimodal feature encoding; specifically: Step 2.11: Obtain the sample set using the publicly available CMU-MOSI, CMU-MOSEI, and UR-FUNNY datasets. The dataset partitioning is shown in Table 1. The dataset contains multiple video clips, each covering a different emotional state. For each video clip, extract the raw features of the three modalities, and use Librosa to extract Mel spectrograms to obtain acoustic features. Visual features are extracted using a pre-trained EfficientNet. Use BERT pre-trained models to obtain text features .

[0021] Table 1. Results of Dataset Partitioning

[0022] Step 2.12: Design unified feature encoding for the acoustic feature modality (a), text feature modality (t), and visual feature modality (v); specifically: The input sequences of each modality are projected to a unified dimension through one-dimensional convolution (Conv1D); Learnable [CLS] tags and positional encodings (PEs) are added and fed into a multi-layer Transformer encoder. For text feature modalities, BERT is used to obtain word vector representations; for visual and audio feature modalities, frame-level features are directly input. The temporal context features of each modality are obtained after passing through the Transformer encoder. As shown in equation (1): (1) in, The features of the initial input for each modality, For convolution projection, For position encoding; Temporal context features of each modality The Transformer encoder is used again for joint modeling to obtain the final encoded features of each modality. ; Step 2.2: Construct a reversible attention mask decoupling module, referring to... Figure 2 For any single-modal feature This study utilizes invertible neural networks (INNs) to construct a bidirectional invertible mapping relationship between implicit representations and sentiment semantic factors. Combined with an attention masking mechanism, a forward transformation function is used... implicit features Explicit decoupling consists of two parts: shared features containing cross-modal consistent sentiment information. And the unique features that retain the unique attributes of each modality. Then, reverse reconstruction is performed, and the specific process is as follows: Step 2.21: Encoding features for each modality Modality-related features and task-related features are obtained separately through two layers of MLP networks, and the two are concatenated to form implicit features. Introduce a set of learnable reflection vectors. Construct orthogonal matrices Achieve orthogonal blending along the channel dimension to obtain features. As shown in equation (2): (2) in, For the first One reflection vector, For feature dimension, Represents the number of reflections, taking It is 3. It is an identity matrix.

[0023] Step 2.22: After orthogonal mixing, adjust the output features. After layer normalization, we get and Subsequently, through reversible forward transformation and channel rearrangement operations, the following was obtained: The forward and reverse transformations and channel rearrangement process are expressed as equations (3)-(5): (3) (4) (5) Equation (3) represents the forward transformation, and equation (4) represents the inverse transformation. Implemented by two layers of MLP, and Orthogonal characteristics After layer normalization, the two features are divided according to the channel dimension. Forward transformation output characteristics and The splicing characteristics are represented as , A fixed but randomly initialized channel permutation matrix is ​​used to enhance cross-channel dependency modeling capabilities. During the inverse process, the model uses its inverse permutation matrix. For batch size, for The transpose of .

[0024] Step 2.23: Repeat steps 2.21 and 2.22 four times; Step 2.24, following step 2.23, incorporates an attention masking mechanism to adjust the information interaction between shared and unique representations. This mechanism obtains the emotional identity modality. The query is obtained by constructing linear projections of the features respectively. ,key ,value Then calculate right Explanation weight as well as right Explanation weight Furthermore, information fusion is performed across each channel dimension to achieve complementary updates between features, thereby obtaining shared features of cross-modal consistent sentiment information. and the unique characteristics of each modality's unique attributes The process is represented by equations (6)-(8): (6) (7) (8) in, For shared features, As a unique feature, They are respectively The query, key, and value vectors after projection through the linear layer. Corresponding to The weight matrix of the query, key, and value vectors projected through the linear layer. They are respectively The query, key, and value vectors after projection through the linear layer. They are respectively The weight matrix of the query, key, and value vectors projected through the linear layer. for right diagonal weights, for right diagonal weights, for The transpose matrix of the key vectors. for The transpose matrix of the key vectors. This indicates an element-wise multiplication operation. The dimension of the key vector. This indicates the extraction of the main diagonal elements from the attention weight matrix.

[0025] Step 2.25: To achieve effective multimodal feature decoupling, four loss function constraints are designed during the decoupling process. First, to avoid losing key modal features during semantic decoupling, reconstruction constraints are introduced into the input space. Secondly, design reversible constraints. To ensure the reversibility of the potential spatial features, similarity constraints based on the difference in central moments and the Hilbert-Schmidt independence criterion are then introduced. and independence constraints , expressed as equations (9)-(14): (9) (10) (11) (12) (13) (14) in, These are the implicit features of each modality. To obtain the temporal context features of each modality after passing through the Transformer encoder, It is the inverse transformation function. Represented as The square of the norm, For in the interval Random samples on for Norm, Expressing expectations, Let and represent two sets of random variables whose statistical independence is to be measured. Indicates the first The central moment of the first order, This represents the maximum order of the central moments. The value is 5. The Gram matrix between samples, For batch size, For a centered matrix, and Representing different modes.

[0026] Step 2.3: Construct the mutual information constraint mechanism module. (Refer to...) Figure 3 The decoupled features are constrained using Relevant Mutual Information (RMI) and Conditional Mutual Information (CMI). The mutual information between shared features across different modalities and between shared features and sentiment labels is maximized to enhance the semantic consistency of shared features. Simultaneously, given shared features, the mutual information between unique features and sentiment labels is minimized to reduce noise redundancy in unique features. The modeling process for this module is as follows: Step 2.31: To ensure shared features after decoupling It truly captures consistent sentiment semantics across modalities, employing the InfoNCE lower bound to maximize the correlation between shared features across different modalities and between shared features and sentiment labels. The mutual information between them. Specifically, constraints are imposed on shared features of each modality, as well as on shared features and labels of each modality, and the relevant mutual information loss is defined. For equation (15): (15) in, , For scoring networks of MLP, Embedding emotional tags, , and Representing different modes.

[0027] Step 2.32: To ensure unique characteristics Only modality-specific attributes not directly related to the emotional task are retained, and conditional mutual information constraints are designed. Due to the true distribution Since the discriminant is difficult to obtain directly, a discriminant variational approximation is used to construct a binary discriminant. To distinguish between joint samples and conditional multiplication distribution samples, if The smaller the value, the better it is within the known range. back, right Conditional independence means that the unique features do not retain noise information directly related to the task label, making decoupling more effective. Therefore, conditional mutual information can be expressed as equation (16): (16) in, Representing the unique characteristics of each modality, It is an embedded representation of sentiment tags. These are features shared across modalities. Let represent the expectation of a sample under the true joint distribution. This represents the expectation of a sample under a conditionally independent negative sample distribution. Indicates the same condition The variables obtained from the hard negative samples selected by KNN sampling within the neighborhood, if The smaller the value, the better it is within the known range. back, right Conditional independence means that the unique features do not retain noise information directly related to the task label, making decoupling more effective.

[0028] Step 2.33: Minimize the conditional mutual information loss. , expressed as equation (17): (17) CMI loss for and The sum of losses is shown in equation (18): (18) Step 2.4: Build a semantically guided adaptive feature fusion module. (Refer to...) Figure 4 Utilizing the shared features obtained from decoupling As contextual information, the unique features obtained through the residual structure are... Semantic guidance and correction are performed. A dual-branch prediction structure is constructed, predicting based on enhanced shared features and corrected specific features respectively. A gating mechanism adaptively assigns weights to the two branches, and the results are fused to obtain the final sentiment prediction. Specifically: Step 2.41: Utilize shared features As a guide for the following text, the unique features are analyzed through residual structure. Make corrections. First, [the following is a list of corrections]. Enhancement Then, through interaction using a linear residual structure, unique features guided by shared semantics are obtained. , expressed as equation (19): (19) in, These are learnable parameters; Step 2.42: Design a two-branch prediction structure, sharing branches. Based on enhanced shared features Predicting sentiment tendencies, a unique branch guided by shared semantics. Based on the modified unique features Capture complementary details. Finally, calculate the weighting coefficients using a gating mechanism. Adaptive weighted ensemble of two prediction results The process is expressed as equation (20): (20) in, It is the Sigmoid activation function. These are the weighting coefficients. For MLP prediction heads with shared branches, For MLP prediction heads with unique branches guided by shared semantics, This is the final prediction result.

[0029] Step 2.5: Construct the overall optimization objective function; specifically: Construct the overall optimization objective function Introducing task losses As the primary supervisor, mean squared error is used as the task loss in regression tasks, while cross-entropy loss is used in classification tasks. The overall optimization objective is to combine the loss terms from steps 2.25 and 2.33 with the task loss into a joint optimization objective function, expressed as equation (21): (twenty one) in, For reversible constraint loss hyperparameters, For similarity loss hyperparameters, Loss of independence hyperparameters, To reconstruct the loss hyperparameters, For mutual information loss Hyperparameters.

[0030] Step 3: Perform end-to-end training of the emotion recognition model using the training set data. The emotion recognition model is iteratively updated on the training set in batches, calculating the joint loss function through forward propagation and optimizing the model parameters using backpropagation. After each training epoch, the current model is evaluated on the validation set, the corresponding performance metrics are calculated, and the model parameters with validation performance better than the historical best results are saved. To avoid overfitting and improve training efficiency, an early stopping strategy is introduced as a termination condition during training. When the improvement of the validation metric is less than a preset threshold of 0.001 within 5 consecutive epochs or the maximum number of training epochs is reached, model training is terminated, and the optimal model parameters from the validation set are loaded. Finally, the optimal emotion recognition model is evaluated on the test set, and the corresponding emotion recognition results are output.

[0031] Example 2 All experiments were conducted using the PyTorch framework on an NVIDIA 4090 GPU. During training, all parameters were optimized using AdamW. The hyperparameter settings for the CMU-MOSI, CMU-MOSEI, and UR-FUNNY datasets during training are shown in Table 2. Table 2 Hyperparameter configurations on different datasets

[0032] This invention presents a multimodal emotion recognition method with reversible decoupling and adaptive fusion of emotion features. It designs a reversible attention mask decoupling module, constructing a reversible mapping structure between implicit representations of each modality's features and emotion semantic factors based on a reversible neural network. Combined with an attention mask mechanism, the implicit features are explicitly decoupled into shared features with cross-modal consistency and unique features that retain the unique attributes of each modality. Simultaneously, mutual information is used to constrain the dependencies between shared features, unique features, and emotion tags, enhancing emotion semantic consistency modeling while reducing modal noise redundancy. A semantically guided adaptive feature fusion module is designed, utilizing the contextual information of shared features to provide fine-grained semantic guidance to unique features, achieving adaptive fusion of shared and unique guidance branches, thus improving the discriminativeness of the fused representation.

[0033] Example 3 The evaluation metrics used and the results are compared below: This invention selects five evaluation metrics commonly used in the field of multimodal emotion recognition. On CMU-MOSI and CMU-MOSEI datasets, binary classification accuracy (Acc-2), F1 score (F1), seven-class classification accuracy (Acc-7), mean absolute error (MAE), and Pearson correlation coefficient (Corr) were used. It is important to note that Acc-2 and F1 scores are calculated in two ways: negative / non-negative (including zero) and negative / positive (excluding zero). On the UR-FUNNY dataset, Acc-2 and F1 were used. Except for MAE, higher metric values ​​indicate better performance. Underlined text indicates the best and second-best results for each column, marked with... The experimental results were obtained by reproducing the original code, and the remaining data were directly cited from relevant literature. Comparisons were made with existing methods. Table 3 shows the performance comparison results of different network models on the CMU-MOSI dataset. On the CMU-MOSI dataset, the method of this invention performed best in the regression index measuring sentiment intensity, with the mean absolute error reduced to 0.722, the Pearson correlation coefficient increased to 0.798, and the binary classification accuracy reached 85.3%. Compared with mainstream models such as TFN, MulT, and MISA, the method of this invention significantly improved the accuracy of sentiment prediction.

[0034] Table 3 Comparison results of different models on the CMU-MOSI dataset

[0035] Example 4 Compared to Example 3, this example was tested on the CMU-MOSEI dataset, which contains 22,856 video segments. The model construction process from steps 1 to 5 is the same as in Example 3. Experimental results are shown in Table 4. The results demonstrate that this invention exhibits excellent robustness, performing exceptionally well in MAE (Mean Absolute Error) and Corr (Pearson Correlation Coefficient) metrics, with values ​​of 0.531 and 0.779 respectively. These values ​​outperform existing mainstream models such as MulT and MISA, validating the effectiveness and robustness of this invention on large-scale datasets.

[0036] Table 4 Comparison results of different models on the CMU-MOSEI dataset

[0037] Example 5 Compared to Example 3, this example was tested on the UR-FUNNY dataset. This dataset contains 16,514 multimodal discourse samples for humor detection (a binary classification task). The model construction process in steps 1 to 5 is the same as in Example 3. The experimental results in Table 5 show that the present invention achieves a binary classification accuracy (Acc-2) of 71.91% and an F1 score of 71.71%, both of which surpass the comparison methods such as TFN, MISA, and FRDIN, demonstrating the generalization ability of the present invention in the humor detection emotion task.

[0038] Table 5 Comparison results of different models on the UR-FUNNY dataset

[0039] Example 6 The present invention proposes an interpretable and reversible decoupling and adaptive fusion method for emotion features in multimodal emotion recognition. This method can explicitly separate emotional semantics from modal redundancy noise, effectively improving the interpretability of the decision-making process. This allows the model to fully capture fine-grained complementary information across modalities. Experimental results fully demonstrate that the present invention has higher accuracy and robustness in complex scenarios, providing a more accurate solution for multimodal emotion recognition.

[0040] Reference Figure 5The figure illustrates the convergence trend of each loss during training on the CMU-MOSI dataset. As shown, each loss exhibits a rapid decreasing trend in the early stages of training, and then smoothly transitions to a stable convergence state without obvious oscillations or gradient conflicts. This result strongly demonstrates that the joint optimization framework proposed in this invention has good numerical stability, can effectively coordinate the optimization directions among multiple constraint objectives, and maintains good convergence while ensuring the effectiveness of feature decoupling.

Claims

1. A multi-modal emotion recognition method with reversible decoupling and adaptive fusion of emotional features, characterized in that, The specific steps are as follows: Step 1: Obtain multimodal dataset samples, which are divided into training set, validation set and test set; Step 2: Construct an emotion recognition model, which includes a multimodal feature encoding module, a reversible attention mask decoupling module, a mutual information constraint mechanism module, and a semantically guided adaptive feature fusion module. Step 3: Train the emotion recognition model using the training set. After each round of training, evaluate the model on the validation set, calculate the corresponding performance indicators, and save the model parameters that have better validation performance than the historical best results. Use the best emotion recognition model to test on the test set and output the corresponding emotion recognition results.

2. The multi-modal emotion recognition method of claim 1, wherein the affective feature reversible decoupling and adaptive fusion is characterized in that, In step 1, the multimodal datasets include the CMU-MOSI dataset, the CMU-MOSEI dataset, and the UR-FUNNY dataset. 3.The method of claim 2, wherein, Step 2 specifically involves: Step 2.1: Use the multimodal feature encoding module to preprocess the multimodal dataset to obtain the original data of each modality, and then perform multimodal feature encoding. Step 2.2, build a reversible attention mask decoupling module for any single-modal feature , and a bidirectional reversible mapping relationship between the implicit representation and the sentiment semantic factor is constructed by using a reversible neural network; combined with the attention mask mechanism, the implicit feature is explicitly decoupled into two parts through the forward transformation function, and then reconstructed reversely; Step 2.3: Construct a mutual information constraint mechanism module; use relevant mutual information and conditional mutual information to constrain the decoupled features; Step 2.4: Build a semantically guided adaptive feature fusion module; Utilizing the shared features obtained through decoupling As contextual information, the unique features obtained through the residual structure are... Semantic guidance and correction are performed; a dual-branch prediction structure is constructed, which makes predictions based on the enhanced shared features and the corrected specific features respectively, and weights are assigned to the two branches through a gating mechanism, and the final sentiment prediction result is obtained by fusion. Step 2.5: Construct the overall optimization objective function.

4. The multimodal emotion recognition method with reversible decoupling and adaptive fusion of emotion features as described in claim 3, characterized in that, In step 2.1, specifically: Step 2.11: Obtain sample sets using the CMU-MOSI, CMU-MOSEI, and UR-FUNNY datasets. For each video segment, extract the raw features of the three modalities, and use Librosa to extract Mel spectrograms to obtain acoustic features. Visual features are extracted using a pre-trained EfficientNet. Use BERT pre-trained models to obtain text features ; Step 2.12: Design a unified feature encoding for acoustic feature a modality, text feature t modality, and visual feature v modality; Specifically: The input sequences of each modality are projected to a unified dimension through one-dimensional convolution; learnable [CLS] tags and positional encodings are added and input into a multi-layer Transformer encoder. For text feature modalities, BERT is used to obtain word vector representations; for visual and audio feature modalities, frame-level features are directly input. The time sequence context features of each modality are obtained after the Transformer encoder As shown in formula (1): （1） wherein, are features of the initial input for each modality, is a convolutional projection, is a positional encoding; The temporal context features of each modality are combined The joint modeling is performed again through the Transformer encoder to obtain final encoding features of each modality .

5. The multi-modal emotion recognition method of claim 4, wherein the affective feature reversible decoupling and adaptive fusion is characterized by, In step 2.2, the specific process is as follows: Step 2.21: The encoded features of each modality are processed through a two-layer MLP network to obtain modality-related features and task-related features, which are then concatenated to form implicit features. Introduce a set of learnable reflection vectors. Construct orthogonal matrices Achieve orthogonal blending along the channel dimension to obtain features. As shown in equation (2): （2） in, For the first One reflection vector, For feature dimension, It is the identity matrix; Step 2.22: After orthogonal mixing, adjust the output features. After layer normalization, we get and Subsequently, through reversible forward transformation and channel rearrangement operations, the following was obtained: The forward and reverse transformations and channel rearrangement process are expressed as equations (3)-(5): （3） （4） （5） Equation (3) represents the forward transformation, and equation (4) represents the inverse transformation. Implemented by two layers of MLP, and Orthogonal characteristics After layer normalization, the two features are divided according to the channel dimension. Forward transformation output characteristics and The splicing characteristics are represented as , The channel permutation matrix is ​​fixed but randomly initialized. For batch size, for The transpose of the matrix; Step 2.23: Repeat steps 2.21 and 2.22 four times; Step 2.24, following step 2.23, obtains the emotional identity modality. The query is obtained by constructing linear projections of the features respectively. ,key ,value Then calculate right Explanation weight as well as right Explanation weight Furthermore, information fusion is performed across each channel dimension to achieve complementary updates between features, thereby obtaining shared features of cross-modal consistent sentiment information. and the unique characteristics of each modality's unique attributes The process is represented by equations (6)-(8): （6） （7） （8） in, They are respectively The query, key, and value vectors after projection through the linear layer. Corresponding to The weight matrix of the query, key, and value vectors projected through the linear layer. They are respectively The query, key, and value vectors after projection through the linear layer. They are respectively The weight matrix of the query, key, and value vectors projected through the linear layer. for right diagonal weights, for right diagonal weights, for The transpose matrix of the key vectors. for The transpose matrix of the key vectors. The dimension of the key vector. This indicates the extraction of the main diagonal elements from the attention weight matrix; Step 2.25: Design four loss function constraints during the decoupling process, including reconstruction constraints. Reversible constraints Similarity constraints and independence constraints , expressed as equations (9)-(14): （9） （10） （11） （12） （13） （14） in, These are the implicit features of each modality. To obtain the temporal context features of each modality after passing through the Transformer encoder, It is the inverse transformation function. Represented as The square of the norm, In the interval Random samples on for Norm, Expressing expectations, Let and represent two sets of random variables whose statistical independence is to be measured. Indicates the first The central moment of the first order, The Gram matrix between samples, For batch size, For a centered matrix, and Representing different modes.

6. The multi-modal emotion recognition method of claim 5, wherein the affective feature reversible decoupling and adaptive fusion is characterized by, In step 2.3, specifically: Step 2.31, define the relevant mutual information loss by applying constraints on each modality-shared feature and each modality-shared feature and label is of formula (15): （15） wherein, , is an MLP scoring network, is an emotion label embedding, ; Step 2.32, design conditional mutual information constraint , construct binary discriminator with discriminative variational approximation to distinguish joint samples from conditional product distribution samples, the conditional mutual information can be expressed as equation (16): （16） in, Representing the unique characteristics of each modality, It is an embedded representation of sentiment tags. These are features shared across modalities. Let represent the expectation of a sample under the true joint distribution. This represents the expectation of a sample under a conditionally independent negative sample distribution. Indicates the same condition The variables are obtained by selecting hard negative samples in the neighborhood of the target neighborhood using KNN sampling; Step 2.33: Minimize the conditional mutual information loss. , expressed as equation (17): （17） CMI loss for and The sum of losses is shown in equation (18): （18）。 7. The multimodal emotion recognition method with reversible decoupling and adaptive fusion of emotion features as described in claim 6, characterized in that, In step 2.4, specifically: Step 2.41: Utilize shared features As a guide for the following text, the unique features are analyzed through residual structure. Make corrections; firstly, Enhancement Then, through interaction using a linear residual structure, unique features guided by shared semantics are obtained. , expressed as equation (19): （19） in, These are learnable parameters; Step 2.42: Design a two-branch prediction structure, sharing branches. Based on enhanced shared features Predicting sentiment tendencies, a unique branch guided by shared semantics. Based on the modified unique features Capture complementary details; finally, calculate weighting coefficients using a gating mechanism. Adaptive weighted ensemble of two prediction results The process is expressed as equation (20): （20） in, It is the Sigmoid activation function. These are the weighting coefficients. For MLP prediction heads with shared branches, This is an MLP prediction head with a unique branch guided by shared semantics.

8. The multimodal emotion recognition method with reversible decoupling and adaptive fusion of emotion features as described in claim 7, characterized in that, Step 2.5 specifically includes: Construct the overall optimization objective function Introducing task loss As the primary supervisor, mean squared error is used as the task loss in regression tasks, while cross-entropy loss is used in classification tasks. The overall optimization objective is to combine the loss terms from steps 2.25 and 2.33 with the task loss into a joint optimization objective function, expressed as equation (21): （21） in, For reversible constraint loss hyperparameters, For similarity loss hyperparameters, Loss of independence hyperparameters, To reconstruct the loss hyperparameters, For mutual information loss Hyperparameters.

Images