Multi-modal sentiment analysis method based on two-stage information bottleneck and modal classification regularization
By employing a two-stage information bottleneck and modality classification regularization approach, the problems of modality redundancy and imbalance in multimodal sentiment analysis are solved, achieving feature compression and modality balance, thereby improving the accuracy and robustness of sentiment analysis.
Patent Information
- Application Number
- CN202511375011.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-25
- Publication Date
- 2026-02-10
AI Technical Summary
Existing multimodal sentiment analysis methods suffer from redundant information both inside and outside the modality and modality collapse, leading to an imbalance between feature representation noise and modality contribution, which affects the model's discriminativeness and fusion performance.
A two-stage information bottleneck and modality classification regularization approach is adopted. The intramodal redundancy is removed through the single-modal information bottleneck, and the excessive dependence of the model on strong modes is alleviated by modality classification regularization. Cross-modal redundancy is suppressed in the multimodal information bottleneck. Feature compression and balancing are achieved by using mutual information neural estimators and modality discriminators.
It effectively improves the classification performance and robustness of multimodal sentiment analysis. By removing redundant information and balancing modal contributions, it enhances the model's discriminative power and the accuracy of sentiment recognition.
Smart Images
Figure FT_1 
Figure SMS_78 
Figure QLYQS_1
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of multi-modal learning, and particularly relates to a multi-modal sentiment analysis method based on two-stage information bottleneck and modal classification regularization. BACKGROUND
[0002] Multi-modal sentiment analysis aims to understand human emotions by integrating text, speech and visual modalities, and is widely used in human-computer interaction, public opinion monitoring and intelligent services. However, existing methods still face two major challenges: one is the problem of redundant information, which exists within different modalities and between modalities, resulting in noise in feature representation and reducing the discriminability of the model; the other is the problem of modal collapse, which causes the model to excessively rely on strong modalities (such as text) with higher information density during training, while ignoring weak modalities such as speech and vision, resulting in unbalanced modal contributions and limiting the fusion effect. Existing methods such as single-modal compression, cross-modal attention or modal expert mechanism alleviate these problems to some extent, but lack a unified redundancy control and modal balancing mechanism, and the overall performance is still not ideal, so there is an urgent need for a new multi-modal sentiment analysis method that can simultaneously suppress intra- and inter-modal redundancies and avoid modal collapse. SUMMARY
[0003] To solve the problems of intra- and inter-modal redundant information and modal collapse in existing multi-modal sentiment analysis methods, the application proposes a multi-modal sentiment analysis method based on two-stage information bottleneck and modal classification regularization, which aims to simultaneously compress redundant features and maintain balanced contributions of each modality, thereby obtaining a compact and discriminative multi-modal representation. The application introduces a single-modal information bottleneck in the first stage to remove intra-modal redundancy, and combines modal classification regularization to alleviate the excessive dependence of the model on strong modalities, and in the second stage, a multi-modal information bottleneck is used to further suppress cross-modal redundancy and capture complementary emotional signals, thereby effectively improving the classification performance and robustness of multi-modal sentiment analysis.
[0004] To achieve the above purpose, the specific scheme of the application is as follows:
[0005] A multi-modal sentiment analysis method based on two-stage information bottleneck and modal classification regularization, characterized in that it comprises the following steps:
[0006] S1, extracting features from input text, speech and visual modal data to obtain original feature representations of each modality;
[0007] S2, the original features of each modality are input into the corresponding single-modal information bottleneck module. The single-modal information bottleneck module maps the original features into latent representations through a transformer encoder, and uses a mutual information neural estimator to calculate the mutual information between the input and the latent representation. At the same time, it retains discriminative information with the goal of minimizing the mean square error between the prediction result and the real sentiment label. By minimizing the input-latent mutual information and maximizing the latent-label mutual information, single-modal compressed features that remove intramodal redundancy and retain discriminativeness are obtained.
[0008] S3, the single-modal compressed features are input into the modality classification regularization module. The modality classification regularization module includes a modality discriminator and an adversarial training unit. The modality discriminator uses cross-entropy loss to distinguish the modality to which the features belong. The adversarial training unit updates the discriminator parameters to minimize the modality discrimination loss while updating the encoder parameters to maximize the modality discrimination loss and minimize the task loss, thereby generating a modality-indistinguishable representation and alleviating modality collapse.
[0009] S4, input the single-modal features processed in step S3 into the multimodal information bottleneck module, the multimodal information bottleneck module comprising:
[0010] The modality expert weighted fusion submodule is used to weight and concatenate the three-modal compressed features with learnable weights to obtain a primary fusion representation.
[0011] An extended long short-term memory network is used to model the temporal dependency of the primary fusion representation and output the final multimodal fusion representation;
[0012] A mutual information neural estimator is used to minimize the mutual information between the primary fusion representation and the final fusion representation, and maximize the mutual information between the final fusion representation and the sentiment label, further suppressing cross-modal redundancy and preserving complementary sentiment signals;
[0013] S5 employs a two-stage alternating training strategy to optimize the overall model:
[0014] In the first stage, the modal encoder and task predictor are fixed, while the mutual information estimator and modal discriminator are optimized.
[0015] In the second stage, the mutual information estimator and modality discriminator are fixed, and the modality encoder and task predictor are optimized.
[0016] The overall model's loss function is composed of a weighted average of the single-modal information bottleneck loss, modality discrimination loss, multimodal information bottleneck loss, and final prediction loss.
[0017] Furthermore, the loss function for the single-modal information bottleneck module in step S2 is:
[0018] ,
[0019] in, For the original features of mode m, For the latent representation of mode m, To adjust the hyperparameters related to compression and task performance, The calculation is approximated using a mutual information neural estimator based on Donsker-Varadhan representation. Belongs to the visual modality. Belongs to the text modality. It belongs to the speech modality.
[0020] Furthermore, the modality discriminator in step S3 is a multilayer perceptron, and its cross-entropy loss is denoted as... The gradient update rule for generating modally indistinguishable representations is:
[0021] ,
[0022] ,
[0023] in, For discriminator parameters, For the parameters of the i-th modal encoder, To aid in classification loss, This is the gradient operator.
[0024] Furthermore, the fusion method of the modal expert weighted fusion submodule in step S4 is as follows:
[0025] ,
[0026] in, This represents a basic fusion representation; Concat is the concatenation operation. For single-modal features compressed by the single-modal information bottleneck module, It is a learnable diagonal weight matrix used to dynamically adjust the contribution ratio of visual, text, and speech modalities during fusion.
[0027] Furthermore, the loss function of the multimodal information bottleneck module in step S4 is:
[0028] ,
[0029] in, This represents the bottleneck loss of multimodal information. This is a primary fusion representation. For the final multimodal fusion representation, To adjust the hyperparameters related to compression and task relevance.
[0030] Furthermore, the loss function of the overall model described in step S5 is:
[0031] ,
[0032] in, To ultimately predict the loss, It is a single-modal information bottleneck loss. It is modal discrimination loss. It is a bottleneck loss of multimodal information.
[0033] The present invention has the following beneficial effects:
[0034] This invention presents a multimodal sentiment analysis method based on a two-stage information bottleneck and modality classification regularization. It is a neural network algorithm with a Transformer and Extended Long Short-Term Memory (xLSTM) network as its backbone. By combining the differences in semantic feature representations across different modalities, it enhances the algorithm's ability to compress redundant multimodal information and capture complementary information. It utilizes the adversarial mechanism of modality classification regularization to suppress modality collapse, thereby achieving balanced modeling of multimodal sentiment. Addressing the problems of internal and external modality redundancy and uneven modality contribution, this invention uses the information bottleneck principle to compress task-irrelevant features, highlighting key modality information. Simultaneously, it introduces modality discrimination and feature alignment mechanisms, enabling the model to learn text, speech, and visual modal features in a balanced manner, thus effectively improving the accuracy and robustness of sentiment recognition. Attached Figure Description
[0035] Figure 1 This is a flowchart of the multimodal sentiment analysis method based on two-stage information bottleneck and modality classification regularization of the present invention. Detailed Implementation
[0036] The present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. It should be noted that the specific embodiments are not intended to limit the scope of the present invention.
[0037] like Figure 1 As shown in the figure, this specific embodiment provides a multimodal sentiment analysis method based on two-stage information bottleneck and modality classification regularization, which includes the following steps:
[0038] S1 extracts features from the input text, speech, and visual modal data respectively to obtain the original feature representations of each modality.
[0039] S2, the original features of each modality are input into the corresponding single-modal information bottleneck module. The single-modal information bottleneck module maps the original features into latent representations through a transformer encoder, and uses a mutual information neural estimator to calculate the mutual information between the input and the latent representation. At the same time, it retains discriminative information with the goal of minimizing the mean square error between the prediction result and the true sentiment label. By minimizing the input-latent mutual information and maximizing the latent-label mutual information, single-modal compressed features that remove intramodal redundancy and retain discriminativeness are obtained.
[0040] The visual, text, and speech modalities are modeled separately using the Single-Modal Information Bottleneck (SMIB) module. Specifically, the three input unimodal features are represented as follows: , , ,in Belongs to the visual modality. Belongs to the text modality. It belongs to the audio modality, and its dimension is N. First, an independent four-layer Transformer encoder is used to extract features from each mode to obtain the single-mode latent representation. Subsequently, based on the Information Bottleneck (IB) principle, the Mutual Information Neural Estimation (MINE) method is used to estimate the mutual information between the input and the latent representation. MINE is based on the Donsker–Varadhan representation and constructs a joint distribution. And by using a batch shuffling method to generate negative samples to construct a marginal distribution. With the help of neural discriminators The difference between the two is maximized to approximate the lower bound of mutual information. Simultaneously, to ensure that the compressed representation does not lose discriminative information, a supervision signal is introduced, and the prediction result is minimized. With real labels The mean squared error (MSE) between them approximately maximizes the mutual information of the single modes. Finally, the loss function for the single-modal information bottleneck module is:
[0041]
[0042] In the formula, For the original features of mode m, For the latent representation of mode m, To adjust the hyperparameters related to compression and task performance, An approximate computation is performed using a mutual information neural estimator based on Donsker-Varadhan representation. This step yields single-modal compressed features that remove redundancy and preserve discriminativeness, where... Belongs to the visual modality. Belongs to the text modality. It belongs to the speech modality.
[0043] S3, the single-modal compressed features are input into the modality classification regularization module. The modality classification regularization module includes a modality discriminator and an adversarial training unit. The modality discriminator uses cross-entropy loss to distinguish the modality to which the features belong. The adversarial training unit updates the discriminator parameters to minimize the modality discrimination loss while updating the encoder parameters to maximize the modality discrimination loss and minimize the task loss, thereby generating a modality-indistinguishable representation and alleviating modality collapse.
[0044] Modality collapse is mitigated through a Modality Classification Regularization (MCR) module. After single-modal compression, a Modal Discriminator (MD) is introduced. The discriminator employs a multilayer perceptron structure and uses cross-entropy loss. Distinguish the modality to which a feature belongs. During training, the discriminator parameters... Update to minimize modality discrimination loss encoder parameters By optimizing against the objective, the goal is to minimize the task loss while maximizing the modality discrimination loss, thus generating modality-indistinguishable representations.
[0045] ,
[0046] ,
[0047] in, For discriminator parameters, For the parameters of the i-th modal encoder, To aid in classification loss, This is the gradient operator. This adversarial mechanism prevents the model from over-relying on strong modalities such as text, and instead makes balanced use of weak modalities such as speech and vision, thereby improving the robustness and fairness of the fusion process.
[0048] S4, input the single-modal features processed in step S3 into the multimodal information bottleneck module, the multimodal information bottleneck module comprising:
[0049] The modality expert weighted fusion submodule is used to weight and concatenate the three-modal compressed features with learnable weights to obtain a primary fusion representation.
[0050] An extended long short-term memory network is used to model the temporal dependency of the primary fusion representation and output the final multimodal fusion representation;
[0051] A mutual information neural estimator is used to minimize the mutual information between the primary fusion representation and the final fusion representation, and to maximize the mutual information between the final fusion representation and the sentiment label, further suppressing cross-modal redundancy and preserving complementary sentiment signals.
[0052] Cross-modal redundancy is further compressed using the Multi-Modal Information Bottleneck (MMIB) module. This further reduces the single-modal features compressed by MMIB. Input modal expert weighted fusion module, the fusion weights of different modalities are composed of learnable parameters. Control, resulting in a primary fusion representation:
[0053] ,
[0054] The `Concat` operation is used to concatenate the data. Subsequently, an Extended Long Short-Term Memory (xLSTM) network is used to model temporal dependencies, resulting in the final multimodal fusion representation. The optimization objective of MMIB is to minimize multimodal mutual information. This aims to compress cross-modal redundancy and maximize the mutual information between the final features and labels. To enhance the ability to discriminate:
[0055] ,
[0056] in This represents the bottleneck loss of multimodal information. This is a primary fusion representation. For the final multimodal fusion representation, To adjust the hyperparameters related to compression and task relevance, mutual information is also approximated by a neural estimation method.
[0057] S5 employs a two-stage alternating training strategy to optimize the overall model:
[0058] In the first stage, the modal encoder and task predictor are fixed, while the mutual information estimator and modal discriminator are optimized.
[0059] In the second stage, the mutual information estimator and modality discriminator are fixed, and the modality encoder and task predictor are optimized.
[0060] The overall model's loss function is composed of a weighted average of the single-modal information bottleneck loss, modality discrimination loss, multimodal information bottleneck loss, and final prediction loss.
[0061] Based on the above steps, the overall loss function designed in this invention is:
[0062] ,
[0063] in, It is a single-modal information bottleneck loss. It is modal discrimination loss. It is a bottleneck loss of multimodal information.
[0064] The model is then optimized based on the final prediction loss. The optimization strategy employs an alternating two-stage training approach: the first stage optimizes the mutual information estimator and modality discriminator to enable the model to suppress redundancy and perform modality adversarial tasks; the second stage optimizes the modality encoder and task predictor to ensure the fused representation has good discriminative performance. This alternating training strategy achieves an effective balance between compressing redundancy, balancing modality contributions, and improving emotion recognition performance.
[0065] This invention evaluates the proposed TIB-MCR method on two publicly available multimodal sentiment analysis benchmark datasets, CMU-MOSI and CMU-MOSEI. The MOSI dataset contains 2199 samples, and the MOSEI dataset contains 23453 samples, both of which are divided into training, validation, and test sets according to standard criteria.
[0066] Regarding evaluation metrics, this invention employs the following four commonly used sentiment analysis performance metrics:
[0067] (1) Binary classification accuracy (Acc-2): This measure is used to evaluate the model's performance in binary classification tasks of positive and negative sentiment, and is one of the most intuitive evaluation metrics. This metric calculates the proportion of correctly predicted samples to the total number of samples, reflecting the overall classification ability of the model. A higher Acc-2 indicates that the model can accurately distinguish between the positive and negative polarities of sentiment, and is particularly effective for coarse-grained sentiment recognition.
[0068] (2) Seven-class accuracy (Acc-7): This metric is used to evaluate the model's ability in fine-grained sentiment classification tasks, specifically dividing sentiment intensity into seven levels (from strong negative to strong positive). This metric measures the model's ability to distinguish between different intensities of sentiment. Compared to Acc-2, Acc-7 is more rigorous and detailed, requiring the model not only to identify sentiment polarity but also to accurately distinguish the strength of sentiment, thus placing higher demands on the model's feature extraction and representation capabilities.
[0069] (3) Macro-average F1 score (F1): This is the harmonic mean of precision and recall, averaged across all categories, and comprehensively reflects the accuracy and coverage of the classification. The macro-average F1 score is not biased towards the majority class due to class imbalance, and can more fairly reflect the model's performance across different categories. A higher F1 score indicates that the model can not only correctly identify the sentiment category, but also maintain the predictive balance between different categories.
[0070] (4) Mean Absolute Error (MAE): This measures the average difference between the model's predicted values and the actual sentiment labels. Unlike classification metrics, MAE is a regression performance metric that assesses the model's accuracy in predicting sentiment intensity. The smaller the MAE value, the closer the predicted result is to the actual sentiment value. This metric is particularly suitable for measuring the model's performance in tasks involving continuous sentiment intensity prediction, and it can intuitively reflect the model's stability and reliability in fine-grained sentiment analysis.
[0071] During training, the text modality used BERT-base-uncased 768-dimensional hidden states, the visual modality used 35-dimensional facial action unit features extracted by Facet, and the speech modality used 74-dimensional acoustic features extracted by the COVAREP toolkit. The optimizer was Adam, the learning rate was set to 1e-4, the batch size was 16, and the training lasted for 200 epochs. All experiments were performed on a single NVIDIA RTX 3090 Ti GPU using the PyTorch framework.
[0072] Table 1 shows a comparison of the evaluation metrics between the method of this embodiment and other prior art methods:
[0073]
[0074] The results of the above specific embodiments show that, compared with the prior art, the method of the present invention can effectively alleviate the problems of modality imbalance and classification imbalance in multimodal sentiment analysis tasks, maintain similar classification scores for positive and negative sentiment samples, and significantly improve the accuracy and robustness of sentiment analysis.
[0075] The above embodiments are merely preferred examples of the present invention and are not intended to limit the invention. Those skilled in the art can make various modifications and variations based on the present invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are within the scope of protection of the present invention.
Claims
1. A multimodal sentiment analysis method based on two-stage information bottleneck and modality classification regularization, characterized in that, Includes the following steps: S1 extracts features from the input text, speech, and visual modal data respectively to obtain the original feature representations of each modality; S2, the original features of each modality are input into the corresponding single-modal information bottleneck module. The single-modal information bottleneck module maps the original features into latent representations through a transformer encoder, and uses a mutual information neural estimator to calculate the mutual information between the input and the latent representation. At the same time, it retains discriminative information with the goal of minimizing the mean square error between the prediction result and the real sentiment label. By minimizing the input-latent mutual information and maximizing the latent-label mutual information, single-modal compressed features that remove intramodal redundancy and retain discriminativeness are obtained. S3, the single-modal compressed features are input into the modality classification regularization module. The modality classification regularization module includes a modality discriminator and an adversarial training unit. The modality discriminator uses cross-entropy loss to distinguish the modality to which the features belong. The adversarial training unit updates the discriminator parameters to minimize the modality discrimination loss while updating the encoder parameters to maximize the modality discrimination loss and minimize the task loss, thereby generating a modality-indistinguishable representation and alleviating modality collapse. S4. The single-modal features processed in step S3 are input into the multimodal information bottleneck module. The multimodal information bottleneck module includes: a modality expert weighted fusion submodule, which is used to weight and concatenate the three-modal compressed features with learnable weights to obtain a primary fusion representation; an extended long short-term memory network, which is used to model the temporal dependency of the primary fusion representation and output the final multimodal fusion representation; and a mutual information neural estimator, which is used to minimize the mutual information between the primary fusion representation and the final fusion representation, and maximize the mutual information between the final fusion representation and the sentiment label, further suppressing cross-modal redundancy and preserving complementary sentiment signals. S5 employs a two-stage alternating training strategy to optimize the overall model: In the first stage, the modal encoder and task predictor are fixed, while the mutual information estimator and modal discriminator are optimized; in the second stage, the mutual information estimator and modal discriminator are fixed, while the modal encoder and task predictor are optimized. The loss function of the overall model is composed of a weighted average of the single-modal information bottleneck loss, the modal discrimination loss, the multimodal information bottleneck loss, and the final prediction loss.
2. The method according to claim 1, characterized in that, The loss function of the single-modal information bottleneck module in step S2 is: ， in, For the original features of mode m, For the latent representation of mode m, To adjust the hyperparameters related to compression and task performance, The calculation is approximated using a mutual information neural estimator based on Donsker-Varadhan representation. Belongs to the visual modality. Belongs to the text modality. It belongs to the speech modality.
3. The method according to claim 1, characterized in that, The modality discriminator in step S3 is a multilayer perceptron, and its cross-entropy loss is denoted as... ; The gradient update rule for generating modally indistinguishable representations is as follows: ， ， in, For discriminator parameters, For the parameters of the i-th modal encoder, To aid in classification loss, This is the gradient operator.
4. The method according to claim 1, characterized in that, The fusion method of the modal expert weighted fusion submodule in step S4 is as follows: ， in, This represents a basic fusion representation; Concat is the concatenation operation. For single-modal features compressed by the single-modal information bottleneck module, It is a learnable diagonal weight matrix used to dynamically adjust the contribution ratio of visual, text, and speech modalities during fusion.
5. The method according to claim 1, characterized in that, The loss function of the multimodal information bottleneck module in step S4 is: ， in, This represents the bottleneck loss of multimodal information. This is a primary fusion representation. For the final multimodal fusion representation, To adjust the hyperparameters related to compression and task relevance.
6. The method according to claim 1, characterized in that, The loss function of the overall model described in step S5 is: ， in, To ultimately predict the loss, It is a single-modal information bottleneck loss. It is modal discrimination loss. It is a bottleneck loss of multimodal information.