Multi-modal sentiment analysis method based on progressive comparative learning
By employing a multimodal sentiment analysis method based on progressive contrastive learning, this approach utilizes the hierarchical features and dynamic transformation matrix of a pre-trained language model to calibrate audio and video features. Combined with a sentiment weight fusion strategy, it addresses the issues of modal imbalance and noise interference in multimodal sentiment analysis, thereby improving the accuracy and robustness of sentiment analysis.
Patent Information
- Application Number
- CN202511928855.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-03-20
AI Technical Summary
Existing multimodal sentiment analysis methods suffer from modality imbalance, noise interference, low feature quality, and a lack of sentiment-specific fusion strategies, resulting in suboptimal feature representations and insufficient sentiment analysis accuracy.
We employ a progressive contrastive learning approach, which separates commonalities and unique characteristics among modalities, constructs multi-layer progressive contrastive learning and conditional transformation feature enhancement, uses the hierarchical features of the pre-trained language model as anchor points to generate a dynamic transformation matrix for feature calibration, and calculates sentiment weights for weighted fusion.
It achieves stronger feature discrimination and higher sentiment analysis performance, effectively alleviates modality imbalance and noise interference, accurately captures sentiment cues, and improves the accuracy and robustness of sentiment analysis.
Smart Images
Figure CN121706014A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of sentiment analysis technology, specifically a multimodal sentiment analysis method based on progressive contrastive learning. Background Technology
[0002] With the widespread use of social media and smart mobile devices, people are increasingly inclined to share their opinions and emotions through a combination of video, audio, and text. This explosive growth of multimodal data has made multimodal sentiment analysis an important research direction in the fields of natural language processing and artificial intelligence.
[0003] Multimodal sentiment analysis aims to integrate various information such as language content, facial expressions, and vocal tone to achieve a more comprehensive and nuanced understanding of human emotions, and has broad application prospects in fields such as human-computer interaction, assisted therapy, and public opinion monitoring. However, existing technologies for multimodal sentiment analysis have the following problems:
[0004] 1) Modal imbalance: With the development of large-scale pre-trained language models such as BERT, the feature extraction capability of text modal is far stronger than that of audio and video modal. Existing methods, such as tensor fusion, often ignore this difference and cannot effectively utilize the complementary information in non-text modal.
[0005] 2) Noise interference and low feature quality: Audio and visual data usually contain a lot of environmental noise and redundant information. Existing methods are difficult to effectively clean up this noise before fusion, resulting in suboptimal feature representation after fusion.
[0006] 3) Insufficient utilization of contrastive learning: Although some works have introduced contrastive learning to align modalities, most of them use simple data augmentation strategies to construct positive and negative samples, failing to make full use of the rich hierarchical semantic information contained in the pre-trained model itself, resulting in unclear discrimination boundaries in the feature space;
[0007] 4) The fusion strategy lacks sentiment-specificity: Existing fusion modules usually treat all feature subspaces the same, lacking a mechanism to clearly evaluate the contribution of different subspaces to the sentiment classification task, which may lead to the dilution of key sentiment information. Summary of the Invention
[0008] The purpose of this invention is to provide a multimodal sentiment analysis method based on progressive contrastive learning, which has stronger feature discriminative ability, can effectively alleviate modality imbalance and noise interference, has a more targeted fusion process, can accurately capture emotional cues, and can achieve high-performance sentiment analysis, thus solving the problems mentioned in the background art.
[0009] To achieve the above objectives, the present invention provides the following technical solution:
[0010] A multimodal sentiment analysis method based on progressive contrastive learning includes:
[0011] Obtain a multimodal dataset, extract features from the text, audio, and video modalities within the dataset, and determine the text features. Audio features and video features ;
[0012] Separating commonalities and unique characteristics among different modes to analyze audio features and video features Perform subspace decomposition to achieve feature subspace modeling;
[0013] Text features are optimized by leveraging the feature subspaces of audio and video, employing multi-layered progressive contrastive learning, and obtaining optimized text features through a contrastive learning strategy. ;
[0014] Feature enhancement based on conditional transformation, utilizing optimized text features Enhance the subspace features of audio and video to obtain enhanced invariant features. and unique characteristics ;
[0015] Enhanced invariant features Unique characteristics and text features Feature stacking and dimensionality reduction are performed, sentiment weights are calculated, and sentiment perception weighted fusion and prediction are performed based on the sentiment weights.
[0016] Preferably, feature extraction is performed on the text, audio, and video modalities in the multimodal dataset to determine text features. Audio features and video features Perform the following operations:
[0017] Obtain sequences from a multimodal dataset Where t, a, and v represent text, audio, and video, respectively;
[0018] When extracting text features, the original text Input the pre-trained text model BERT and extract the initial word vectors of the last layer as text features. ;
[0019] When extracting audio and video features, for audio modalities The COVAREP tool was used to extract pitch and Mel-spectral features for video modalities. The Facet tool was used to extract facial expression and head movement features. Then, the audio and video features were input into the Long Short-Term Memory (LSTM) unit for temporal modeling to obtain the output features. and .
[0020] Preferably, the commonalities and characteristics between the separation modes are analyzed for audio features. and video features Perform subspace decomposition and execute the following operations:
[0021] Using space-invariant encoders Extracting modality-invariant features from audio and video ;
[0022] Utilizing space-specific encoders Extract modality-specific features from audio and video. ;
[0023] We use the difference in central moments as the invariant space for the similarity loss constraint, and the orthogonality constraint as the unique space for the difference loss constraint.
[0024] Preferably, the text is optimized using the feature subspaces of audio and video, and multi-layer progressive contrastive learning is performed to obtain optimized text features through the contrastive learning strategy. Perform the following operations:
[0025] Anchor point selection: Select the output features of the last layer of the BERT model as the anchor point. ;
[0026] Construction of difficult positive samples: Selecting output features from the first 11 layers of the BERT model As positive samples, the shallow features and deep features have different levels of semantic abstraction, which constitutes a hierarchical difficult positive sample.
[0027] Construction of difficult-to-negative samples: Samples opposite to the anchor sentiment label are selected from the subspace features of audio and video as negative samples. To construct hierarchical difficult-to-negative samples, a layer-dependent pressure is applied to the negative samples. Varying dynamic Gaussian noise :
[0028] (1)
[0029] in, Represents variance;
[0030] Calculate the contrastive loss: Optimize using the InfoNCE loss function:
[0031] (2)
[0032] Where r represents the number of BERT layers, Let K represent the set of least positive samples, and K represent the number of negative samples. Indicates being Optimized text features are obtained through a contrastive learning strategy using a noise-invariant space and a set of hard-to-bear samples in a specific space. .
[0033] Preferably, feature enhancement based on conditional transformation utilizes optimized text features. Enhance the subspace features of audio and video to obtain enhanced invariant features. and unique characteristics Perform the following operations:
[0034] Generate transformation matrix: transform text features Generates a product using linear layers and reshaping operations, similar to... The conditional transformation matrix for dimension matching;
[0035] Non-linear activation: using the Sigmoid activation function Normalize the transformation matrix to ensure that the transformation values are within a reasonable range;
[0036] Feature calibration: Comparing the transformation matrix with the original features Element-wise multiplication yields the enhanced features. :
[0037] (3)
[0038] (4)
[0039] Where T represents the transformation matrix. R represents the dimension, and R represents the reshaping operation. Represents matrix multiplication. Represents the weights of the linear layer. Indicates deviation, The symbol represents the Sigmoid activation function, and ⊙ represents element-wise multiplication. Similarly, the enhanced unique spatial features can be obtained. .
[0040] Preferably, for the enhanced invariant features Unique characteristics and text features Feature stacking and dimensionality reduction are performed, including: enhancing invariant features. Unique characteristics and text features The initial fused features U are obtained by stacking and then mapped to a low-dimensional space through a dimensionality reduction layer.
[0041] Preferably, the sentiment weight is calculated by performing the following operations:
[0042] Sentiment weights q are calculated for the three feature spaces using fully connected layers, and feature descriptors are obtained by nonlinear activation of the dimensionality-reduced features. And use the Softmax function to generate the attention matrix c;
[0043] The attention matrix c is as follows:
[0044] (5)
[0045] in, ⊙ represents the Softmax function, and ⊙ represents element-wise multiplication.
[0046] Preferably, based on sentiment weights, sentiment perception weighted fusion and prediction are performed, and the following operations are performed:
[0047] The initial feature U is weighted according to the weights, and the final sentiment prediction result is output through the regression layer:
[0048] (6)
[0049] in, Represents the weights of the linear layer. ⊙ represents the deviation, and ⊙ represents element-wise multiplication. The overall model uses the mean squared error as the task loss.
[0050] Compared with the prior art, the beneficial effects of the present invention are:
[0051] 1. The present invention has stronger feature discrimination and significantly improved model performance. This is due to the multi-layer progressive contrastive learning in the technical solution. By using the hierarchical features inside the pre-trained language model as hard positive samples and combining them with cross-modal progressive hard negative samples, a text semantic space with clearer boundaries and more optimized structure is constructed. Experiments show that the method performs well on the MOSI and MOSEI datasets.
[0052] 2. This invention can effectively alleviate modal imbalance and noise interference. It is derived from the feature enhancement based on conditional transformation in the technical solution. Existing methods often directly fuse low-quality non-text features, which easily introduces noise. This invention uses high-quality text features to generate a dynamic transformation matrix, which guides the calibration of audio and video features, suppresses redundancy and noise in the modality-invariant space and the idiomatic space, and achieves fairer and more robust modal interaction.
[0053] 3. The fusion process of this invention is more targeted and accurately captures emotional cues. It originates from the emotional perception fusion in the technical solution. Unlike the traditional direct splicing or general attention mechanism, emotional perception fusion explicitly calculates the emotional weight of each feature subspace. This means that the model can automatically identify and amplify features containing strong emotional signals, thereby improving the accuracy and relevance of sentiment analysis. Attached Figure Description
[0054] Figure 1 This is a flowchart illustrating the framework of the multimodal sentiment analysis method based on progressive contrastive learning of the present invention.
[0055] Figure 2 This is a flowchart of the feature enhancement based on conditional transformation of the present invention;
[0056] Figure 3 This is a flowchart of the emotion perception fusion of the present invention. Detailed Implementation
[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0058] To address the existing problems of modality imbalance, noise interference and low feature quality, insufficient utilization of contrastive learning, and lack of sentiment-specificity in fusion strategies, please refer to [link / reference]. Figures 1-3 This embodiment provides the following technical solution:
[0059] A multimodal sentiment analysis method based on progressive contrastive learning is proposed. The overall process includes feature extraction, subspace modeling, multi-layer progressive contrastive learning, feature enhancement, and sentiment perception fusion prediction.
[0060] Specifically, firstly, leveraging the hierarchical advantages of pre-trained language models, a robust text anchor space is constructed through multi-layered progressive contrastive learning. Then, this high-quality text space is used as a condition to guide and enhance weaker audio and video modalities, eliminating noise. Finally, through a fusion module that perceives sentiment weights, the most emotionally valuable information from each modality is integrated. This precise fusion strategy effectively overcomes modality imbalance and noise problems, achieving high-performance sentiment analysis.
[0061] Obtain a multimodal dataset, extract features from the text, audio, and video modalities within the dataset, and determine the text features. Audio features and video features ;
[0062] In this embodiment, feature extraction is performed on the text, audio, and video modalities in the multimodal dataset to determine the text features. Audio features and video features Perform the following operations:
[0063] Obtain sequences from a multimodal dataset Where t, a, and v represent text, audio, and video, respectively;
[0064] When extracting text features, the original text Input the pre-trained text model BERT and extract the initial word vectors of the last layer as text features. ;
[0065] When extracting audio and video features, for audio modalities The COVAREP tool was used to extract features such as pitch and Mel spectrum for video modalities. The Facet tool was used to extract facial expression and head movement features. Then, the audio and video features were input into the Long Short-Term Memory (LSTM) unit for temporal modeling to obtain the output features. and .
[0066] Separating commonalities and unique characteristics among different modes to analyze audio features and video features Perform subspace decomposition to achieve feature subspace modeling;
[0067] In this embodiment, the commonalities and unique characteristics between the separation modes are analyzed to determine the audio features. and video features Perform subspace decomposition and execute the following operations:
[0068] Using space-invariant encoders Extracting modality-invariant features from audio and video ;
[0069] Utilizing space-specific encoders Extract modality-specific features from audio and video. ;
[0070] We use the difference in central moments as the invariant space for the similarity loss constraint, and the orthogonality constraint as the unique space for the difference loss constraint.
[0071] Text features are optimized by leveraging the feature subspaces of audio and video, employing multi-layered progressive contrastive learning, and obtaining optimized text features through a contrastive learning strategy. ;
[0072] In this embodiment, the text is optimized using the feature subspaces of audio and video, and multi-layer progressive contrastive learning is performed to obtain optimized text features through a contrastive learning strategy. Perform the following operations:
[0073] Anchor point selection: Select the output features of the last layer (12th layer) of the BERT model as the anchor point. ;
[0074] Construction of difficult positive samples: Selecting output features from the first 11 layers of the BERT model As positive samples, shallow features and deep features have different levels of semantic abstraction, which constitutes a hierarchical difficult positive sample.
[0075] Construction of difficult-to-negative samples: Samples opposite to the anchor sentiment label are selected from the subspace features of audio and video as negative samples. To construct hierarchical difficult-to-negative samples, a layer-dependent pressure is applied to the negative samples. Varying dynamic Gaussian noise :
[0076] (1)
[0077] in, Represents variance;
[0078] Calculate the contrastive loss: Optimize using the InfoNCE loss function:
[0079] (2)
[0080] Where r represents the number of BERT layers, Let K represent the set of least positive samples, and K represent the number of negative samples. Indicates being Optimized text features are obtained through a contrastive learning strategy using a noise-invariant space and a set of hard-to-bear samples in a specific space. .
[0081] It should be noted that existing multimodal sentiment analysis suffers from modality imbalance, where text modalities typically contain richer semantic information than audio and visual modalities, and simple contrastive learning strategies cannot fully utilize the hierarchical semantic structure of pre-trained language models. This embodiment constructs a highly discriminative text feature space as an anchor. Specifically, it uses the outputs of different layers of the pre-trained language model as hierarchically structured hard positive samples, while simultaneously constructing hard negative samples containing varying noise intensities from the audio and video modalities. This multi-layered contrast forces the model to learn a more robust and well-structured feature space, enhancing the discriminative boundaries of the features.
[0082] Feature enhancement based on conditional transformation, utilizing optimized text features Enhance the subspace features of audio and video to obtain enhanced invariant features. and unique characteristics ;
[0083] In this embodiment, feature enhancement based on conditional transformation utilizes optimized text features. Enhance the subspace features of audio and video to obtain enhanced invariant features. and unique characteristics Perform the following operations:
[0084] Generate transformation matrix: transform text features Generates a product using linear layers and reshaping operations, similar to... The conditional transformation matrix for dimension matching;
[0085] Non-linear activation: using the Sigmoid activation function Normalize the transformation matrix to ensure that the transformation values are within a reasonable range;
[0086] Feature calibration: Comparing the transformation matrix with the original features Element-wise multiplication yields the enhanced features. :
[0087] (3)
[0088] (4)
[0089] Where T represents the transformation matrix. R represents the dimension, and R represents the reshaping operation. Represents matrix multiplication. Represents the weights of the linear layer. Indicates deviation, The symbol represents the Sigmoid activation function, and ⊙ represents element-wise multiplication. Similarly, the enhanced unique spatial features can be obtained. .
[0090] It should be noted that, in response to the problem of suboptimal feature representation caused by inherent noise and redundant information in audio and video modalities, as well as the interference that direct fusion may bring, this embodiment uses the constructed strong text feature representation as a condition to generate a dynamic transformation matrix. This transformation matrix is used to recalibrate the modality-invariant feature subspace and modality-specific feature subspace of audio and video modalities. Guided by text semantics, it selectively enhances the effective information in non-text modalities and suppresses noise, promoting fairer and more effective cross-modal interaction.
[0091] Enhanced invariant features Unique characteristics and text features Feature stacking and dimensionality reduction are performed, sentiment weights are calculated, and sentiment perception weighted fusion and prediction are performed based on the sentiment weights.
[0092] In this embodiment, the enhanced invariant features Unique characteristics and text features Feature stacking and dimensionality reduction are performed, including: enhancing invariant features. Unique characteristics and text features The initial fused features U are obtained by stacking and then mapped to a low-dimensional space through a dimensionality reduction layer.
[0093] In this embodiment, the sentiment weight is calculated by performing the following operations:
[0094] Sentiment weights q are calculated for the three feature spaces using fully connected layers, and feature descriptors are obtained by nonlinear activation of the dimensionality-reduced features. And use the Softmax function to generate the attention matrix c;
[0095] The attention matrix c is as follows:
[0096] (5)
[0097] in, ⊙ represents the Softmax function, and ⊙ represents element-wise multiplication.
[0098] In this embodiment, emotion perception weighted fusion and prediction are performed based on emotion weights, and the following operations are performed:
[0099] The initial feature U is weighted according to the weights, and the final sentiment prediction result is output through the regression layer:
[0100] (6)
[0101] in, Represents the weights of the linear layer. ⊙ represents the deviation, and ⊙ represents element-wise multiplication. The overall model uses the mean squared error as the task loss.
[0102] It should be noted that, in response to the problem that traditional fusion strategies often treat all features equally and fail to explicitly prioritize the most relevant information in the downstream sentiment analysis task, this embodiment designs an explicit sentiment weighting mechanism. It can dynamically calculate the sentiment saliency weight of each feature subspace. By multiplying the features with these sentiment weights and combining them with an attention mechanism, it ensures that the subspace containing stronger sentiment signals contributes more to the final representation, thereby improving the accuracy of sentiment prediction.
[0103] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0104] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A multimodal sentiment analysis method based on progressive contrastive learning, characterized in that, include: Obtain a multimodal dataset, extract features from the text, audio, and video modalities within the dataset, and determine the text features. Audio features and video features ; Separating commonalities and unique characteristics among different modes to analyze audio features and video features Perform subspace decomposition to achieve feature subspace modeling; Text features are optimized by leveraging the feature subspaces of audio and video, employing multi-layered progressive contrastive learning, and obtaining optimized text features through a contrastive learning strategy. ; Feature enhancement based on conditional transformation, utilizing optimized text features Enhance the subspace features of audio and video to obtain enhanced invariant features. and unique characteristics ; Enhanced invariant features Unique characteristics and text features Feature stacking and dimensionality reduction are performed, sentiment weights are calculated, and sentiment perception weighted fusion and prediction are performed based on the sentiment weights.
2. The multimodal sentiment analysis method based on progressive contrastive learning according to claim 1, characterized in that, Feature extraction is performed on text, audio, and video modalities in a multimodal dataset to determine text features. Audio features and video features Perform the following operations: Obtain sequences from a multimodal dataset Where t, a, and v represent text, audio, and video, respectively; When extracting text features, the original text Input the pre-trained text model BERT and extract the initial word vectors of the last layer as text features. ; When extracting audio and video features, for audio modalities The COVAREP tool was used to extract pitch and Mel-spectral features for video modalities. The Facet tool was used to extract facial expression and head movement features. Then, the audio and video features were input into the Long Short-Term Memory (LSTM) unit for temporal modeling to obtain the output features. and .
3. The multimodal sentiment analysis method based on progressive contrastive learning according to claim 2, characterized in that, Separating commonalities and unique characteristics among different modes to analyze audio features and video features Perform subspace decomposition and execute the following operations: Using space-invariant encoders Extracting modality-invariant features from audio and video ; Utilizing space-specific encoders Extract modality-specific features from audio and video. ; We use the difference in central moments as the invariant space for the similarity loss constraint, and the orthogonality constraint as the unique space for the difference loss constraint.
4. The multimodal sentiment analysis method based on progressive contrastive learning according to claim 3, characterized in that, Text features are optimized by leveraging the feature subspaces of audio and video, employing multi-layered progressive contrastive learning, and obtaining optimized text features through a contrastive learning strategy. Perform the following operations: Anchor point selection: Select the output features of the last layer of the BERT model as the anchor point. ; Construction of difficult positive samples: Selecting output features from the first 11 layers of the BERT model As positive samples, the shallow features and deep features have different levels of semantic abstraction, which constitutes a hierarchical difficult positive sample. Construction of difficult-to-negative samples: Samples opposite to the anchor sentiment label are selected from the subspace features of audio and video as negative samples. To construct hierarchical difficult-to-negative samples, a layer-dependent pressure is applied to the negative samples. Varying dynamic Gaussian noise : （1）； in, Represents variance; Calculate the contrastive loss: Optimize using the InfoNCE loss function: （2）； Where r represents the number of BERT layers, Let K represent the set of least positive samples, and K represent the number of negative samples. Indicates being Optimized text features are obtained through a contrastive learning strategy using a noise-invariant space and a set of hard-to-bear samples in a specific space. .
5. A multimodal sentiment analysis method based on progressive contrastive learning according to claim 4, characterized in that, Feature enhancement based on conditional transformation, utilizing optimized text features Enhance the subspace features of audio and video to obtain enhanced invariant features. and unique characteristics Perform the following operations: Generate transformation matrix: transform text features Generates a product using linear layers and reshaping operations, similar to... The conditional transformation matrix for dimension matching; Non-linear activation: using the Sigmoid activation function Normalize the transformation matrix to ensure that the transformation values are within a reasonable range; Feature calibration: Comparing the transformation matrix with the original features Element-wise multiplication yields the enhanced features. : （3）； （4）； Where T represents the transformation matrix. R represents the dimension, and R represents the reshaping operation. Represents matrix multiplication. Represents the weights of the linear layer. Indicates deviation, The symbol represents the Sigmoid activation function, and ⊙ represents element-wise multiplication. Similarly, the enhanced unique spatial features can be obtained. .
6. A multimodal sentiment analysis method based on progressive contrastive learning according to claim 5, characterized in that, Enhanced invariant features Unique characteristics and text features Feature stacking and dimensionality reduction are performed, including: enhancing invariant features. Unique characteristics and text features The initial fused features U are obtained by stacking and then mapped to a low-dimensional space through a dimensionality reduction layer.
7. A multimodal sentiment analysis method based on progressive contrastive learning according to claim 6, characterized in that, To calculate the sentiment weight, perform the following operations: Sentiment weights q are calculated for the three feature spaces using fully connected layers, and feature descriptors are obtained by nonlinear activation of the dimensionality-reduced features. And use the Softmax function to generate the attention matrix c; The attention matrix c is as follows: （5）； in, ⊙ represents the Softmax function, and ⊙ represents element-wise multiplication.
8. A multimodal sentiment analysis method based on progressive contrastive learning according to claim 7, characterized in that, Based on sentiment weights, perform sentiment perception weighted fusion and prediction, and perform the following operations: The initial feature U is weighted according to the weights, and the final sentiment prediction result is output through the regression layer: （6）； in, Represents the weights of the linear layer. ⊙ represents the deviation, and ⊙ represents element-wise multiplication. The overall model uses the mean squared error as the task loss.