Text-guided sentiment analysis method of comparative learning and gating fusion mechanism

Abstract

The invention requests to protect an emotion analysis method based on a text-guided comparative learning and gating fusion mechanism, which is used for solving the problems of insufficient modal feature representation discrimination and insufficient modal information fusion in multi-modal emotion analysis. Firstly, features of three modes are extracted in a feature extraction module, then a text mode is selected as a dominant mode, a text-guided weighted comparison learning strategy is designed, the strategy screens positive and negative sample pairs through the text mode, and weights are dynamically distributed for hard pairs and easy pairs, so that the model keeps universal features while focusing on the hard pairs, and the model is optimized. Therefore, the discrimination capability of modal feature representation is improved. Secondly, in a feature fusion module, a text-guided gating fusion mechanism is provided, weights of text modals and cross-modal fusion features are adaptively adjusted through a gating unit, and a self-attention mechanism is used to suppress fusion noise, so that the modals are fused more fully. Experiments on public data sets CMU-MOSI and CMU-MOSEI show that compared with a reference method, the method provided by the invention has advanced performance on multiple indexes such as F1, ACC-2 and the like.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, computer vision, and language processing technology, and specifically to a method for multimodal sentiment analysis based on text-guided contrastive learning and gating fusion mechanism. Background Technology

[0002] With the rapid rise and popularization of social media platforms and the short video industry, users' emotional expression has become increasingly diversified and complex. Emotional carriers are no longer limited to traditional text formats but broadly encompass audio, video, and combinations of text and images. Multimodal sentiment analysis, as an important research topic, aims to understand user emotions more comprehensively and accurately by deeply exploring and synergistically utilizing the potential correlations and inherent differences between different modal data (such as text, visual, and audio). Compared to traditional methods that rely solely on a single modality, multimodal analysis can effectively compensate for the one-sidedness and ambiguity of single-modal information in expressing emotions, and has already demonstrated significant value and potential in various practical application scenarios such as fake news detection, network content security, personalized recommendation systems, and human-computer intelligent interaction. Although this field has attracted the attention of many researchers in recent years and achieved many remarkable results, current mainstream methods still face two key and fundamental challenges when dealing with real-world complex scenarios: first, how to learn modal feature representations with strong discriminative power; and second, how to achieve sufficient and efficient cross-modal information fusion.

[0003] To address the issue of insufficient discriminative power in modal feature representations, traditional research often employs structures such as Long Short-Term Memory (LSTM) networks or Transformers for intra-modal feature modeling, but frequently neglects the semantic relationships between samples. This lack of inter-sample interaction may lead to semantically similar samples failing to learn consistent feature representations, thereby weakening the model's generalization ability and feature discriminative power. Although supervised contrastive learning improves feature robustness by constructing positive and negative sample pairs to bring similar samples closer together and push away dissimilar ones, current methods have significant limitations: on the one hand, they overemphasize hard sample pairs that are difficult to distinguish, while ignoring the positive role of easily distinguishable sample pairs in stabilizing training and preventing overfitting; on the other hand, they fail to effectively utilize the dominant role of textual modalities in multimodal semantic alignment, limiting further improvements in feature representation capabilities.

[0004] To address the issue of insufficient cross-modal information fusion, existing research generally employs fusion networks based on cross-modal attention to bridge the semantic gap between modalities. However, most methods suffer from two limitations: first, most studies assign equal weights to all modalities, and this modal balance-based design does not adequately consider the relative importance of different modalities, potentially leading to insufficient fusion of information across modalities; second, using cross-modal attention mechanisms can easily introduce noise and redundant information during modal fusion, such as non-emotionally relevant features in audio / images that may interfere with the semantic expression of the text. Summary of the Invention

[0005] This invention aims to address the problems in existing multimodal sentiment analysis, such as insufficient discriminative power of modal feature representations, inadequate cross-modal information fusion, and model sensitivity to noise and misaligned data. To this end, a multimodal sentiment analysis method based on text-guided contrastive learning and gating fusion mechanism is proposed.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] S1. First, a single-modal feature extraction module is established. This module is responsible for extracting high-level feature representations from the raw data of text, audio, and video modalities respectively. Text features are obtained by a pre-trained language model to capture its semantic and emotional information; audio and video features are modeled through corresponding deep feature extraction networks to extract low-level features such as speech intonation, rhythm, and facial expression changes, providing high-quality input for subsequent multimodal alignment and fusion.

[0008] S2. Secondly, a text-guided weighted contrastive learning module is constructed. This module is one of the core innovations of this invention, aiming to enhance feature discriminativeness by guiding the modeling of positive and negative relationships between samples through highly reliable text modal information. Specifically, the method first normalizes the sentiment labels to adapt to the differences in annotation scales across different datasets; based on this, positive and negative sample pairs are divided according to the sentiment distance between samples, achieving hierarchical contrastive learning across samples and modalities. Unlike existing methods, this invention consistently uses the text modality as the anchor and dominant modality during the construction of positive and negative samples, guiding the learning direction of audio and video modalities through the stability and reliability of its semantic expression, effectively reducing the interference of modal ambiguity on contrastive learning. Simultaneously, to improve the robustness and efficiency of the model during training, this invention designs a dynamically weighted contrastive loss function. This loss function adaptively adjusts the learning weights of different sample pairs through a smooth, non-linear weighting function. It assigns higher weights to "hard pairs" that are difficult to distinguish and moderate weights to "easy pairs" that are easy to distinguish, thereby achieving a balance and optimization of the learning process. This allows the model to maintain training stability and prevent overfitting while bringing positive samples closer to negative samples and pushing them further away from negative samples.

[0009] S3. After obtaining the weighted and optimized single-modal feature representation, this invention further proposes a text-guided gated fusion network to achieve efficient cross-modal feature fusion. This network takes the text modality as the primary driver, dynamically retrieving key information from audio and video modalities through a cross-modal attention mechanism, and using text semantics as the core to guide feature alignment and fusion of other modalities. To avoid interference caused by temporal or semantic misalignment between different modalities, an adaptive gating mechanism is introduced into the network. This mechanism dynamically adjusts the fusion ratio of features from each modality based on text information, strengthening the transmission of relevant information and suppressing redundant or noisy information, thereby improving the discriminativeness and robustness of the fused features. The fused multimodal features are further refined by a self-attention optimization layer and a feedforward neural network to form a unified representation with high semantic consistency and complementarity.

[0010] S4. In the prediction phase, the model utilizes fused feature inputs into a multilayer perceptron structure to output the final sentiment intensity value. During model training, the contrastive learning loss and sentiment prediction loss are simultaneously optimized to achieve a balance between feature discriminativeness and prediction accuracy.

[0011] The beneficial effects of this invention are as follows: through the synergistic effect of text-guided contrastive learning strategy and gating fusion mechanism, the model can not only fully explore the semantic associations and complementary relationships between multimodal data, but also significantly improve the discriminative power and generalization ability of feature representation. Compared with existing methods, this invention exhibits higher robustness and stability in handling complex scenarios such as modality misalignment, ambiguous emotional signals, and noise interference, providing an efficient, reliable, and scalable solution for multimodal sentiment analysis tasks. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the sentiment analysis framework based on text-guided contrastive learning and gating fusion mechanism of the present invention.

[0013] Figure 2 This is a schematic diagram of the sentiment analysis framework based on text-guided contrastive learning and gating fusion mechanism of the present invention.

[0014] Figure 3 This is a schematic diagram of the sigmoid-based weight function in the text-guided contrastive learning of this invention.

[0015] Figure 4 This is a schematic diagram of the text-guided gating fusion network of the present invention.

[0016] Figure 5 This is a visualization of the T-SNE representation of the multimodal fusion method of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0018] This invention proposes a sentiment analysis method based on contrastive learning and gating fusion mechanism. The main framework of the method is shown in the attached figure. Figure 2 As shown, the specific implementation process is as follows:

[0019] S1. Single-peak feature extraction module, extracts high-level feature representations of different modalities.

[0020] First, this invention extracts high-level feature representations from three modalities: text, audio, and video. For the text modality, a pre-trained BERT model is used to extract low-level text features X. t Processing is performed to obtain high-level text features F t For audio modalities, the CAVAREP tool is used to extract low-level acoustic features X. a This is then fed into the Transformer encoder to obtain the high-level audio features F. a For video modalities, the FACET tool is used to extract low-level visual features X. v This is then fed into the Transformer encoder to obtain the high-level video features F. v To better capture the overall semantic information of each modality, this invention adds a special [CLS] marker before the feature sequence of each modality. In subsequent processing, this marker gradually aggregates the contextual information of the entire sequence through the model structure, thereby forming a global representation of the entire modality content. This feature representation with the [CLS] marker not only includes the abstraction of local features but also integrates global semantic information, providing a more discriminative feature foundation for subsequent contrastive learning tasks.

[0021] The specific implementation steps of step S1 are as follows:

[0022]

[0023] in, This represents the parameters of the BERT model. and These represent the parameters of the Transformer encoder. The high-level features of the three modes can be represented as follows: T m d represents the length of the modal sequence. m Represented as modality embedding dimension. Linear m Linear layers representing different modes, Linear m The parameter H mThe [CLS] label feature representation represents the features after passing through a linear layer.

[0024] S2. Based on the [CLS] identifier of different modalities, perform basic-guided weighted comparison learning.

[0025] To enhance the discriminative power of modal features, this invention introduces a contrastive learning mechanism. However, different modalities exhibit significant differences in their reliability in expressing emotions: text modalities typically contain explicit emotional vocabulary and syntactic structures (such as "great" and "frustrating"), directly and clearly conveying emotional semantics; while audio modalities are susceptible to noise, accents, and speech rate interference, and their intonation and pitch cues may be blurred; video modalities are limited by factors such as lighting, occlusion, and individual expression habits, leading to ambiguity in facial expression recognition.

[0026] To address this issue, this invention proposes a "text-guided weighted contrastive learning" mechanism. The core idea of ​​this method is to use the text modality as a guiding modality to direct the contrastive learning process of other modalities (audio and video), thereby mitigating the negative impact caused by the reliability differences between modalities.

[0027] The specific implementation steps of step S2 are as follows:

[0028] In supervised contrastive learning based on sentiment labels, the separation of positive and negative sample pairs needs to consider the distance between labels. Due to scale differences in the labeling systems of different datasets (e.g., inconsistent sentiment polarity rating ranges), directly using the original label values ​​may lead to inconsistent contrastive loss weights between different datasets. Therefore, this invention employs a label normalization strategy, mapping the original sentiment rating lines to the same range [-1, 1]. For two samples i and j in a batch, assuming i is the anchor point, the positive and negative pairs are calculated using the following formula:

[0029]

[0030] Among them, y m Let m∈(i,j) represent the sentiment polarity value, and the hyperparameter z represent the threshold for sample labels. If the sentiment polarity difference exceeds this threshold, it indicates a large difference in sentiment between samples, and they should be treated as negative pairs; if the difference does not exceed the threshold, they should be treated as positive pairs.

[0031] Furthermore, a text-guided sample pair construction strategy is proposed. By designing a sample pair selection strategy based on text modality, semantic ambiguity in multimodal data can be suppressed, and the robustness of the model's feature representation can be improved. Specifically, it is divided into interactions between different samples of the same modality, between different samples of different modalities, and between the same sample of different modalities. For anchor point i, the selection of positive and negative sample pairs is as follows:

[0032] 1. Between different samples of the same modality:

[0033]

[0034] 2. Between different samples and different modalities:

[0035]

[0036] 3. Between different modalities of the same sample:

[0037]

[0038] Let m ∈ {t, a, v} represent a sample pair indicating an anchor point, where m represents text (t), audio (a), and video modality (v), respectively. Compared to other studies, this invention chooses the text modality as the dominant modality in constructing cross-modal positive and negative sample pairs. Compared to the ambiguity of facial expressions in images and the semantic ambiguity of audio, the text modality can explicitly express emotional semantics through sentiment words and syntax, thus exhibiting higher emotional reliability than image and audio modalities. This strategy makes the contrastive learning process more refined, effectively utilizing information between different modalities to obtain a more robust unimodal representation.

[0039] Furthermore, in pair-based contrastive learning methods, the design of the loss function typically relies on the comparison of positive and negative sample pairs. To improve the representational power of modal features, existing research largely focuses on identifying hard pairs—sample combinations that the model struggles to distinguish correctly. Specifically, hard pairs can be divided into two categories: hard positive pairs and hard negative pairs. Hard positive pairs refer to sample pairs belonging to the same category but with significantly different representations, easily misclassified as dissimilar; hard negative pairs refer to sample pairs from different categories that are easily confused due to semantic or appearance similarity. In contrast, easy pairs can be easily and correctly classified by the model.

[0040] While hard pairs play a crucial role in improving model performance, this invention argues that completely ignoring the role of easy pairs may lead to instability in the model learning process and even overfitting. In fact, easy pairs help the model quickly establish basic feature representations and stabilize the convergence direction in the early stages of training. Therefore, this invention designs a dynamic weighting function based on the Sigmoid function (as shown in the appendix). Figure 3 As shown in the diagram, different weights are assigned to sample pairs based on their contributions. The smooth and asymptotic nature of the sigmoid function allows it to naturally assign higher weights to hard pairs while appropriately reducing the weights of standard pairs, ensuring a more balanced and robust model when handling different sample pairs. The weight function is designed as follows:

[0041]

[0042] After the above steps, the loss function for contrastive learning is as follows:

[0043]

[0044] B is the number of samples in a batch, and w(i,j) is the weight function. However, this traditional loss function easily falls into suboptimal solutions, meaning that while the distance between negative samples has been pushed to its maximum, the distance between positive samples has not yet approached its minimum. To address this issue, Euclidean distance is introduced to improve the loss function. Euclidean distance directly measures the geometric distance between samples, allowing the loss function to not only maximize the distance between negative samples but also minimize the distance between positive samples. This combined approach enables the loss function to more effectively bring positive samples closer and push negative samples further apart in the feature space, thereby improving the model's robustness and effectiveness.

[0045]

[0046] Finally, the text-guided contrastive learning loss function is as follows:

[0047] L cl =λL weightSup +βL distance

[0048] S3. Text-guided gated fusion network (as attached) Figure 4 (As shown).

[0049] Text-Guided Gated Fusion Network (TGFN) is the key multimodal fusion module proposed in this invention. It aims to guide and regulate the feature fusion of multiple modalities, including audio, video, and text, using information from the text modality. Specifically, after preprocessing, audio, video, and text modalities generate high-level feature representations. These representations contain basic information about each modality, but individual modalities often struggle to fully capture the complex relationships and sentiment information within multimodal data.

[0050] Against this backdrop, TGFN introduces a gating mechanism to dynamically adjust and fuse features from other modalities based on textual information. Specifically, the text modality is considered the dominant modality. Through refined modeling of textual features, it guides the feature representation of audio and video modalities, ensuring that the fused features more accurately reflect sentiment information. The gating mechanism weights and fuses features from other modalities based on the correlation between different modalities and the guiding information from the text modality, thereby enhancing the complementarity between multimodalities and improving the accuracy of sentiment analysis. In TGFN, the gating layer plays a central role. It adjusts the information transmission method of audio and video modalities according to the feature weights of the text modality, ensuring that textual information plays a dominant role in the final decision during feature fusion, while information from other modalities is appropriately weighted according to their relevance. In this way, the model can effectively handle information redundancy and missing information in multimodal inputs, ensuring that each modality contributes the most valuable information to the final sentiment prediction.

[0051] The specific steps for S3 are as follows:

[0052] First, the preprocessed text features F t The input is fed into a first-layer self-attention mechanism to enhance the contextual modeling capability within the text. This mechanism highlights key semantic units by calculating the correlations between positions within a sequence, thereby obtaining more discriminative context-enhanced text features.

[0053]

[0054] in, These are the projected weight matrices for the query (Q), key (K), and value (V), respectively. Add & Norm represent the residual network and normalization. T represents the context-enhanced text feature representation. t and d t These represent the length of the text sequence and the embedding dimension, respectively.

[0055] Secondly, to effectively promote feature fusion between different modalities and capture the interaction relationships of multimodal sequences at different time steps, this study introduces a cross-modal attention mechanism. In this mechanism, context-enhanced text features are selected as the query, while high-level feature representations of video and audio modalities serve as the key and value, respectively. Through this design, the model can dynamically focus on key features in video and audio, guided by deepened text semantics, thereby achieving selective fusion of multimodal information. This method not only enhances the alignment and complementarity between modalities but also effectively improves the model's ability to model cross-modal correlations. The specific formula is as follows:

[0056]

[0057] Where m∈{a,v},

[0058] However, data from different modalities are often not perfectly aligned in time and space. Directly relying on cross-modal attention mechanisms for modal information fusion may lead to insufficient information fusion or excessive noise, causing the model to overemphasize irrelevant information and thus affecting its performance. Therefore, this invention proposes using the text modality as the primary modality and introducing an adaptive gating mechanism. This mechanism dynamically adjusts the weights of the cross-modal fusion features and the text features, outputting gated and refined cross-modal features. The specific formula is as follows:

[0059]

[0060] Considering the potential introduction of irrelevant information, redundant features, and even noise interference during cross-modal fusion, this invention further inputs the fused features into a second-layer self-attention mechanism to obtain noise-suppressed fused features. This mechanism, while further optimizing and refining the cross-modal interaction results, helps suppress the influence of irrelevant noise, enhancing the discriminative ability and semantic consistency of the features. The formula is as follows:

[0061]

[0062] Finally, the model of the present invention includes a feedforward neural network, which consists of fully connected layers and a ReLU activation function, and includes a residual mechanism to further optimize the encoded feature representation through the feedforward neural network, thereby obtaining the final cross-modal fusion feature.

[0063] S4. Predict a person's emotions based on the intensity of their emotions.

[0064] After obtaining the final multimodal fusion representation, it needs to be mapped to sentiment intensity prediction values, and the model is then optimized end-to-end using a joint loss function. This method employs a multilayer perceptron for sentiment prediction, combining task loss and contrastive learning loss to guide model training.

[0065] The specific implementation steps of step S4 are as follows:

[0066]

[0067] L = L task +w1L cl

[0068] Among them, F at and F vt Represents the fusion features, where Representing the connection fusion feature, θ MLP B represents the learnable parameters of the MLP layer, and B represents the number of samples in a batch. Indicates the strength of sentiment prediction. L task Let w1 represent the loss of the sentiment prediction task, L represent the overall loss function, and w1 be a hyperparameter.

[0069] The above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that various modifications, changes, or equivalent substitutions can be made to the above embodiments without departing from the principles and spirit disclosed in the present invention. The scope of protection of the present invention should be determined by the scope defined in the claims and their equivalents.

Claims

1. A sentiment analysis method based on text-guided contrastive learning and gate fusion mechanism, characterized in that, Comprising the following steps:

101. Collecting original video data containing multiple modalities, and extracting low-level modal feature representations of text, audio and vision therefrom; 102. Encoding the low-level modal features extracted in step 101 to extract high-level modal feature representations thereof; 103. Taking the text modality as an anchor point, performing a text-guided weighted contrastive learning task on the high-level modal feature representations obtained in step 102; 104. Inputting the high-level modal feature representations obtained in step 102 into a text-guided gating fusion network to perform cross-modal interaction and fusion, and outputting three-modal fusion feature representations rich in complementary information; 105. Inputting the multi-modal fusion feature representations obtained in step 104 into a classifier to predict the final sentiment category.

2. The method of claim 1, wherein, The extraction of low-level modal features and the encoding of high-level modalities in steps 101 and 102 are as follows: 201、The high-level modality feature representation of the text modality is extracted using the pre-trained language model BERT, and the calculation process is 202、Audio and visual modalities low-level features are extracted using COVAREP and Facet tools respectively, and then the random initialized Transformer encoder is used to capture their temporal relationship, the calculation process is where X m∈{t,a,v} represents the low-level features of each modality, represents the parameters of the BERT model, and represents the parameters of the Transformer encoder. At this time, the high-level features of the three modalities can be represented as T m represents the length of the modality sequence, d m represents the modality embedding dimension.

3. The method of claim 1, wherein, The text-guided weighted contrastive learning task in step 103 includes:

301. After the audio, video and text modalities are processed in step 101, [CLS] is used for contrastive learning. In order to ensure that the embedding dimensions of the three modalities are consistent, the following fully connected layer is used: where Linear m represents a linear layer for different modalities, represents Linear m parameters, H m represents the [CLS] token feature representation passed through a linear layer.

302. Due to the scale difference of the annotation system of different data sets, the present application adopts a label normalization strategy to map the original sentiment score line to the same [-1, 1]. For two samples i, j in a batch, assuming i is the anchor point, the positive and negative pairs are calculated as follows: where y m , m e (i, j) represents the sentiment polarity value, and z is a hyperparameter representing the threshold value of the sample label.

303. A text-guided sample pair construction strategy is proposed. Specifically, it will be divided into interaction between different samples of the same modality, between different samples of different modalities, and between different modalities of the same sample. For anchor point i, the selection of positive and negative sample pairs is as follows: Between different samples of the same modality: Between different samples of different modalities: Between different modalities of the same sample: wherein, m e {t,a,v} denotes one sample pair of anchor points, m e {t,a,v} represents text (t), audio (a) and video modality (v) respectively.

4. The method of claim 1, wherein, When calculating the loss of the text-guided weighted contrastive learning in step 104:

401. The present application designs a dynamic weighting function based on the Sigmoid function, which gives different weights to sample pairs according to their contributions. The design of the weight function is as follows: After step 401, the loss function of the contrastive learning is as follows:

402. Further, in order to avoid the contrastive learning loss function falling into a suboptimal solution, the Euclidean distance is introduced to improve the loss function:

403. Finally, the text-guided contrastive learning loss function is as follows: L cl = λL weightSup + βL distance。 5. The method of claim 1, wherein, The text-guided gating fusion network in step 104 includes: 501、the preprocessed text features F t input to the first layer self-attention mechanism, thereby obtaining more discriminative context-enhanced text features: where, are the projection weight matrices for queries (Q), keys (K) and values (V) respectively, Add&Norm represents the residual network and normalization, denotes the context-enhanced text feature representation, T t and d t denote the length of the text sequence and the embedding dimension respectively.

502. Cross-modal attention mechanism is introduced, which uses text features as a guide for fusion: where m e {a, v}, 503. An adaptive gating mechanism is introduced to dynamically adjust the cross-modal fusion feature weight and the weight of the text feature, and output the cross-modal feature after gating refinement:

504. Further input the fused features into the second layer of self-attention mechanism to obtain noise-suppressed fusion features: 。 6. The method of claim 5, wherein, The text-guided gated fusion network takes the text modality as the main guide, dynamically retrieves key information in the audio and video modalities through a cross-modal attention mechanism, and guides the alignment and fusion of features in other modalities with the text semantic as the core. In order to avoid the interference caused by the time or semantic misalignment between different modalities, an adaptive gating mechanism is introduced in the network, which dynamically adjusts the fusion ratio of features in each modality according to the text information, strengthens the transmission of relevant information and suppresses redundant or noise information, thereby improving the discriminability and robustness of the fused features.

7. The method of claim 1, wherein, The method for predicting the emotion of the character based on the emotional intensity in step 105 comprises: 601, the method adopts a multi-layer perception machine to perform emotion prediction, and combines a task loss and a contrast learning loss to jointly guide model training: L = L task + w1L cl where F at and F vt denote fusion features, where denote concatenation of fusion features, θ MLP denote the learnable parameters of the MLP layer, B denotes the number of batch samples, denote the sentiment prediction strength. L task denote the sentiment prediction task loss, L denotes the overall loss function, w1 is a hyperparameter.

Images