Multimodal sentiment analysis method based on comparative decomposition feature multitask network

Abstract

The invention discloses a multi-modal sentiment analysis method based on a comparative decomposition feature multi-task network, and belongs to the technical field of multi-modal sentiment analysis. The problems that in the prior art, information interaction between different sample modes is difficult to achieve through a traditional multi-mode emotion analysis method, and single-mode labels in a multi-mode emotion data set are generally lacked or manual labeling is inaccurate are solved. The method comprises the following steps: constructing a multi-modal sentiment analysis model, inputting a to-be-processed data set into the multi-modal sentiment analysis model for feature extraction and decomposition, and integrating obtained six decomposition features; according to the multi-mode sentiment prediction loss, the contrast learning loss and the single-mode sentiment prediction loss, completing a multi-mode sentiment analysis task, a contrast learning optimization task and a single-mode sentiment analysis task; and performing double-layer loss optimization on the multi-modal sentiment analysis model, constructing a multi-task loss function, and outputting a multi-modal sentiment analysis result. The method improves the accuracy of the multi-modal sentiment analysis model, and can be applied to multi-modal sentiment classification.

Description

Technical Field

[0001] The present invention relates to a multimodal sentiment analysis method, in particular to a multimodal sentiment analysis method based on a contrast decomposition feature multi-task network, and belongs to the technical field of multimodal sentiment analysis. Background Art

[0002] Interpersonal communication is always carried out through the comprehensive expression and fusion of information from different modalities. Multimodal sentiment analysis can complement and eliminate ambiguity between information from different modalities, better understand and apply the rich emotional information of each modality, and is widely used in multiple fields such as artificial intelligence and human-computer interaction.

[0003] In the prior art, the trimodal sentiment analysis task was first proposed, and a method of automatically extracting multimodal features was used to connect them together, and a trimodal HMM classifier was used to learn the input information of different modalities. Fusion of different modal information is the key to the multimodal sentiment analysis (MSA) task. Some scholars proposed a tensor fusion network (TFN) model, which uses tensor representation and tensor operation to fully combine the unimodal, bimodal and trimodal interactions in tensor fusion. The multimodal (MulT) Transformer proposed by some scholars directly focuses on the low-level features in other modalities through the cross-modal attention mechanism to fuse multimodal information and strengthen the target modality, and finally performs end-to-end inter-modal fusion. In order to better In studying the impact of modal-specific information on MSA, some scholars considered unimodal sentiment prediction and decomposed each modality into modality-invariant representation and modality-specific representation. However, when normalizing the prediction results, they treated all modalities equally, ignoring the different learning capabilities of feature extraction tools in different modalities. The text modality representation ability is often stronger than that of visual and audio modalities, and is more effective in multimodal sentiment analysis tasks. Some scholars designed a label generation module based on a self-supervised learning strategy to obtain independent unimodal labels, completely using the late fusion method, ignoring the interaction between the modalities. Some scholars proposed a contrastive learning method HyCon, which takes into account both inter-sample and intra-sample contrast information. However, in inter-sample contrastive learning, positive and negative pairs are only randomly extracted based on labels.

[0004] Transformer is a deep learning model architecture that includes self-attention mechanism, multi-head attention, position encoding, feedforward neural network, residual connection and layer normalization. The above modules and the special encoder-decoder structure make the model have good parallelism and global context capture capabilities. It has achieved remarkable success in various tasks in the field of natural language processing and has been widely used in other fields such as image processing, speech recognition, and multimodality. BERT (Bidirectional Encoder Representations from BERT is a pre-trained natural language processing model. BERT uses the self-attention mechanism of the Transformer architecture and adopts a two-way context modeling approach. It considers the left and right context information of a word at the same time, understands the context more comprehensively, and learns common language representations from a large amount of text data through large-scale unsupervised learning. Then, it is fine-tuned according to downstream tasks. The BERT model has strong generalization capabilities in different languages, fields, and tasks, and has caused revolutionary changes in the field of natural language processing. The key issue of contrastive learning is the construction of contrast pairs and the determination of contrast relationships, which reduces the distance with positive samples and expands the distance with negative samples, thereby achieving the true distance of their original spatial distribution, so that the model learns feature representations through contrastive losses. Some scholars have proposed to learn the graph-level representations of nodes and edges by comparing the encodings obtained in different views, and complete the task of graph classification. To solve the problem of data imbalance, some scholars proposed a self-supervised framework SimCLR, which learns visual representations through contrast loss between random transformation of the same image sample and the augmented view generated by the encoder; some scholars also proposed supervised contrast loss, which does not require large batches to solve the problem of data imbalance, combines supervised contrast learning with curriculum learning, and reduces the impact of extreme samples; some scholars proposed a multimodal emotion knowledge sharing framework (UniMSE), which unifies the multimodal sentiment analysis (MSA) and emotion recognition in conversations (ERC) tasks from features, labels, and models, performs modal fusion at the syntactic and semantic levels, and introduces contrastive learning between modalities and samples to better capture the differences and consistency between emotions and sentiments.

[0005] In summary, a multimodal sentiment analysis method based on contrastive decomposition feature multi-task network is needed. Summary of the invention

[0006] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0007] In view of this, in order to solve the problems in the prior art that traditional multimodal sentiment analysis methods are difficult to achieve information interaction between different sample modalities, and that single modal labels are generally missing or manually labeled inaccurately in multimodal sentiment datasets, the present invention provides a multimodal sentiment analysis method based on a contrastive decomposition feature multi-task network.

[0008] The technical solution is as follows: A multimodal sentiment analysis method based on a contrastive decomposition feature multi-task network includes the following steps:

[0009] S1. Build a multimodal sentiment analysis model, input the data set to be processed into the multimodal sentiment analysis model for feature extraction and decomposition, and obtain six decomposition features;

[0010] S2. Integrate the six decomposed features, use MLP to stack multi-layer linear classifiers to obtain multimodal analysis results, and complete the multimodal sentiment analysis task based on the multimodal sentiment prediction loss obtained;

[0011] S3. Integrate the six decomposition features, perform contrastive learning optimization on the multimodal sentiment analysis model, and complete the contrastive learning optimization task based on the contrastive learning loss obtained;

[0012] S4. Integrate the six decomposed features, and through self-supervised learning, stack multi-layer linear classifiers with weight-sharing MLP to obtain the unimodal analysis results. According to the solved unimodal sentiment prediction loss, complete the unimodal sentiment analysis task;

[0013] S5. Based on the multimodal sentiment prediction loss, the unimodal sentiment prediction loss and the contrastive learning loss, the multimodal sentiment analysis model is optimized by double-layer loss to obtain the multimodal sentiment analysis model after double-layer loss optimization;

[0014] S6. Based on the multimodal sentiment prediction loss, unimodal sentiment prediction loss and contrastive learning loss, a multi-task loss function is constructed to train the multimodal sentiment analysis model after double-layer loss optimization, and the multimodal sentiment analysis results are output.

[0015] Furthermore, in said S1, the network architecture of the multimodal sentiment analysis model includes a text pre-training model BERT and two Transformer encoding layers;

[0016] The audio features of the audio data in the processing data set are extracted through a Transformer encoding layer to obtain the audio feature A. The dimension is transformed through the Dense layer for subsequent feature interaction. Each mode of the audio feature A is decomposed into audio similarity feature A through the similarity projection layer and the dissimilar projection layer. s Dissimilar feature A to audio d ;

[0017] Through another Transformer encoding layer, the video features in the video data to be processed are extracted to obtain the video features V. The dimensions are transformed through the Dense layer for subsequent feature interaction. Each mode of the video features V is decomposed into video similarity features V through the similarity projection layer and the dissimilar projection layer. s Dissimilar features from the video V d ;

[0018] The text feature T is obtained by extracting text features from the text data through the text pre-training model BERT. The dimension is transformed through the Dense layer for subsequent feature interaction. Each mode of the text feature T is decomposed into text similarity feature T through the similarity projection layer and the dissimilar projection layer. s Dissimilar features to the text T d .

[0019] Furthermore, in S2, the video similarity feature V is obtained. s After the video dissimilar feature V, the six decomposed features are integrated through the concat function, namely the audio similarity feature A s , audio dissimilar feature A d , video similarity features V s , video dissimilar features V, text similar features T s Dissimilar features to the text T d , use MLP stacking multi-layer linear classifier to obtain multimodal analysis results, according to the multimodal sentiment prediction loss L m , complete the multimodal sentiment analysis task;

[0020] Multimodal sentiment prediction loss L m It is expressed as:

[0021]

[0022] in, is the result of multimodal analysis. is the multimodal sentiment label, and n is the total number of samples.

[0023] Furthermore, in S3, the text similarity feature T is obtained. s Dissimilar features to the text T dFinally, the decomposed features are integrated to perform comparative learning optimization on the multimodal sentiment analysis model;

[0024] The specific steps of contrastive learning optimization are as follows:

[0025] The intra-sample comparison and inter-sample comparison are unified into the NT-Xent contrast loss framework, using the text similarity feature of the i-th information As an anchor point, establish a contrast relationship between decomposed features

[0026] Decomposing the contrast relationship between features It is expressed as:

[0027]

[0028] Among them, (a, p) and (a, k) are a pair of feature vectors decomposed in a sample, sim is the similarity function, exp is the exponential function, and τ is the temperature coefficient;

[0029] During the intra-sample and inter-sample comparisons, construct the positive example set P i and negative example set N i ;

[0030] Positive example set P i It is expressed as:

[0031]

[0032] in, is the positive pair within the sample, The samples are aligned;

[0033] Negative example set N i It is expressed as:

[0034]

[0035] in, is a negative pair in the sample, is a negative pair between samples;

[0036] Positive in sample It is expressed as:

[0037]

[0038] in, is the audio similarity feature of the i-th information, is the video similarity feature of the i-th information, is the audio similarity feature of the jth information, is the video similarity feature of the jth information, is the text similarity feature of the jth information, Neighbor i is the positive example set of the i-th information, Outlier i is the negative example set of the i-th information;

[0039] In-sample negative pair It is expressed as:

[0040]

[0041] in, is the audio dissimilarity feature of the i-th sample, is the video dissimilar feature of the i-th sample, is the text dissimilar feature of the i-th sample, is the audio dissimilarity feature of the jth sample, is the video dissimilar feature of the jth sample, is the text dissimilar feature of the jth sample;

[0042] Positive alignment between samples It is expressed as:

[0043]

[0044] Negative pair between samples It is expressed as:

[0045]

[0046] in, is the audio similarity feature of the k-th information, is the video similarity feature of the k-th information, is the text similarity feature of the k-th information;

[0047] After contrastive learning optimization, the loss function L is optimized according to contrastive learning cl , Complete the contrastive learning optimization task.

[0048] Furthermore, in S4, the audio similarity feature A is obtained s Dissimilar feature A to audio d After that, the decomposed features are integrated, and the multimodal sentiment analysis model is combined with the multimodal comprehensive sentiment label y given in the dataset to be processed through the label generation module. m , according to the relative distance α from the modal representation to the cluster center i Get the offset δ sm , generate unimodal sentiment label y s , based on the momentum update strategy, the final unimodal sentiment label value after i iterations is obtained After self-supervised learning, the single-modal analysis results are obtained by stacking multi-layer linear classifiers with weight-sharing MLP According to the unimodal sentiment prediction loss L u , complete the unimodal sentiment analysis task;

[0049] The specific offset calculation process is as follows: Get the positive center of different modal representations and the negative center of different modal representations, and use L2 regularization to represent a single mode The distance from the cluster center, i.e., the single mode Distance from the exact center of different modal representations and unimodal Distances from negative centers of different modal representations Define the relative distance value ε and evaluate the relative distance α from the modal representation to the class center i ;

[0050] The exact center of different modal representations It is expressed as:

[0051]

[0052] in, is the global representation of the jth sample in modality i, i∈{m,t,a,v}, m represents a multimodal task, t represents text modality, a represents audio modality, v represents video modality, and y i are emotion labels of different modalities, I is the indicator function, and N is the number of training samples;

[0053] Negative center of different modal representations It is expressed as:

[0054]

[0055] The mode represents the relative distance α to the cluster center i It is expressed as:

[0056]

[0057] Unimodal sentiment label y s It is expressed as:

[0058]

[0059] Where s∈{t,a,v};

[0060] The final unimodal sentiment label value after i iterations It is expressed as:

[0061]

[0062] in, represents the unimodal sentiment label value generated by the label generation module after the i-th iteration, Represents the final unimodal sentiment label value after i-1 iterations;

[0063] Unimodal sentiment prediction loss L u It is expressed as:

[0064]

[0065] in, Decompose the unimodal sentiment analysis results for each.

[0066] Furthermore, in S5, the optimization parameter φ and the parameter ω of the backbone basic model are set, based on the multimodal sentiment prediction loss L m , unimodal sentiment prediction loss L u and contrastive learning loss L cl , double-layer loss optimization for multimodal sentiment analysis models;

[0067] The two-layer loss optimization process is expressed as:

[0068]

[0069] L aux =L u +L cl

[0070] Among them, L aux To assist in unifying the loss, ω * is the upper layer decision vector, φ * are the model parameters to be optimized in the lower layer.

[0071] Furthermore, in S6, the multi-task loss function L is expressed as:

[0072] L=L m +β u L u +β cl L cl

[0073] Among them, L m is the multimodal sentiment prediction loss, L u is the unimodal sentiment prediction loss, L cl is the contrastive learning loss, β u Learning rate for unimodal sentiment analysis task, β cl is the learning rate of the comparative learning process.

[0074] The beneficial effects of the present invention are as follows: the present invention proposes a multimodal sentiment analysis model based on a contrast decomposition feature multi-task network, decomposes each modality into similar features and dissimilar features, and uses the similar features of the text as anchor points to establish a contrast relationship between all decomposed features; based on a data sampler, the range of contrast is expanded, and inter-sample contrast learning and intra-sample modal decomposition contrast are completed in a simple unified loss function; a label generation module is designed, and self-supervised learning is used to generate sentiment labels for single-modal tasks, so as to solve the problem that the missing single-modal labels of multimodal sentiment data sets affect the classification accuracy, and provide more accurate single-modal labels for the prediction of the results of each modality by the dissimilar features in the single-modal sentiment prediction loss; based on the above-mentioned multimodal representation learning, a multi-task prediction loss that depends on the representation of each decomposed modality is further introduced, so that the model can learn from multimodal prediction and single-modal prediction, and can capture the consistent information shared between different modalities and retain the sentiment information of a specific modality; a single loss function is nonlinearly converted through a double-layer loss optimization unit, and combined into an effective unified sentiment classification loss;

[0075] The indicators of the evaluation model are 2-category accuracy (Acc-2), 5-category accuracy (Acc-5), 7-category accuracy (Acc-7), F1 score (F1-score), mean absolute error (MAE) and Pearson correlation coefficient (Corr). Among them, Acc-2 and F1-score have two forms: Has0 / Non0;

[0076] With reference to the evaluation indicators, the following baseline models are selected to verify the performance of the multimodal sentiment analysis model on the multimodal sentiment analysis task through comparative experiments. The baselines can be divided into early multimodal fusion methods, such as low-rank multimodal fusion method LMF, tensor fusion network TFN and multimodal factor decomposition model MFM, as well as methods that focus on modal consistency and difference by modeling modal interaction fusion, such as multimodal TransformerMulT. Among them, the baseline model MISA controls the modal representation space, the baseline model MAG-BERT designs a fusion gate, the baseline model HyCon hybrid contrastive learning of trimodal representation, the baseline model Self-MM uses multi-task learning to learn from unimodal predictions, and the baseline model MMIM hierarchically maximizes the true information;

[0077] The results of the comparative experiments show that the present invention is superior to the above baseline models in all indicators on the two datasets CMU-MOSI and CMU-MOSEI. Compared with the best baseline model MMIM, the multimodal sentiment analysis model of the present invention increases the Acc-2 containing 0 by 1.45%, the Acc-2 not containing 0 by 2.37%, the F1 value containing 0 by 1.66%, the F1 value not containing 0 by 2.44%, and the Acc-7 in multi-classification by 1.82%. On the CMU-MOSEI dataset, the Acc-2 containing 0 increases by 1.94%, and the F1 value containing 0 increases by 1.19%. The excellent classification effect proves that the model proposed by the present invention can provide better sentiment features for downstream classifiers for classification, and the multi-classification Acc-7, MAE and Corr indicators on the two datasets are better or comparable to most baseline models, and better binary classification results can be obtained.

[0078] In order to prove the effectiveness of different modules of the multimodal sentiment analysis model, ablation experiments were further conducted. Refer to Table 1 for the ablation experiment results, where / on the left are the results containing 0, / on the right are the results not containing 0, -L u is the result without unimodal sentiment prediction task, -y s The results without the unimodal label generation module and the results of unimodal sentiment prediction using multimodal labels are shown in -L cl is the result without contrastive learning task, -MEL is the result without double-layer loss optimization, Inter represents the model with only inter-sample contrastive learning, Intra represents the model with intra-sample contrastive learning, Ours represents the model of the present invention, and the ablation experiment results show that the performances of the four variant models, namely the model without unimodal sentiment prediction task, the model without unimodal label generation module, the model without contrastive learning task and the model without double-layer loss optimization, are worse than the original model;

[0079] Table 1

[0080]

[0081] The unimodal prediction task can retain unimodal sentiment information for the model, and at the same time, a unimodal label generation module can be added to obtain more accurate unimodal sentiment labels, solving the problem that the missing unimodal labels of multimodal sentiment data sets affect the classification accuracy. Contrastive learning obtains more accurate information through inter-modal and intra-modal feature optimization, helping the classifier to better understand the sample, and adaptively combining various auxiliary losses to assist in the fine-tuning of the target downstream task. Usually, the visual modality and audio modality are noisier than the text modality, so more accurate predictions can be made with text as the center. Using text features as anchor points, intra-sample contrastive learning can filter out noise in the visual and audio modalities, thereby making better predictions. Inter-sample contrastive learning can obtain common information and different information between learning samples. Therefore, the model of the present invention has greatly improved the index values ​​of Acc-7 and MAE.

[0082] In order to verify the effectiveness of the model provided by the present invention in using unimodal and multimodal features to improve the performance of the model, the CH-SIMS dataset was used for comparative verification. The CH-SIMS dataset is a Chinese multimodal sentiment dataset, which not only has multimodal labels but also labels of each modality. Referring to Table 2, the accuracy and F1 value results of the two-classification of the CH-SIMS dataset using unimodal labels and only using multimodal labels indicate that the lack of unimodal labels will affect the final classification results of the sentiment. The comprehensive unimodal sentiment prediction and multimodal sentiment prediction information enable the model to make more correct judgments.

[0083] Table 2

[0084] Model <![CDATA[L u ]]> <![CDATA[-L u ]]> Acc-2 82.02 81.18 F1 81.55 80.98 BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0086] Figure 1 It is a flowchart of a multimodal sentiment analysis method based on contrast decomposition feature multi-task network;

[0087] Figure 2 This is a schematic diagram of the structure of the multimodal sentiment analysis model;

[0088] Figure 3 Schematic diagram of the pairing strategy for decomposing features during contrastive learning optimization;

[0089] Figure 4 It is a flowchart of the sampling algorithm;

[0090] Figure 5 Schematic diagram of the unimodal label generation module;

[0091] Figure 6 Schematic diagram of the two-dimensional projection of all similar decomposition features and dissimilar decomposition features on the CMU-MOSI dataset. (a) represents the text similarity feature T s Dissimilar features to the text T d , (b) represents the video similarity feature V s Dissimilar features from the video V d , (c) represents the audio similarity feature A s Dissimilar feature A to audio d , (d) represents all decomposition features before the multimodal sentiment analysis model, and (e) represents all decomposition features after the multimodal sentiment analysis model;

[0092] Figure 7 Schematic diagrams for visual image enhancement and supplementary textual information visualization;

[0093] Figure 8 Schematic diagram of pseudocode for the unimodal label generation module. DETAILED DESCRIPTION

[0094] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0095] refer to Figure 1-Figure 8 The present embodiment is described in detail, and the multimodal sentiment analysis method based on the contrast decomposition feature multi-task network specifically includes the following steps:

[0096] S1. Build a multimodal sentiment analysis model, input the data set to be processed into the multimodal sentiment analysis model for feature extraction and decomposition, and obtain six decomposition features;

[0097] S2. Integrate the six decomposed features, use MLP to stack multi-layer linear classifiers to obtain multimodal analysis results, and complete the multimodal sentiment analysis task based on the multimodal sentiment prediction loss obtained;

[0098] S3. Integrate the six decomposition features, perform contrastive learning optimization on the multimodal sentiment analysis model, and complete the contrastive learning optimization task based on the contrastive learning loss obtained;

[0099] S4. Integrate the six decomposed features, and through self-supervised learning, stack multi-layer linear classifiers with weight-sharing MLP to obtain the unimodal analysis results. According to the solved unimodal sentiment prediction loss, complete the unimodal sentiment analysis task;

[0100] S5. Based on the multimodal sentiment prediction loss, the unimodal sentiment prediction loss and the contrastive learning loss, the multimodal sentiment analysis model is optimized by double-layer loss to obtain the multimodal sentiment analysis model after double-layer loss optimization;

[0101] S6. Based on the multimodal sentiment prediction loss, unimodal sentiment prediction loss and contrastive learning loss, a multi-task loss function is constructed to train the multimodal sentiment analysis model after double-layer loss optimization, and the multimodal sentiment analysis results are output.

[0102] Furthermore, in said S1, the network architecture of the multimodal sentiment analysis model includes a text pre-training model BERT and two Transformer encoding layers;

[0103] Audio feature extraction is performed on the audio data in the processed data through a Transformer encoding layer, that is, emotion-related features are extracted to obtain audio feature A. The dimension is transformed through a Dense layer (fully connected layer) for subsequent feature interaction. Each mode of audio feature A is decomposed into audio similarity feature A through a similar projection layer and a dissimilar projection layer. s Dissimilar feature A to audio d ;

[0104] Another Transformer encoding layer is used to extract video features from the video data in the processed data, that is, to extract emotion-related features to obtain video features V. The dimensions are transformed through the Dense layer for subsequent feature interaction. Each mode of the video feature V is decomposed into video similarity features V through the similarity projection layer and the dissimilar projection layer. s Dissimilar features from the video V d ;

[0105] The text feature extraction of text data is performed through the text pre-training model BERT to obtain the text feature T (sentence vector). The dimension is transformed through the Dense layer for subsequent feature interaction. Each mode of the text feature T is decomposed into text similarity feature T through the similarity projection layer and the dissimilar projection layer. s Dissimilar features to the text T d .

[0106] Furthermore, in S2, the video similarity feature V is obtained. s Dissimilar features from the video V d Finally, the six decomposed features are integrated through the concat function, namely the audio similarity feature A s , audio dissimilar feature A d , video similarity features V s , video dissimilar features V d , text similarity feature T sDissimilar features to the text T d , use MLP stacking multi-layer linear classifier to obtain multimodal analysis results, according to the multimodal sentiment prediction loss L m , complete the multimodal sentiment analysis task;

[0107] Multimodal sentiment prediction loss L m It is expressed as:

[0108]

[0109] in, is the result of multimodal analysis. is the multimodal sentiment label, and n is the total number of samples.

[0110] Furthermore, in S3, the text similarity feature T is obtained. s Dissimilar features to the text T d Finally, the decomposed features are integrated and the multimodal sentiment analysis model is optimized by contrastive learning. The specific steps of contrastive learning optimization are as follows:

[0111] The intra-sample comparison and inter-sample comparison are unified into the NT-Xent contrast loss framework, using the text similarity feature of the i-th information As an anchor point, establish a contrast relationship between decomposed features

[0112] Decomposing the contrast relationship between features It is expressed as:

[0113]

[0114] Among them, (a, p) and (a, k) are a pair of feature vectors decomposed in a sample, sim is the similarity function, exp is the exponential function, and τ is the temperature coefficient;

[0115] During the intra-sample and inter-sample comparisons, construct the positive example set P i and negative example set N i ;

[0116] Positive example set P i It is expressed as:

[0117]

[0118] in, is the positive pair within the sample, The samples are aligned;

[0119] Negative example set N i It is expressed as:

[0120]

[0121] in, is a negative pair in the sample, is a negative pair between samples;

[0122] Positive in sample It is expressed as:

[0123]

[0124] in, is the audio similarity feature of the i-th information, is the video similarity feature of the i-th information, is the audio similarity feature of the jth information, is the video similarity feature of the jth information, is the text similarity feature of the jth information, Neighbor i is the positive example set of the i-th information, Outlier i is the negative example set of the i-th information;

[0125] In-sample negative pair It is expressed as:

[0126]

[0127] in, is the audio dissimilarity feature of the i-th sample, is the video dissimilar feature of the i-th sample, is the text dissimilar feature of the i-th sample, is the audio dissimilarity feature of the jth sample, is the video dissimilar feature of the jth sample, is the text dissimilar feature of the jth sample;

[0128] Samples are aligned It is expressed as:

[0129]

[0130] Negative pair between samples It is expressed as:

[0131]

[0132] in, is the audio similarity feature of the kth information, V s k is the video similarity feature of the kth information, T s k is the text similarity feature of the k-th information;

[0133] After contrastive learning optimization, the loss function L is optimized according to contrastive learning cl , Complete contrastive learning optimization tasks;

[0134] Specifically, for the constructed positive and negative pairs within the sample, since text modal information is usually more reliable in the MSA (measurement system analysis) task, the text similarity feature T is selected. s i As an anchor point, make the video similar feature V s i Similar features to audio Closer to the text similarity feature T s i , all modality-dissimilar features are pushed away from the text-similar features T s i ,The selection of positive and negative example sets can increase the difficulty of the training task, so that the multimodal sentiment analysis model can be better trained;

[0135] Since the similar features of similar samples of the same class are close, while the similar features of samples of different classes are far, only similar features are used to obtain pairing between samples. The positive example set Neighbor of the i-th information i And the negative example set Outlier of the i-th information i The sampling algorithm is used to expand the contrast range and make full use of the advantages of contrastive learning;

[0136] refer to Figure 3 The sampling algorithm is as follows: Calculate the cosine similarity between the i-th information and all other information and sort them to obtain the sorted information, and then take out two information with the same label and the highest cosine similarity to form the positive example set Neighbor of the i-th information. i , from the sorted information, we take out the two pieces of information with different labels and the highest cosine similarity and the two pieces of information with the lowest cosine similarity. A total of four pieces of information constitute the negative example set Outlier of the i-th information i ;

[0137] The cosine similarity C between the i-th information and all other information ij It is expressed as:

[0138] C ij =sim([T i ; V i ; A i ],[T j ; V j ; A j ])

[0139] Contrastive learning focuses on learning the common features between similar instances and distinguishing the differences between non-similar instances. Compared with generative learning, contrastive learning does not need to pay attention to the tedious details of the instances, but only needs to learn to distinguish the data in the feature space at the abstract semantic level. Therefore, the model based on contrastive learning and its optimization become simpler and have stronger generalization ability. The goal of contrastive learning is to narrow the distance with positive samples and expand the distance with negative samples, so that the similarity between positive samples and anchor points is much greater than the similarity between negative samples and anchor points, so as to achieve the real distance of their original spatial distribution and obtain an enhanced relationship representation.

[0140] Furthermore, in S4, the audio similarity feature A is obtained s Dissimilar feature A to audio d After that, the decomposed features are integrated, and the multimodal sentiment analysis model is combined with the multimodal comprehensive sentiment label y given in the dataset through the label generation module. m , according to the relative distance α from the modal representation to the cluster center i Get the offset δ sm , generate unimodal sentiment label y s , based on the momentum update strategy, the final unimodal sentiment label value after i iterations is obtained After self-supervised learning, the single-modal analysis results are obtained by stacking multi-layer linear classifiers with weight-sharing MLP According to the unimodal sentiment prediction loss L u , complete the unimodal sentiment analysis task;

[0141] The specific offset calculation process is as follows: Get the positive center of different modal representations and the negative center of different modal representations, and use L2 regularization to represent a single mode The distance from the cluster center, i.e., unimodal Distance from the exact center of different modal representations and unimodal Distances from negative centers of different modal representations Since different modal representations exist in different feature spaces, the absolute distance value is not accurate enough. Therefore, a relative distance value ε is defined to evaluate the relative distance α from the modal representation to the class center. i ;

[0142] The exact center of different modal representations It is expressed as:

[0143]

[0144] in, is the global representation of the jth sample in modality i, i∈{m,t,a,v}, m represents a multimodal task, t represents text modality, a represents audio modality, v represents video modality, and y i are emotion labels of different modalities, I is the indicator function, and N is the number of training samples;

[0145] Negative center of different modal representations It is expressed as:

[0146]

[0147] The mode represents the relative distance α to the cluster center i It is expressed as:

[0148]

[0149] Among them, the relative distance value ε is a very small number, which is used to avoid the situation where the denominator is zero;

[0150] Unimodal sentiment label y s It is expressed as:

[0151]

[0152] Where s∈{t,a,v};

[0153] The final unimodal sentiment label value after i iterations It is expressed as:

[0154]

[0155] in, represents the unimodal sentiment label value generated by the label generation module after the i-th iteration, Represents the final unimodal sentiment label value after i-1 iterations;

[0156] Unimodal sentiment prediction loss L u It is expressed as:

[0157]

[0158] in, Decompose unimodal sentiment analysis results for each;

[0159] Specifically, based on the prior knowledge that the ratio of unimodal labels to multimodal labels is proportional to the corresponding relative distance ratio, each unimodal label value can be obtained according to the derivation of the formula, which can also be regarded as the multimodal label value plus a bias term;

[0160] Since the modal representation is dynamically changing, the generated unimodal sentiment labels are not stable enough. Therefore, a momentum-based update strategy is designed to solve the adverse effects of unimodal instability by combining historical values ​​with newly generated values.

[0161] For similar features, multimodal sentiment labels are selected for calculation to capture the consistent sentiment information shared between different modalities. For dissimilar features, unimodal sentiment labels are selected to retain the unique sentiment information specific to the unimodality. Since the multimodal sentiment labels of most multimodal datasets are manually annotated and do not contain unimodal sentiment labels, in order to obtain more accurate multimodal sentiment labels and unsupervised unimodal sentiment labels, a label generation module is added to the unimodal sentiment analysis task.

[0162] Furthermore, in S5, the auxiliary loss fine-tuning model is nonlinearly combined through hyperparameter optimization, which can be expressed as: setting the optimization parameter φ and the parameter ω of the backbone basic model, based on the multimodal sentiment prediction loss L m , unimodal sentiment prediction loss L u and contrastive learning loss L cl , double-layer loss optimization for multimodal sentiment analysis models;

[0163] The two-layer loss optimization process is expressed as:

[0164]

[0165] L aux =L u +L cl

[0166] Among them, L aux To assist in unifying the loss, ω is the upper-layer decision vector, and φ is the model parameter to be optimized in the lower layer.

[0167] Specifically, a variant of the approximate implicit differentiation (AID) scheme is used for optimization, that is, the super gradient L of the multimodal sentiment prediction loss is first decomposed m (ω * (φ)), through the implicit function theorem (IFT), the super gradient L of the multimodal sentiment prediction loss is m (ω * (φ)) is defined as the approximate gradient of the transformation

[0168] Approximate gradient of transformation The calculation of needs to calculate the inverse of the second-order derivative. In order to speed up the calculation, the Neumann series is usually used. In practice, the sum of infinite series is usually approximated as a finite series in exchange for a slight performance degradation. In the iterative calculation of the Hessian matrix, it takes a lot of time. It is further simplified by approximating the Hessian matrix to the identity matrix I to avoid the inversion of the Hessian matrix.

[0169] The Neumann series is expressed as:

[0170]

[0171] Approximate gradient of transformation The solution process is as follows:

[0172]

[0173] Optimize the Kth step of ω. After the Kth step, continue to optimize φ:

[0174] The process of optimizing ω in the Kth step is expressed as:

[0175]

[0176] Among them, α is the learning rate of the basic model;

[0177] The process of optimizing φ is expressed as:

[0178]

[0179] Among them, β is the learning rate of the optimized model.

[0180] Furthermore, in S6, the multi-task loss function L is expressed as:

[0181] L=L m +β u L u +β cl L cl

[0182] Among them, L m is the multimodal sentiment prediction loss, L u is the unimodal sentiment prediction loss, L cl is the contrastive learning loss, β u Learning rate for unimodal sentiment analysis task, β cl To compare the learning rate of the learning process;

[0183] Specifically, in this embodiment, the data sets to be processed are two multimodal sentiment analysis data sets, namely CMU-MOSEI (CMU Multimodal Opinion Sentiment and Emotion Intensity) and CMU-MOSI (CMU Multimodal Opinion-level Sentiment Intensity). The data sets to be processed are input into the multimodal sentiment analysis model to perform multi-task sentiment analysis to obtain multimodal sentiment analysis results. Refer to Table 3 for the multimodal sentiment analysis results based on CMU-MOSI and CMU-MOSEI, where Train represents the training set, Valid represents the validation set, Test represents the test set, and Total represents the total samples;

[0184] Table 3

[0185]

[0186] The specific settings for multi-task sentiment analysis are as follows: the analysis is performed on an NVIDIA A800 GPU, the text pre-training uses the BERT model for fine-tuning, and for the video modality and audio modality, the Transformer encoder is used to extract sentiment-related features for pre-training, and finally multi-modal sentiment classification is performed to obtain the multi-modal sentiment analysis results. For the CMU-MOSI dataset, the multi-modal sentiment analysis model is trained with a batch size of 25 epochs, and the overall learning rate of the model is set to 0.0001. For the CMU-MOSEI dataset, the multi-modal sentiment analysis model is trained with a batch size of 18 epochs, and the overall learning rate of the model is set to 0.0001.

[0187] refer to Figure 6 , red indicates text similarity feature T s , green indicates video similarity features V s , blue represents audio similarity feature A s , cyan indicates text dissimilar features T d , yellow indicates video dissimilar features V d , magenta indicates audio dissimilar features A d , similar features are becoming closer and closer, while different features are getting farther and farther away from their corresponding similar features, which proves the effectiveness of multimodal sentiment analysis model in learning consistency and difference between modalities;

[0188] refer to Figure 7,To demonstrate the interaction between different modalities of the model, ,the utterances with text and visual features from the CMU-MOSI dataset are visualized. ,It can be observed that the positive word “interesting” has a stronger correlation with ,visual features of upturned mouth corners, but a weaker correlation with ,visual features representing normal faces. The reason is that the model can enhance and ,complement textual information with different visual images through the different ,contributions to the model by unimodal and multimodal features.

[0189] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A multimodal sentiment analysis method based on contrastive decomposition feature multi-task network, characterized in that: The following steps are involved: S1. Build a multimodal sentiment analysis model, input the data set to be processed into the multimodal sentiment analysis model for feature extraction and decomposition, and obtain six decomposition features; S2. Integrate the six decomposed features, use MLP to stack multi-layer linear classifiers to obtain multimodal analysis results, and complete the multimodal sentiment analysis task based on the multimodal sentiment prediction loss obtained; S3. Integrate the six decomposition features, perform contrastive learning optimization on the multimodal sentiment analysis model, and complete the contrastive learning optimization task based on the contrastive learning loss obtained; S4. Integrate the six decomposed features, and through self-supervised learning, stack multi-layer linear classifiers with weight-sharing MLP to obtain the unimodal analysis results. According to the solved unimodal sentiment prediction loss, complete the unimodal sentiment analysis task; S5. Based on the multimodal sentiment prediction loss, the unimodal sentiment prediction loss and the contrastive learning loss, the multimodal sentiment analysis model is optimized by double-layer loss to obtain the multimodal sentiment analysis model after double-layer loss optimization; S6. Based on the multimodal sentiment prediction loss, unimodal sentiment prediction loss and contrastive learning loss, a multi-task loss function is constructed to train the multimodal sentiment analysis model after double-layer loss optimization, and the multimodal sentiment analysis results are output.

2. The multimodal sentiment analysis method based on contrast decomposition feature multi-task network according to claim 1 is characterized in that: In S1, the network architecture of the multimodal sentiment analysis model includes a text pre-training model BERT and two Transformer encoding layers; The audio features of the audio data in the processing data set are extracted through a Transformer encoding layer to obtain the audio feature A. The dimension is transformed through the Dense layer for subsequent feature interaction. Each mode of the audio feature A is decomposed into audio similarity feature A through the similarity projection layer and the dissimilar projection layer. s Dissimilar feature A to audio d ; Through another Transformer encoding layer, the video features in the video data to be processed are extracted to obtain the video features V. The dimensions are transformed through the Dense layer for subsequent feature interaction. Each mode of the video features V is decomposed into video similarity features V through the similarity projection layer and the dissimilar projection layer. s Dissimilar features from the video V d ; The text feature T is obtained by extracting text features from the text data through the text pre-training model BERT. The dimension is transformed through the Dense layer for subsequent feature interaction. Each mode of the text feature T is decomposed into text similarity feature T through the similarity projection layer and the dissimilar projection layer. s Dissimilar features to the text T d .

3. The multimodal sentiment analysis method based on contrast decomposition feature multi-task network according to claim 2 is characterized in that: In S2, the video similarity feature V is obtained. s Dissimilar features from the video V d Finally, the six decomposed features are integrated through the concat function, namely the audio similarity feature A s , audio dissimilar feature A d , video similarity features V s , video dissimilar features V d , text similarity feature T s Dissimilar features to the text T d , use MLP stacking multi-layer linear classifier to obtain multimodal analysis results, according to the multimodal sentiment prediction loss L m , complete the multimodal sentiment analysis task; Multimodal sentiment prediction loss L m It is expressed as: in, is the result of multimodal analysis. is the multimodal sentiment label, and n is the total number of samples.

4. The multimodal sentiment analysis method based on contrast decomposition feature multi-task network according to claim 3 is characterized in that: In S3, the text similarity feature T is obtained s Dissimilar features to the text T d Finally, the decomposed features are integrated to perform comparative learning optimization on the multimodal sentiment analysis model; The specific steps of contrastive learning optimization are as follows: The intra-sample comparison and inter-sample comparison are unified into the NT-Xent contrast loss framework, using the text similarity feature of the i-th information As an anchor point, establish a contrast relationship between decomposed features Decomposing the contrast relationship between features It is expressed as: Among them, (a, p) and (a, k) are a pair of feature vectors decomposed in a sample, sim is the similarity function, exp is the exponential function, and τ is the temperature coefficient; During the intra-sample and inter-sample comparisons, construct the positive example set P i and negative example set N i ; Positive example set P i It is expressed as: in, is the positive pair within the sample, The samples are aligned; Negative example set N i It is expressed as: in, is a negative pair in the sample, is a negative pair between samples; Positive in sample It is expressed as: in, is the audio similarity feature of the i-th information, is the video similarity feature of the i-th information, is the audio similarity feature of the jth information, is the video similarity feature of the jth information, is the text similarity feature of the jth information, Neighbor i is the positive example set of the i-th information, Outlier i is the negative example set of the i-th information; In-sample negative pair It is expressed as: in, is the audio dissimilarity feature of the i-th sample, is the video dissimilar feature of the i-th sample, is the text dissimilar feature of the i-th sample, is the audio dissimilarity feature of the jth sample, is the video dissimilar feature of the jth sample, is the text dissimilar feature of the jth sample; Positive alignment between samples It is expressed as: Negative pair between samples It is expressed as: in, is the audio similarity feature of the k-th information, is the video similarity feature of the k-th information, is the text similarity feature of the k-th information; After contrastive learning optimization, the loss function L is optimized according to contrastive learning cl , Complete the contrastive learning optimization task.

5. The multimodal sentiment analysis method based on contrast decomposition feature multi-task network according to claim 4 is characterized in that: In S4, the audio similarity feature A is obtained. s Dissimilar feature A to audio d After that, the decomposed features are integrated, and the multimodal sentiment analysis model is combined with the multimodal comprehensive sentiment label y given in the dataset to be processed through the label generation module. m , according to the relative distance α from the modal representation to the cluster center i Get the offset δ sm , generate unimodal sentiment label y s , based on the momentum update strategy, the final unimodal sentiment label value after i iterations is obtained After self-supervised learning, the single-modal analysis results are obtained by stacking multi-layer linear classifiers with weight-sharing MLP According to the unimodal sentiment prediction loss L u , complete the unimodal sentiment analysis task; The specific offset calculation process is as follows: Get the positive center of different modal representations and the negative center of different modal representations, and use L2 regularization to represent a single modality The distance from the cluster center, i.e., unimodal Distance from the exact center of different modal representations and unimodal Distances from negative centers of different modal representations Define the relative distance value ε and evaluate the relative distance α from the modal representation to the class center i ; The exact center of different modal representations It is expressed as: in, is the global representation of the jth sample in modality i, i∈{m,t,a,v}, m represents a multimodal task, t represents text modality, a represents audio modality, v represents video modality, and y i are emotion labels of different modalities, I is the indicator function, and N is the number of training samples; Negative center of different modal representations It is expressed as: The mode represents the relative distance α to the cluster center i It is expressed as: Unimodal sentiment label y s It is expressed as: Where s∈{t,a,v}; The final unimodal sentiment label value after i iterations It is expressed as: in, represents the unimodal sentiment label value generated by the label generation module after the i-th iteration, Represents the final unimodal sentiment label value after i-1 iterations; Unimodal sentiment prediction loss L u It is expressed as: in, Decompose the unimodal sentiment analysis results for each.

6. The multimodal sentiment analysis method based on contrast decomposition feature multi-task network according to claim 5 is characterized in that: In S5, the optimization parameter φ and the parameter ω of the backbone basic model are set, based on the multimodal sentiment prediction loss L m , unimodal sentiment prediction loss L u and contrastive learning loss L cl , double-layer loss optimization for multimodal sentiment analysis models; The two-layer loss optimization process is expressed as: L aux =L u +L cl Among them, L aux To assist in unifying the loss, ω * is the upper layer decision vector, φ * are the model parameters to be optimized in the lower layer.

7. The multimodal sentiment analysis method based on contrast decomposition feature multi-task network according to claim 6 is characterized in that: In S6, the multi-task loss function L is expressed as: L=L m +b u L u +b cl L cl Among them, L m is the multimodal sentiment prediction loss, L u is the unimodal sentiment prediction loss, L cl is the contrastive learning loss, β u The learning rate for the unimodal sentiment analysis task, β cl is the learning rate of the comparative learning process.

Images