A multimodal sentiment analysis method improved by multi-task learning and contrastive learning

Abstract

The present invention discloses a multimodal sentiment analysis method improved by multi-task learning and contrastive learning. Aiming at the multimodal sentiment analysis task, the present invention proposes a method through multi-task learning and contrastive learning to comprehensively assist in learning single-modal representation, bimodal representation and multimodal fusion representation, which is helpful for the final multimodal sentiment polarity prediction. Aiming at the problem of adjusting the weight coefficients of multiple tasks in the multi-task learning loss function, the present invention proposes a method for adaptively adjusting the weight coefficients using homoscedastic uncertainty to solve the problems of time-consuming manual adjustment and low adjustment accuracy, and weighs the importance of each task through model adaptive learning weight parameters.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically relates to a multimodal sentiment analysis method improved by multi-task learning and contrastive learning. Background Art

[0002] With the rapid development of information and communication technologies, more and more users are expressing their opinions and emotions on social platforms. This content carries a wealth of personal emotions and opinions. Analyzing and identifying the underlying emotional information in social data has always been an important foundational task in the fields of sentiment computing and natural language processing. With the development of communication technologies, the content format of traditional social data is also constantly changing. The content posted by users is no longer just text information, but multimodal data containing different modalities such as text, images, and voice. Multimodal Sentiment Analysis (MSA) research plays an important role in fields such as social media public opinion analysis and recommendation systems. Therefore, this research has important research significance and application value.

[0003] While multimodal data analysis can provide richer feature information, its heterogeneity and heterogeneity make it more challenging than traditional unimodal sentiment analysis. When extracting initial representations from raw data, different modalities require different methods to transform the raw data information and extract their characteristic representations. Later, during feature fusion, the spatial distribution of data from different modalities differs, so simple splicing is ineffective. More rational feature fusion methods are necessary.

[0004] Multimodal sentiment analysis not only needs to process each unimodal representation separately, but also needs to consider the fusion between different modal representations. Zadeh et al. proposed the Tensor Fusion Network (TFN), which uses the Cartesian outer product of different modal feature representations as the fusion result. This method can simultaneously capture the interaction between unimodality and multimodality. Hazarika et al. proposed modality-invariant representations and modality-specific representations for multimodal representation learning. Tsai et al. proposed a multimodal Transformer that can capture interactions with other modalities and learn cross-modal attention to strengthen the target modality.

[0005] Although there are already quite a number of excellent deep learning models and methods in the field of multimodal sentiment analysis, past research has mostly focused on the fusion of multimodal features while ignoring the learning of unimodal representations. Learning modal representations is an early part of the entire task and affects the quality of the actual prediction results in the later stage. Therefore, representation learning plays an important role in the entire multimodal sentiment analysis task. Summary of the Invention

[0006] Aiming at the early representation learning, this paper proposes a multimodal sentiment analysis method improved by multi-task learning and contrastive learning.

[0007] The present invention utilizes the ideas of multi-task learning and contrastive learning to assist in learning unimodal and bimodal representations in stages, and uses contrastive learning to assist in learning multimodal fusion representations. Based on homoscedastic uncertainty, a method for adaptively adjusting the weight coefficients of the multi-task loss function is proposed to improve the accuracy of multimodal sentiment polarity classification.

[0008] The present invention specifically comprises the following steps:

[0009] Step 1: Prepare the MOSI dataset, process the multimodal data in the dataset, and obtain the initial unimodal representation;

[0010] Step 2: Input the unimodal representation into the first subtask module, namely the unimodal representation prediction module, to perform the unimodal sentiment polarity prediction subtask and obtain the unimodal sentiment polarity prediction result;

[0011] Step 3: Input the unimodal representation into the bimodal gating module to generate a bimodal representation, and then input the bimodal representation into the second subtask module, namely the bimodal representation prediction module, to perform the bimodal sentiment polarity prediction subtask and obtain the bimodal sentiment polarity prediction result;

[0012] Step 4: Concatenate the unimodal and bimodal representations to perform the main task of sentiment polarity prediction, and use the mean squared error between any two outputs as the contrastive learning loss through contrastive learning.

[0013] Step 5: The loss of the main task and the two groups of subtasks is used as the multi-task learning loss. The loss weight coefficients of different tasks in the multi-task learning loss are adaptively adjusted through homoscedastic uncertainty. The weighted sum of the multi-task learning loss and the contrastive learning loss is used as the final loss function of the model.

[0014] As an example, the multimodal data in the data set is processed to obtain the initial unimodal representation, specifically as follows: Step 1-1: convert the text vector f in the multimodal data into t Perform initial processing through the BERT pre-training model to obtain the text representation F t , the text modality features are expressed as follows:

[0015] F t =BERT(f t θ t )

[0016] where θ t Represents learnable parameters, the same below;

[0017] Step 1-2: Transform the audio vector f in the multimodal dataa Perform initial processing through bidirectional LSTM to obtain audio representation F a , the audio modal features are expressed as follows:

[0018] F a =LSTM(f a θ a )

[0019] Step 1-3: Transform the video vector f in the multimodal data v Perform initial processing through bidirectional LSTM to obtain video representation F v , the video modality features are expressed as follows:

[0020] F v =LSTM(f v θ v )

[0021] Preferably, the unimodal representation is input into the first subtask module to perform the unimodal sentiment polarity prediction subtask.

[0022] Step 2-1: Represent the single modal representation F obtained in step 1 s , s∈{t, a, v} are input into three independent single-modal prediction modules, first mapping them into a low-dimensional space In , the process is expressed as follows:

[0023]

[0024] in, ReLU is the activation function, ds represents the dimension of the feature, Indicates bias.

[0025] Step 2-2: F′ s Send it to the linear layer to get the prediction result. The process is as follows:

[0026]

[0027] in, y s is the single-modal sentiment polarity prediction result, Indicates bias.

[0028] Preferably, the unimodal representation is input into the bimodal gating module to generate a bimodal representation, and the bimodal representation is input into the second subtask module, i.e., the bimodal representation prediction module, to perform the bimodal emotion polarity prediction subtask; specifically, step 3-1: the unimodal representation F obtained in step 1 is input into the bimodal gating module to generate a bimodal representation. s, s∈{t, a, v} is input into the bimodal gating module, which learns the interaction information between the two modalities and generates a bimodal vector. For two single-modal inputs {F α , F β}, where (α, β)∈{(t, a), (t, v), (v, a)}, and the bimodal representation h αβ The calculation process is as follows:

[0029]

[0030]

[0031]

[0032] h αβ =g αβ h α +(1-g αβ )h β

[0033] Among them, tanh and ReLU are activation functions, h α 、h β represents the intermediate representation, g αβ is the gate value, Indicates bias;

[0034] Step 3-2: The bimodal representation h obtained in step 3-1 is αβ Input into three independent bimodal prediction modules respectively, first mapping them into a low-dimensional space In , the process is expressed as follows:

[0035]

[0036] in, Indicates bias;

[0037] Step 3-3: F αβ Send it to the linear layer to get the prediction result. The process is as follows:

[0038]

[0039] in, y αβ is the bimodal sentiment polarity prediction result, Indicates bias.

[0040] As a preference, the single-modal and bimodal representations are spliced ​​together to perform the main task of sentiment polarity prediction, and through contrastive learning, the mean square error of any two outputs is used as the contrastive learning loss; specifically: Step 4-1: The single-modal representation F obtained in step 1 iss The bimodal representation h obtained in step 3 αβ Splicing to obtain multimodal fusion representation F m , input multimodal sentiment polarity prediction module, first map it to low-dimensional space In , the process is expressed as follows:

[0041]

[0042] in, Indicates bias;

[0043] Step 4-2: F m ′ is sent to the linear layer to obtain the prediction result. The process is expressed as follows:

[0044]

[0045] in, y m is the multimodal sentiment polarity prediction result, Indicates bias.

[0046] Step 4-3: Using the randomness of Dropout, the multimodal fusion representation F m Input the multimodal sentiment polarity prediction module twice, and use the mean square error of the two outputs as the loss function of the contrastive learning part. The process is expressed as follows:

[0047]

[0048] Where N is the size of the dataset, and It is the multimodal prediction result of two outputs.

[0049] As a preference, the loss of the main task and the two groups of subtasks is used as the multi-task learning loss. The loss weight coefficients of different tasks in the multi-task learning loss are adaptively adjusted through homoscedastic uncertainty. The weighted sum of the multi-task learning loss and the contrastive learning loss is used as the final loss function of the model. Specifically: Step 5-1: Adaptively adjust the weight coefficients of the main task and the two groups of subtasks in the loss function according to homoscedastic uncertainty. The process is expressed as follows:

[0050]

[0051] Among them, k∈{m, t, a, v, ta, tv, va}, σ i is an adaptively adjustable parameter, y ij Represents the subtask prediction result, represents the true value, m represents the multimodal main task, t, a, v represent the unimodal subtask, ta, tv, va represent the bimodal subtask.

[0052] Step 5-2: Add the multi-task learning loss and contrastive learning loss as the final loss function of the model. The process is as follows:

[0053]

[0054] Among them, α is a custom hyperparameter, y ij represents the jth sample of the i-th subtask, Represents the tag value, and It is the multimodal prediction result of two outputs.

[0055] The beneficial effects of the present invention are:

[0056] 1. Aiming at the multimodal sentiment analysis task, the present invention proposes a method through multi-task learning and comparative learning to comprehensively assist in learning single-modal representation, bimodal representation and multimodal fusion representation, which is helpful for the final multimodal sentiment polarity prediction.

[0057] 2. Aiming at the problem of adjusting the weight coefficients of multiple tasks in the multi-task learning loss function, a method of adaptively adjusting the weight coefficients using homoscedastic uncertainty is proposed to solve the problem of time-consuming manual adjustment and low adjustment accuracy. The importance of each task is weighed by adaptively learning the weight parameters through the model.

[0058] 3. Experiments were conducted on the CMU-MOSI dataset. The results showed that the present invention significantly improved the performance of multimodal sentiment analysis. Compared with the popular multimodal sentiment analysis methods in recent years, the accuracy and F1 value of the present invention were improved. Compared with MAG-BERT, the best performing baseline model, the two classification accuracies increased by 0.82 and 1.07, respectively, and the F1 values ​​increased by 0.64 and 1.05, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a structural diagram of the model proposed in the present invention. DETAILED DESCRIPTION

[0060] like Figure 1 As shown in Figure 1, a multi-modal sentiment analysis method improved by multi-task learning and contrastive learning is presented. The specific implementation steps are as follows:

[0061] Step 1: Download the CMU-MOSI dataset. The MOSI dataset consists of 2,199 short monologue video clips. Each video clip is manually annotated with a sentiment score. The sentiment score ranges from -3 to +3, with smaller values ​​indicating more negative sentiment and larger values ​​indicating more positive sentiment. The MOSI dataset has 1,284 clips in the training set, 229 clips in the validation set, and 686 clips in the test set. Process the multimodal data in the dataset, extracting video features using Facet and audio features using COVAREP.

[0062] Step 1-1: Transform the text vector f t Perform initial processing through the BERT pre-training model to obtain the text representation F t , the text modality features are expressed as follows:

[0063] F t =BERr(f t θ r )

[0064] where θ t Represents learnable parameters, the same below;

[0065] Step 1-2: Convert the audio vector f a Perform initial processing through bidirectional LSTM to obtain audio representation F a , the audio modal features are expressed as follows:

[0066] F a =LSTM(f a θ a )

[0067] Step 1-3: Transform the video vector f v Perform initial processing through bidirectional LSTM to obtain video representation F v , the video modality features are expressed as follows:

[0068] F v =LSTM(f v θ v )

[0069] Step 2: Input the unimodal representation into the first subtask module, the unimodal representation prediction module, to perform the unimodal sentiment polarity prediction subtask;

[0070] Step 2-1: Represent the single modal representation F obtained in step 1 s , s∈{t, a, v} are input into three independent single-modal prediction modules, first mapping them into a low-dimensional space In , the process is expressed as follows:

[0071]

[0072] in, ReLU is the activation function.

[0073] Step 2-2: F s ′ is sent to the linear layer to obtain the prediction result. The process is expressed as follows:

[0074]

[0075] in, y s This is the unimodal sentiment polarity prediction result.

[0076] Step 3: Input the unimodal representation into the bimodal gating module to generate a bimodal representation, and then input the bimodal representation into the second subtask module, the bimodal representation prediction module, to perform the bimodal sentiment polarity prediction subtask;

[0077] Step 3-1: Represent the single modal representation F obtained in step 1 s , s∈{t, a, v} is input into the bimodal gating module, which can learn the interaction information between the two modalities and generate a bimodal vector. α , F β}, where (α, β)∈{(t, a), (t, v), (v, a)}, the computational process of the bimodal representation is expressed as follows:

[0078]

[0079]

[0080]

[0081] h αp =g αβ h α +(1-g αβ )h β

[0082] Among them, tanh and ReLU are activation functions,

[0083] Step 3-2: The bimodal representation h obtained in step 3-1 is αβ Input into three independent bimodal prediction modules respectively, first mapping them into a low-dimensional space In , the process is expressed as follows:

[0084]

[0085] in,

[0086] Step 3-3: F αβ Send it to the linear layer to get the prediction result. The process is as follows:

[0087]

[0088] in, y αβ The results of bimodal sentiment polarity prediction are shown.

[0089] Step 4: Concatenate the unimodal and bimodal representations to perform the main task of sentiment polarity prediction. Through contrastive learning, the mean squared error of the two outputs is used as the contrastive learning loss.

[0090] Step 4-1: Represent the single modal representation F obtained in step 1 s The bimodal representation h obtained in step 3 αβ Splicing to obtain multimodal fusion representation F m , input multimodal sentiment polarity prediction module, first map it to low-dimensional space In , the process is expressed as follows:

[0091]

[0092] in,

[0093] Step 4-2: F m ′ is sent to the linear layer to obtain the prediction result. The process is expressed as follows:

[0094]

[0095] in, y m Multimodal sentiment polarity prediction results.

[0096] Step 4-3: Using the randomness of Dropout, the multimodal fusion representation F m Input the multimodal sentiment polarity prediction module twice, and use the mean square error of the two outputs as the loss function of the contrastive learning part. The process is expressed as follows:

[0097]

[0098] Where N is the size of the dataset, and It is the multimodal prediction result of two outputs.

[0099] Step 5: The loss of the main task and the two groups of subtasks is used as the multi-task learning loss. The loss weight coefficients between different tasks in the multi-task learning loss are adaptively adjusted through homoscedastic uncertainty. The weighted sum of the multi-task learning loss and the contrastive learning loss is used as the final loss function of the model.

[0100] Step 5-1: Adaptively adjust the weight coefficients of the main task and the two groups of subtasks in the loss function based on the homoscedastic uncertainty. The process is as follows:

[0101]

[0102] Among them, k∈{m, t, a, v, ta, tv, va}, σ i is an adaptively adjustable parameter, y ij Represents the subtask prediction result, Represents a true value.

[0103] Step 5-2: Add the multi-task learning loss and contrastive learning loss as the final loss function of the model. The process is as follows:

[0104]

[0105] Among them, α is a custom hyperparameter, and α = 1.0 is set in our experiment.

[0106] Based on the CMU-MOSI dataset, the present invention is compared with some previous methods in the field. The experimental results show that the multimodal sentiment analysis method proposed in the present invention has improved performance compared with traditional methods.

[0107] Four metrics are used to evaluate the model's sentiment analysis performance: 1) Acc_2 binary classification accuracy; 2) F1 score; 3) Mean Absolute Error (MAE); and 4) Correlation Coefficient (Corr). Except for MAE, larger values ​​for the other metrics indicate better performance.

[0108] To fully validate the performance of the proposed model, we selected several mainstream models for multimodal sentiment analysis. The experimental results are shown in Table 1. Compared to the top-performing baseline model, MAG-BERT, the binary classification accuracy (Acc-2) increased by 1.07, the F1 value increased by 1.05, the mean absolute error (MAE) decreased by 0.004, and the correlation coefficient increased by 0.005. This model outperformed the comparison models on all four evaluation metrics, demonstrating that the model can learn modal representations with richer sentiment information.

[0109] Table 1 Experimental results of each model on the MOSI test set

[0110]

[0111] To more intuitively analyze the reliability of adaptive weight adjustment, we analyze the reliability of adaptive weight adjustment by outputting the weight correlation v in 5 experiments. As shown in Table 2, the relationship between v and the variable σ in the weight coefficient is:

[0112] σ 2 =e v

[0113] Set the weight coefficient in the multi-task learning loss function to That is, the smaller v is, the higher the weight coefficient of that task in the loss function. During the experiment, we observed that the prediction accuracy of the four tasks M, T, TV, and TA was the highest, so the weight coefficients of these four tasks should also be higher. Analysis of the data in Table 2 shows that the adaptively adjusted weight coefficients of M, T, TV, and TA are the highest, which is consistent with the above analysis and proves the reliability of adaptive weight adjustment.

[0114] Table 2 Multi-task learning adaptive adjustment weight coefficient related quantity v

[0115]

[0116]

Claims

1. A multi-modal sentiment analysis method improved by multi-task learning and contrastive learning, characterized in that: The method specifically comprises the following steps: Step 1: Prepare the MOSI dataset, process the multimodal data in the dataset, and obtain the initial unimodal representation; Step 2: Input the unimodal representation into the first subtask module, namely the unimodal representation prediction module, to perform the unimodal sentiment polarity prediction subtask and obtain the unimodal sentiment polarity prediction result; Step 3: Input the unimodal representation into the bimodal gating module to generate a bimodal representation, and then input the bimodal representation into the second subtask module, namely the bimodal representation prediction module, to perform the bimodal sentiment polarity prediction subtask and obtain the bimodal sentiment polarity prediction result; Step 4: Concatenate the unimodal and bimodal representations to perform the main task of sentiment polarity prediction, and use the mean squared error between any two outputs as the contrastive learning loss through contrastive learning. Step 5: The loss of the main task and the two groups of subtasks is used as the multi-task learning loss. The loss weight coefficients of different tasks in the multi-task learning loss are adaptively adjusted through homoscedastic uncertainty. The weighted sum of the multi-task learning loss and the contrastive learning loss is used as the final loss function of the model.

2. The multi-modal sentiment analysis method improved by multi-task learning and contrastive learning according to claim 1, characterized in that: The multimodal data in the data set are processed to obtain the initial single-modal representation, specifically: Step 1-1: convert the text vector f in the multimodal data into t Perform initial processing through the BERT pre-training model to obtain the text representation F t , the text modality features are expressed as follows: F t =BERT(f t ;θ t ) where θ t Represents learnable parameters, the same below; Step 1-2: Transform the audio vector f in the multimodal data a Perform initial processing through bidirectional LSTM to obtain audio representation F a , the audio modal features are expressed as follows: F a =LSTM(f a ;θ a ) Step 1-3: Transform the video vector f in the multimodal data v Perform initial processing through bidirectional LSTM to obtain video representation F v , the video modality features are expressed as follows: F v =LSTM(f v ;θ v )。 3. The multi-task learning and contrastive learning improved multimodal sentiment analysis method according to claim 1, characterized in that: The unimodal representation is input into the first subtask module to perform the unimodal sentiment polarity prediction subtask. Step 2-1: Represent the single modal representation F obtained in step 1 s , s∈{t, a, v} are input into three independent single-modal prediction modules, first mapping them into a low-dimensional space In , the process is expressed as follows: in, ReLU is the activation function, ds represents the dimension of the feature, Indicates bias; Step 2-2: F′ s Send it to the linear layer to get the prediction result. The process is as follows: in, y s is the unimodal sentiment polarity prediction result, Indicates bias.

4. The multi-modal sentiment analysis method improved by multi-task learning and contrastive learning according to claim 1, characterized in that: The unimodal representation is input into the bimodal gating module to generate a bimodal representation, and the bimodal representation is input into the second subtask module, namely the bimodal representation prediction module, to perform the bimodal sentiment polarity prediction subtask; specifically: Step 3-1: The unimodal representation F obtained in step 1 is input into the bimodal representation prediction module. s , s∈{t, a, v} is input into the bimodal gating module, which learns the interaction information between the two modalities and generates a bimodal vector; for two single-modal inputs {F α , F β }, where (α, β)∈{(t, a), (t, v), (v, a)}, and the bimodal representation h αβ The calculation process is as follows: h αβ =g αβ h α +(1-g αβ )h β Among them, tanh and ReLU are activation functions, h α 、h β represents the intermediate representation, g αβ is the gate value, Indicates bias; Step 3-2: The bimodal representation h obtained in step 3-1 is αβ Input into three independent bimodal prediction modules respectively, first mapping them into a low-dimensional space In , the process is expressed as follows: in, Indicates bias; Step 3-3: F αβ Send it to the linear layer to get the prediction result. The process is as follows: in, y αβ is the bimodal sentiment polarity prediction result, Indicates bias.

5. The multi-modal sentiment analysis method improved by multi-task learning and contrastive learning according to claim 1, characterized in that: The unimodal and bimodal representations are spliced ​​together to perform the main task of sentiment polarity prediction, and through contrastive learning, the mean square error of any two outputs is used as the contrastive learning loss; specifically: Step 4-1: The unimodal representation F obtained in step 1 is s The bimodal representation h obtained in step 3 αβ Splicing to obtain multimodal fusion representation F m , input multimodal sentiment polarity prediction module, first map it to low-dimensional space In , the process is expressed as follows: in, Indicates bias; Step 4-2: F m ′ is sent to the linear layer to obtain the prediction result. The process is expressed as follows: in, y m is the multimodal sentiment polarity prediction result, Indicates bias; Step 4-3: Using the randomness of Dropout, the multimodal fusion representation F m Input the multimodal sentiment polarity prediction module twice, and use the mean square error of the two outputs as the loss function of the contrastive learning part. The process is expressed as follows: Where N is the size of the dataset, and It is the multimodal prediction result of two outputs.

6. The multi-modal sentiment analysis method improved by multi-task learning and contrastive learning according to claim 1, characterized in that: The loss of the main task and the two groups of subtasks is used as the multi-task learning loss. The loss weight coefficients of different tasks in the multi-task learning loss are adaptively adjusted through homoscedastic uncertainty. The weighted sum of the multi-task learning loss and the contrastive learning loss is used as the final loss function of the model. Specifically: Step 5-1: Adaptively adjust the weight coefficients of the main task and the two groups of subtasks in the loss function according to homoscedastic uncertainty. The process is expressed as follows: Among them, k∈{m, t, a, v, ta, tv, va}, σ i is an adaptively adjustable parameter, y ij Represents the subtask prediction result, represents the true value, m represents the multimodal main task, t, a, v represent the unimodal subtask, ta, tv, va represent the bimodal subtask; Step 5-2: Add the multi-task learning loss and contrastive learning loss as the final loss function of the model. The process is as follows: Among them, α is a custom hyperparameter, y ij represents the jth sample of the i-th subtask, Represents the label value, and It is the multimodal prediction result of two outputs.

Images