Multimodal sentiment analysis method based on multi-task learning and cascade cross-modal fusion
Through multi-task learning and cascaded cross-modal fusion methods, combined with single-modal feature extraction and cross-modal feature fusion network, the problem of difficult heterogeneity in modal feature fusion in the existing technology is solved, and the problem of heterogeneity in complex scenarios is realized multimodal sentiment analysis under high accuracy.
Patent Information
- Application Number
- CN202210364764.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-08
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-04-08
AI Technical Summary
The existing multimodal emotion analysis method lacks effective modeling of features within the modal when fusing different modal features, making it difficult to retain the heterogeneity of single modal features, and the use of a ternary symmetric architecture does not conform to the Chinese text of emotional communication. , the different contributions of audio and visual information lead to a decrease in the accuracy of emotional classification in complex scenarios.
Using a method based on multi-task learning and cascaded cross-modal fusion, text, audio and visual features are input into the single-modal feature extraction module and the cascaded cross-modal feature fusion module respectively, and the multi-layer perceptron and gated cross-modal Transformer The network performs feature fusion, dynamically adjusts the loss weight to preserve modal heterogeneity, and realizes emotional classification.
Effectively identifying multimodal emotions in complex scenarios improves the accuracy of emotional classification, especially in the presence of complex emotional expressions such as ambiguity and irony.
Smart Images

Figure CN114694076B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of natural language processing, and specifically relates to a multimodal sentiment analysis method based on multi-task learning and stacked cross-modal fusion. The method sequentially inputs features of different modalities into multiple stacked gated cross-modal trasnformer networks for fusion, and uses multi-task learning to guide the model to preserve feature heterogeneity. The method has good robustness in the presence of complex emotional expressions such as ambiguity and irony. Background Art
[0002] Multimodal sentiment analysis is an emerging deep learning technology that is not only used for emotion polarity recognition in videos, but also the basis for downstream tasks such as user behavior analysis and dialogue generation. As an extension of text sentiment analysis, its data source includes not only text modality, but also audio and visual modality. Different modalities in the same data segment often complement each other, providing additional clues for semantic and sentiment disambiguation. How to simulate the process of human acceptance of multimodal information, effectively model the intra-modal features, so that the heterogeneity of the modalities can be retained and noise can be filtered out as much as possible; when modeling between modalities, it can effectively integrate heterogeneous data and extract and integrate meaningful information. These are the two major challenges of multimodal sentiment analysis.
[0003] In recent years, multimodal sentiment analysis methods are mainly divided into: (1) Methods based on time series fusion, such as recurrent embedding network (REVEN), memory fusion network (MFN), recurrent memory fusion network (RMFN), MAG-BERT, etc.; such methods need to align the features of different modalities first, and then fuse the features of each time step in sequence. Although the fine emotion of each time step can be detected, the connection between the different time series features of different modalities is severed. (2) Non-time series fusion methods, such as tensor fusion network (TFN), cross-modal Transformer, low-rank tensor fusion network (LMF) and Bi-ATT, etc. Such methods often do not need to align the data, and fuse the data of different modalities as a whole.
[0004] The above method has two obvious problems: (1) It adopts a ternary symmetric architecture, models one pair of the three modalities in turn, and treats each modality equally in terms of its contribution to the final emotion. But in fact, in the process of emotional communication, the amount of information carried by text, audio, and visual information is different. Text is the main carrier of multimodal emotions, while audio and visual information play an auxiliary role. (2) It lacks effective modeling of intra-modal features, and it is difficult to retain the heterogeneity of unimodal features while filtering out noise.
[0005] Therefore, it is necessary to find a method that can retain the heterogeneity of unimodal features and use an asymmetric architecture to fuse different modalities to ensure the performance of multimodal sentiment analysis algorithms. Summary of the invention
[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned background technology and propose a multimodal sentiment analysis method based on multi-task learning and cascaded cross-modal fusion. The method should be able to coherently integrate the features of different modalities while retaining the heterogeneity of the modalities, so as to effectively identify multimodal emotions in complex scenarios.
[0007] The technical solution provided by the present invention is:
[0008] A multimodal sentiment analysis method based on multi-task learning and cascade cross-modal fusion, the specific execution steps are as follows:
[0009] Step 1: Split the original video into video clips, mark the corresponding emotional labels (positive, neutral, negative) of the video clips, and extract text, audio and visual features that can be recognized by the model from the video;
[0010] Step 2: Use a single-modal feature extraction module and a stacked cross-modal feature fusion module
[0011] The text, audio and visual features extracted in step 1) are first input into a unimodal feature extraction module, which contains three long short-term memory networks (LSTM) to perform intra-modal interaction to obtain unimodal hidden layer features with contextual semantic information. These unimodal hidden layer features are then input into a stacked cross-modal feature fusion module for feature fusion and extraction to obtain multimodal high-level features.
[0012] Step 3: After concatenating the multimodal high-level features obtained in step 2 with the unimodal high-level features, the final sentiment classification result is output through a multi-layer perceptron (MLP). After calculating the multi-task loss value, the loss weights of different tasks (text sentiment prediction, audio sentiment prediction, visual sentiment prediction, and overall sentiment prediction) are dynamically adjusted according to the algorithm to iteratively optimize the model.
[0013] Furthermore, step 1 includes the following sub-steps:
[0014] Step 1a: According to the speaker's speech interval in the original video, the video is cut into 5-10 second video clips, each of which contains a complete sentence, and the speech is multimodally (text, audio, visual, overall) emotionally annotated.
[0015] Step 1b: Transcribe the subtitles in the video clip into text and use the BERT word embedding model to convert the text into word vectors X T ∈R length×dim, and set the maximum sentence length to 39 and the word embedding dimension to 762;
[0016] Step 1c: For audio clips, use the LibROSA audio toolkit to extract 33-dimensional frame-level audio features, including one-dimensional logarithmic fundamental frequency (log F0), 20-dimensional Mel-frequency cepstral coefficients (MFCCs), and 12-dimensional constant q spectrum (CQT), all of which are related to emotions and speech tone;
[0017] Step 1d: Extract image frames from the 30 Hz video clips and use the MTCNN face detection algorithm to extract aligned faces. Then use the MultiComp OpenFace2.0 toolkit to extract multiple frame-level visual features, including facial landmarks and facial action units.
[0018] Furthermore, step 2 includes the following steps:
[0019] Step 2a: In order to enable the unimodal features to learn their contextual information before cross-modal fusion, a long short-term memory network is used to model the internal interaction of the unimodal features; that is, the text, audio and visual features extracted in step 1 are respectively input into the long short-term memory network for extraction to obtain the unimodal hidden layer features (text hidden layer features, audio hidden layer features, visual hidden layer features);
[0020] Step 2b, using a stacked cross-modal feature fusion module to fuse the single-modal hidden layer features obtained in step 2a; wherein the stacked cross-modal feature fusion module includes two stacked gated cross-modal transformer networks, and the gated cross-modal transformer network is used to fuse the two modal features of the input.
[0021] The specific steps of the stacked cross-modal feature fusion module are: first, the text modality hidden layer features are used as the main modality, and the audio hidden layer features are used as the auxiliary modality, and the unimodal high-level features of the corresponding modality (text high-level features, audio high-level features) are input for guidance, and new language features (including fusion features of text and audio information) are obtained after fusion; then, the newly obtained language features are used as the main modality, and the visual features are used as the auxiliary modality, and the corresponding modality high-level features (unimodal high-level features) are input for guidance to obtain the fusion features of the three modalities of text, audio, and vision, which are called multimodal fusion features.
[0022] Further, step 2a includes the following sub-steps:
[0023] Step 2a-1: For the text, audio and visual features X obtained in step 1 m ∈R batch×length×dimm , m∈[t,a,v], first input it into the long short-term memory network to obtain the hidden layer features, X mh =LSTM m (X m ), where X m h ∈R batch ×length×dimh ; The input dimensions of LSTM networks of different modalities are different, but the hidden layer dimensions are unified as dim h , to adapt to the subsequent gated cross-modal transformer fusion;
[0024] Step 2a-2: X m h As the input of the gated cross-modal transformer network, take X m h The last time step X m,l h , and pass it through a multi-layer perceptron to obtain a single-modal high-level feature F m finall : F m final =MLP(X m,l h ), where the multilayer perceptron consists of multiple feed-forward layers.
[0025] Step 2a-3: During the training process, in order to filter out the noise in the original features, the model often filters out important information such as the heterogeneous features of the single modality, and only retains the common features in the modality. However, the heterogeneity of features is very important for model fusion. In order to retain the heterogeneity of single modality features, F m finall After multi-layer perceptron and softmax, the unimodal classification result y′ is obtained m , calculate the cross entropy loss when training the true label to guide the long short-term memory network to extract unimodal hidden layer features: y′ m =softmax(MLP(F m finall ), the loss of the subtask corresponding to this mode is:
[0026]
[0027] Among them, softmax is the logistic regression function, y′ m is the probability of different sentiment classifications, log is the logarithmic function, N is the batch size, d is the number of sentiment categories, y′ m,i,k To predict the probability that the i-th sample in the batch belongs to class k, y m,i,k is the true label of the i-th sample in the batch.
[0028] Further, step 2b includes the following sub-steps:
[0029] Step 2b-1, for the unimodal hidden layer feature X obtained from the unimodal feature extraction module t h , X a h , X v h With the unimodal high-level features F t finall 、F a finall 、F v finall , first change X t h , X a h 、F t finall 、F a finall Input the gated cross-modal transformer network to obtain the text feature X enhanced by the audio feature l h , because it integrates audio features, its pattern is closer to language, so this feature is defined as a language feature; then, X l h The last time step feature of is input into the multi-layer perceptron to obtain the language high-level feature F l finall ;
[0030] Step 2b-2, X l h , X v h 、F l finall 、F v finall Input the gated cross-modal transformer network to obtain the multimodal fusion feature X multi h , take X multi h The last time step X multi,l h , X multi,l h Input to the multi-layer perceptron to obtain the multimodal high-level features F multi finall .
[0031] Furthermore, the gated cross-modal transformer network in step 2b includes the following sub-steps:
[0032] Step 2b-1-1: The cross-modal transformer α→β contains D layers of cross-modal attention blocks. For a certain layer of cross-modal attention blocks, the target modality X is input. α [i-1] , X β [0] , define Query as The key is Values are in is the mapping matrix; the auxiliary modality β is used to guide the multi-head attention of the target modality α to obtain the guided fusion feature The formula is as follows:
[0033]
[0034] Step 2b-1-2: In order to strengthen the feature information of the target modality, the transformer uses residual connections to make the original target modality feature X β To obtain the supplement of the auxiliary modality, in order to prevent the large difference between the target modality and the auxiliary modality from interfering with the fusion in complex scenarios, a gating mechanism is added to the residual connection of the cross-modal transformer; by comparing the cosine similarity of the high-level features corresponding to the target modality and the auxiliary modality, the similarity is obtained. αβ As the gating of residual connections, it guides cross-modal transformer fusion to obtain new features Finally Input the feedforward network and perform layer-by-layer normalization to obtain the cross-modal fusion features of this layer As the target modality input of the next transformer layer; the formula is as follows:
[0035]
[0036]
[0037]
[0038] In the formula, and is the mapping matrix, and is the high-level feature passed in step 2b-2, and is the eigenvalue corresponding to the i-th dimension, the cos() function is used to calculate the cosine similarity of two input features, LayerNorm is the normalization layer, and fowrwad is the feedforward layer.
[0039] Furthermore, step 3 includes the following steps:
[0040] Step 3a: In order to further supplement the feature content and prevent the loss of key information, the multimodal high-level feature F obtained in step 2b is multi finall With the unimodal high-level features F t finall , F v finall After splicing, the supplemented multimodal high-level features F are obtained multi cat , input to a multi-layer perceptron for sentiment classification;
[0041]
[0042]
[0043] Among them, cat represents the splicing operation.
[0044] Step 3b, calculate the multi-task loss of sentiment analysis for this batch, Among them, i∈{t,a,v,m}, hyperparameter ω i Indicates the weight of different modal losses in the overall loss. Set the initial time w t ,w a ,w v The weight is 1, w m The weight of is 0. As the number of training times increases, the loss weight of the unimodal task is gradually reduced to 0.6, and the loss weight of the multimodal task is gradually increased to 1.
[0045] The beneficial effects of the present invention are as follows: existing multimodal sentiment analysis methods often only use the multimodal overall label as the only task loss training model, and adopt a ternary symmetric fusion model to equally fuse the features of different modalities. However, the emotional information contained in different modalities is not equal. In fact, the text modality should be the main body of multimodal emotion, and the audio and visual modalities are auxiliary. On the other hand, the context feature vector after modeling by the unimodal feature extraction network will tend to be consistent in cross-modal fusion due to reasons such as feature collapse, and the heterogeneity of unimodal features will be difficult to retain, resulting in a decrease in the accuracy of sentiment classification of the model in complex environments. In view of the above problems, the present invention solves the problem of inaccurate classification of existing multimodal sentiment analysis technology in complex scenes by sequentially inputting the features of different modalities into multiple stacked gated cross-modal transformer networks for fusion; the multi-task learning mechanism adopted by the present invention can supervise the extraction of single-modal features and prevent feature collapse due to modal fusion, and the stacked cross-modal fusion network adopted can effectively capture the association between modalities; the multimodal sentiment analysis method proposed by combining the single-modal feature extraction network with the cross-modal feature fusion network can coherently fuse the features of different modalities while retaining the heterogeneity of modalities, thereby effectively identifying multimodal emotions in complex scenes. It has good robustness in the presence of complex emotional expressions such as ambiguity and irony, and obtains a higher accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 The figure is an overall flow chart of the method of the present invention.
[0047] Figure 2 This is a diagram of the single-modal feature extraction network structure in the present invention.
[0048] Figure 3 This is the overall network structure diagram of the present invention.
[0049] Figure 4 This is a diagram of the gated cross-modal transformer network structure in the present invention. DETAILED DESCRIPTION
[0050] For ease of understanding the present invention, further description is given below in conjunction with embodiments. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.
[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification are only for describing specific implementation purposes and are not intended to limit the present invention.
[0052] like Figure 1 As shown, a multimodal sentiment analysis method based on multi-task learning and stacked cross-modal fusion provided by an embodiment of the present invention comprises the following steps:
[0053] First, the original video is divided into video segments and the emotion labels corresponding to the video segments are marked (emotion labels are divided into three categories: positive, neutral and negative); then, text, audio and visual data (features) that can be recognized by the model are extracted from the video and input into the unimodal feature extraction module to obtain unimodal hidden layer features and unimodal high-level features; then, the extracted unimodal hidden layer features and unimodal high-level features are input into the stacked cross-modal feature fusion module to obtain multimodal high-level features; the multimodal high-level features are spliced with the unimodal high-level features (text, vision) and input into the multi-layer perceptron, and the final emotion classification result is output. After calculating the multi-task loss value, the loss weights of different tasks (text emotion prediction, audio emotion prediction, visual emotion prediction, overall emotion prediction) are dynamically adjusted according to the algorithm, and the model is iteratively optimized.
[0054] The specific steps are as follows:
[0055] Step 1: First, slice the complete video according to the utterances so that each video segment only contains a complete utterance and the corresponding emotion label; pre-process the text, audio and visual data contained in each video segment to extract text feature vectors, audio feature vectors and video feature vectors.
[0056] The data preprocessing for text modality is divided into two steps: first, the number of characters in each text in the data set is counted, and its average length L is calculated. 2L is used as the text truncation length to cover the longer text content in the sample set to the greatest extent. The final text length is 39; then, the truncation text is input into the 762-dimensional BERT word embedding model, and for the batch size n, the word embedding vector X is obtained. T ∈n×39×762.
[0057] For the audio modality, the LibROSA audio toolkit is used to extract 33-dimensional frame-level audio features related to emotions and speech tone, including one-dimensional logarithmic fundamental frequency (log F0), 20-dimensional Mel frequency cepstral coefficients (MFCCs) and 12-dimensional constant q spectrum (CQT), and the audio feature vector X is obtained. A ∈n×39×33.
[0058] For the visual modality, image frames are extracted from 30Hz video clips, and the MTCNN face detection algorithm is used to extract aligned faces. Then, the MultiComp OpenFace2.0 toolkit is used to extract 68 facial landmarks, 17 facial action units, head pose, head direction, and eye gaze. A total of 709-dimensional frame-level visual features are extracted to obtain the visual feature vector X V ∈n×39×762.
[0059] Step 2: Input text, audio and visual features into a unimodal feature extraction network respectively, and then input the obtained features into a cross-modal feature fusion network for feature fusion; this is one of the innovative points of the present invention.
[0060] The process is as follows:
[0061] For the text feature X t , audio feature X a With visual features X v , respectively input them into the LSTM network to obtain the hidden layer feature X t h , X a h , X v h , as the input of the cascaded cross-modal feature fusion module.
[0062] X t h =LSTM(X t ,W T )
[0063] X a h =LSTM(X a ,W A )
[0064] X v h =LSTM(X v ,W V )
[0065] Where W T , W A , W V They are the network parameters of the long and short cycle memory neural network for text (t), auditory (a), and visual (v) modalities respectively.
[0066] Take the hidden layer features X respectively t h , X a h , X v h The terminal state hidden representation Xt,l h , X a,l h , X v,l h . X t,l h , X a,l h , X v,l h Input the multi-layer perceptron respectively to obtain the high-level features X of text, vision, and audio modalities t finall , X a finall , X v finall ,These features are used on one hand to predict unimodal sentiment polarity as part of the ,multitask loss for sentiment analysis to supervise the model to preserve unimodal feature heterogeneity ,in the training phase, and on the other hand as the input of the cascaded cross-modal ,feature fusion module to guide cross-modal feature fusion.
[0067] X t finall =MLP(X t,l h ,W M T )
[0068] X a finall =MLP(X a,l h ,W M A )
[0069] X v finall =MLP(X v,l h ,W M V )
[0070] y′ m =softmax(MLP(F m finall ))
[0071] Among them, W M T ,W M A ,W M V The network parameters of the multi-layer perceptron for text (t), auditory (a), and visual (v) modalities, respectively. Softmax is the logistic regression function, and MLP is the multi-layer perceptron.
[0072] For the unimodal hidden layer feature X obtained from the unimodal feature extraction module t h , X a h , X v h With unimodal high-level features X t finall , X a finall , X v finall , first change X t h , X a h , X t finall , X a finall Input to the gated cross-modal transformer network to obtain the text feature X enhanced by the audio feature l h , because it integrates audio features, its pattern is closer to language, and is defined as language feature X l h , and then X l h According to the process of steps 2a-3, after passing through the long short-term memory network and the multi-layer perceptron, the language high-level feature X is obtained l finall Then, X l h , X v h , X l finall , X v finall Input to the gated cross-modal transformer network to obtain the multimodal emotional feature X multi .
[0073] The gated cross-modal transformer network includes multiple layers of gated cross-modal transformer unit blocks. For any layer of cross-modal attention blocks, the following steps are included:
[0074] Enter the target mode X α [i-1] , X β [0] , define Query as The key is Values are in is the mapping matrix; the auxiliary modality β is used to guide the multi-head attention of the target modality α to obtain the guided fusion feature The formula is as follows:
[0075]
[0076] In order to strengthen the characteristic information of the target modality, the transformer uses residual connection to make the original target modality characteristic X β To obtain the supplement of the auxiliary modality, in order to prevent the large difference between the target modality and the auxiliary modality from interfering with the fusion in complex scenarios, a gating mechanism is added to the residual connection of the cross-modal transformer. By comparing the cosine similarity of the high-level features corresponding to the target modality and the auxiliary modality, the similarity is obtained. αβ As the gating of the residual connection, it guides the cross-modal transformer fusion to obtain Finally Input the feedforward network and perform layer-by-layer normalization to obtain the cross-modal fusion features of this layer As the target modality input of the next transformer layer. The formula is as follows:
[0077]
[0078]
[0079]
[0080] In the formula, and is the mapping matrix, and is the high-level feature passed in step 2b-2, and is the eigenvalue corresponding to the i-th dimension, the cos() function is used to calculate the cosine similarity of two input features, LayerNorm is the normalization layer, and fowrwad is the feedforward layer.
[0081] Step 3: The multimodal high-level features F obtained in step 2b multi finall With the unimodal high-level features F t finall , F v finall Splice to obtain the supplemented multimodal high-level features F multi cat , and input into a multi-layer perceptron for sentiment classification;
[0082]
[0083]
[0084] Among them, cat represents the splicing operation.
[0085] After calculating the multi-task loss values, the loss weights of different tasks (text emotion prediction, audio emotion prediction, visual emotion prediction, and overall emotion prediction) are dynamically adjusted according to the algorithm to facilitate iterative optimization of the model; this is another innovation of the present invention.
[0086] Calculate the multi-task loss for sentiment analysis of this batch, Among them, i∈{t,a,v,m}, hyperparameter ω i Indicates the weight of different modal losses in the overall loss. Set the initial time w t ,w a ,w v The weight is 1, w m The weight of is 0. As the number of training times increases, the loss weight of the unimodal task is gradually reduced to 0.6, and the loss weight of the multimodal task is gradually increased to 1.
[0087] Experimental analysis:
[0088] Dataset:
[0089] The dataset used in the experiment is the Chinese multimodal sentiment analysis dataset SIMS. Compared with traditional single-label datasets such as MOSI (Zadeh et al. 2016) and MOSEI (Zadeh et al. 2018b), SIMS provides additional text, audio and visual single-modal labels. The dataset consists of 2281 video clips extracted from 60 original videos, and the labeler is asked to give the sentiment score of the modality from -1 (strongly negative) to +1 (strongly positive) when only accepting a single modality input.
[0090] Evaluation Metrics:
[0091] This experiment is a classification task, so we choose 3-class classification accuracy (Acc-3), positive and negative sentiment (p / g) classification accuracy (Acc-2) and F1 value (F1-Score).
[0092] In order to prove that the present invention and the proposed innovations are effective, this experiment is verified by comparing the performance with the baseline model and performing ablation experiments.
[0093] Baseline model comparison experiment:
[0094] The proposed model is compared with single-task models (TFN, Graph-MFN, MISA, MULT) and multi-task models (MLF-DNN, M-LMF), and the experimental results are shown in Table 1. The multimodal sentiment analysis method (MSGC) based on multi-task learning and stacked cross-modal fusion has surpassed the most advanced methods in various evaluation indicators. Compared with the single-task optimal baseline model MULT, the binary classification accuracy and F1 value of MSGC on this dataset increased by 3.66% and 3.97% respectively, and the three-classification accuracy and F1 value increased by 4.16% and 4.45% respectively. Compared with the multi-task optimal model M-LMF, the binary classification accuracy and F1 value of MSGC increased by 2.42% and 2.73% respectively, and the three-classification accuracy and F1 value increased by 2.06% and 0.97% respectively.
[0095] In addition, the performance of the multi-task baseline method is significantly higher than that of the single-task method. This is because the unimodal label provides more information sources for multimodal sentiment analysis, reduces the error of unimodal feature extraction, and retains important information. In order to compare more fairly with the single-task model, the MSGC-single model is also set up. This model only takes the loss value of multimodal sentiment classification and does not use multi-task labels. MSGC-single is still significantly improved compared to the optimal single-task baseline model.
[0096]
[0097] Ablation experiment:
[0098] In order to examine the functionality of the entire architecture and the components introduced in this work, an ablation study is performed on the SIMS dataset as follows.
[0099] MSGC-single: Trained using only single-task labels based on the full model.
[0100] MSGC w / o late-fusion: Late-fusion was removed based on the full model.
[0101] MSGC w / o late-fusion&gate: The cross-modal similarity gating module is deleted based on MSGC w / o late-fusion.
[0102] MSGC-concat: Replace the stacked transformer structure of MSGC w / o late-fusion & gate with a concatenated structure (similar to the structure of mult).
[0103]
[0104] Table 2 gives the ablation experimental results of the module combination on the SIMS dataset. The experimental results show that the complete MSGC model has the best experimental effect, and the lack of different modules will have different effects on the experimental results. For MSGC w / o late-fusion and MSGC, the late fusion part is directly omitted in the multimodal emotion output, and the high-level features of the unimodal are not spliced with the high-level features of the multimodal. Only the multimodal high-level features are taken as the input of the classifier. All aspects of its performance are lower than the MSGC model, which shows that the high-level features of the unimodal guided by the unimodal label can provide complementarity for multimodal emotions. Compared with w / o late-fusion, w / o late-fusion&gate omits the cross-modal similarity gating module, and its overall performance is slightly lower than that of the w / olate-fusion model. It can be seen that by using the cosine similarity of the high-level features of the two modalities as the gating of the residual connection during cross-modal fusion, the features of different modalities can be better integrated. In addition, the difference between w / olate-fusion&gate and MSGC-concat is that MSGC-concat does not use a stacked transformer structure, but instead combines the three input modalities in pairs, performs a cross-modal transformer, and concatenates the 6 sets of bimodal fusion features obtained, so its parameter count and operation time are much higher than w / o late-fusion&gate. On the SIMS dataset, the performance of w / o late-fusion&gate is significantly higher than that of MSGC-concat, which further illustrates the superiority of the stacked transformer structure.
[0105] Noun reference list
[0106] Unimodal features (text modal features X t , audio modal feature X a , visual modality feature X v )
[0107] Unimodal hidden layer features (text hidden layer modality features X t h , audio hidden layer modal features X a h , visual hidden layer modal features X v h )
[0108] Unimodal high-level features (text high-level features F t finall , audio high-level features F a finall , visual high-level features F v finall , language high-level features)
[0109] Multimodal fusion feature X multi h
[0110] Multimodal high-level features F multi final
[0111] The supplemented multimodal high-level features F multi cat
[0112] Cascaded cross-modal feature fusion module (including 2 gated cross-modal transformer networks).
[0113] Unimodal feature extraction module (including 3 long short-term memory networks)
[0114] Long Short-Term Memory (LSTM)
[0115] Feedforward layer
[0116] Multilayer Perceptron (MLP)
[0117] Logistic regression function (Softmax).
Claims
1. A multimodal sentiment analysis method based on multi-task learning and cascade cross-modal fusion, the specific execution steps of which are as follows: Step 1) split the original video into video clips, mark the emotion tags corresponding to the video clips, and then extract text, audio and visual features that can be recognized by the model from the video clips; The emotional labels are divided into three categories: positive, neutral and negative; Step 2) The text, audio and visual features extracted in step 1) are first input into a unimodal feature extraction module, which contains three long short-term memory networks for intra-modal interaction to obtain unimodal hidden layer features with contextual semantic information, and then these unimodal hidden layer features are input into a stacked cross-modal feature fusion module for feature fusion and extraction to obtain multimodal high-level features; Step 3) After splicing the multimodal high-level features obtained in step 2 with the unimodal high-level features, a multi-layer perceptron is used to output the final sentiment classification result, and after calculating the multi-task loss value, the loss weights of different tasks are dynamically adjusted according to the algorithm, and the model is iteratively optimized; the different tasks include text sentiment prediction, audio sentiment prediction, visual sentiment prediction, and overall sentiment prediction; Step 2 includes the following sub-steps: Step 2a: In order to enable the unimodal features to learn their contextual information before cross-modal fusion, a long short-term memory network is used to model the internal interaction of the unimodal features; that is, the text, audio and visual features extracted in step 1 are respectively input into the long short-term memory network for extraction to obtain unimodal hidden layer features, namely, text hidden layer features, audio hidden layer features and visual hidden layer features; Step 2b, using a stacked cross-modal feature fusion module to perform feature fusion on the single-modal hidden layer features obtained in step 2a; wherein the stacked cross-modal feature fusion module comprises two stacked gated cross-modal transformer networks, and the gated cross-modal transformer network is used to fuse the two modal features of the input; The specific steps of the stacked cross-modal feature fusion module are: first, the text modality hidden layer features are used as the main modality, the audio hidden layer features are used as the auxiliary modality, and the single modality high-level features of the corresponding modality are input for guidance, and new language features are obtained after fusion; the single modality high-level features include text high-level features and audio high-level features, and the new language features include the fusion features of text and audio information; The newly obtained language features are used as the main modality, and the visual features are used as the auxiliary modality. The high-level features of the corresponding modality are input for guidance to obtain the fusion features of the three modalities of text, audio, and vision. This fusion feature is called a multimodal fusion feature.
2. The multimodal sentiment analysis method based on multi-task learning and cascaded cross-modal fusion according to claim 1, characterized in that: Step 1 includes the following steps: Step 1a: according to the speaker's speech interval in the original video, the video is cut into 5-10 second video clips, each video clip contains a complete sentence, and the speech is multimodal emotional annotation is performed on the speech; the multimodal emotional annotation includes text, audio, vision and overall emotion; Step 1b: Transcribe the subtitles in the video clip into text and use BERT word embedding to convert the text into word vector X T ∈R length×dim , and set the maximum sentence length to 39 and the word embedding dimension to 762; Step 1c: For audio clips, use the LibROSA audio toolkit to extract 33-dimensional frame-level audio features, including one-dimensional logarithmic fundamental frequency, 20-dimensional Mel-frequency cepstral coefficients, and 12-dimensional constant q spectrum, all of which are related to emotions and speech tone; Step 1d: Extract image frames from the 30 Hz video clips, use the MTCNN face detection algorithm to extract aligned faces, and then use the MultiComp OpenFace2.0 toolkit to extract multiple frame-level visual features, including facial landmarks and facial action units.
3. The multimodal sentiment analysis method based on multi-task learning and cascaded cross-modal fusion according to claim 2 is characterized in that: Step 2a includes the following steps: Step 2a-1: For the text, audio and visual features X obtained from step 1 m ∈R batch×length×dimm , m∈[t,a,v], first input it into the long short-term memory network to obtain the hidden layer features, X m h =LSTM m (X m ), where X m h ∈R batch ×length×dimh ; The input dimensions of LSTM networks of different modalities are different, but the hidden layer dimensions are unified as dim h , to adapt to the subsequent gated cross-modal transformer fusion; Step 2a-2: X m h As the input of the gated cross-modal transformer network, take X m h The last time step X m,l h , and pass it through a multi-layer perceptron to obtain a single-modal high-level feature F m finall : F m final =MLP(X m,l h ), where the multilayer perceptron consists of multiple feed-forward layers; Step 2a-3: During the training process, F m finall The unimodal classification result is obtained through the multi-layer perceptron. The cross entropy loss is calculated when training the real label to guide the long short-term memory network to extract the unimodal hidden layer feature y′ m =softmax(MLP(F m finall )); The loss of the subtask corresponding to this mode is Among them: softmax is the logistic regression function, y′ m is the probability of different sentiment classifications, log is the logarithmic function, N is the batch size, d is the number of sentiment categories, y′ m,i,k To predict the probability that the i-th sample in the batch belongs to class k, y m,i,k is the true label of the i-th sample in the batch.
4. The multimodal sentiment analysis method based on multi-task learning and stacked cross-modal fusion according to claim 3 is characterized by: Step 2b includes the following steps: Step 2b-1: For the unimodal hidden layer feature X obtained from the unimodal feature extraction module t h , X a h , X v h With the unimodal high-level features F t finall 、F a finall 、F v finall , first change X t h , X a h 、F t finall 、F a finall Input the gated cross-modal transformer network to obtain the text feature X enhanced by the audio feature l h ; Because the audio feature is integrated, its pattern is closer to language, so this feature is defined as a language feature; then, X l h The last time step feature of is input into the multi-layer perceptron to obtain the language high-level feature F l finall ; Step 2b-2: X l h , X v h 、F l finall 、F v finall Input the gated cross-modal transformer network to obtain the multimodal fusion feature X multi h , take X multi h The last time step X multi,l h , X multi,l h Input to the multi-layer perceptron to obtain the multimodal high-level features F multi finall .
5. The multimodal sentiment analysis method based on multi-task learning and stacked cross-modal fusion according to claim 4 is characterized in that: The gated cross-modal transformer network used in step 2b-1 includes the following steps: Step 2b-1-1: The gated cross-modal transformer α→β contains D layers of cross-modal attention blocks. For a certain layer of cross-modal attention blocks, the target modality X is input α [i-1] , X β [0] , define Query as The key is Values are in: is the mapping matrix; Use the auxiliary modality β to guide the target modality α with multi-head attention to obtain the guided fusion features The formula is as follows: Step 2b-1-2, Gated Cross-modal Transformer Network adds a gating mechanism to the residual connection of the traditional cross-modal Transformer network; by comparing the cosine similarity of the high-level features corresponding to the target modality and the auxiliary modality, the similarity is obtained. αβ As the gating of residual connections, it guides cross-modal transformer fusion to obtain new features Finally Input the feedforward network and perform layer-by-layer normalization to obtain the cross-modal fusion features of this layer As the target modality input of the next transformer layer; The formula is as follows: In the formula, and is the mapping matrix, and is the high-level feature passed in step 2b-2, and is the eigenvalue corresponding to the i-th dimension, the cos() function is used to calculate the cosine similarity of two input features, LayerNorm is the normalization layer, and fowrwad is the feedforward layer.
6. The multimodal sentiment analysis method based on multi-task learning and cascaded cross-modal fusion according to claim 1, characterized in that: Step 3 includes the following steps: Step 3a: In order to further supplement the feature content and prevent the loss of key information, the multimodal high-level feature F obtained in step 2b is multi finall With the unimodal high-level features F t finall , F v finall Splice to obtain the supplemented multimodal high-level features F multi cat , and input into a multi-layer perceptron for sentiment classification; Among them, cat represents the splicing operation; Step 3b: Calculate the multi-task loss of sentiment analysis for this batch. Among them, i∈{t,a,v,m}, hyperparameter ω i Indicates the weight of different modal losses in the overall loss; Set the initial time t ,w a ,w v The weight is 1, w m The weight of is 0; as the number of training times increases, the loss weight of the unimodal task is gradually reduced to 0.6, and the loss weight of the multimodal task is gradually increased to 1.
Citation Information
Patent Citations
Attention fusion-based online short video multi-modal emotion recognition method
CN111275085A
Multi-modal emotion analysis method based on multi-dimensional attention fusion network
CN111680541A
Audio and video multi-mode sentiment classification method and system
CN113408385A
Weak supervision neural network sign language recognition method based on multi-layer time sequence attention fusion mechanism
CN113537024A
