Multi-modal interaction sentiment analysis method based on semi-supervised learning

Abstract

The invention discloses a multi-modal interaction sentiment analysis method based on semi-supervised learning. According to the method, firstly, feature extraction is carried out on voice, text and visual modals, then feature fusion is carried out through an intra-modal interaction branch network and an inter-modal interaction branch network, finally, fusion information of the two interaction branches is extracted in a self-adaptive mode based on a gating mechanism, and the intra-modal interaction network and the inter-modal interaction network are based on a transformer encoder structure. The difference is that the provided attention mask mechanism is introduced into the multi-head self-attention layer; according to the training method, a pseudo label is generated in a semi-supervised learning mode to assist in stable training of the model, and dependence on label data is reduced. According to the method, the problems of high manual annotation cost and how to mine effective interaction information in the field of multi-modal sentiment analysis are solved, and the sentiment recognition precision is improved.

Description

Technical Field

[0001] The present invention belongs to the field of emotion computing in multimodal learning, relates to the fields of human-computer interaction and artificial intelligence, and specifically relates to a multimodal interaction emotion analysis method based on semi-supervised learning. Background Art

[0002] With the rapid development of social media, people can express their personal opinions and feelings through text and video. Sentiment analysis, as a subtask of natural language processing, aims to understand the subject's attitude or opinion. Traditional sentiment analysis methods mainly start from the text modality and analyze the subject's emotional state by mining the potential semantic information of the text. Due to the correlation and complementarity between different modal data, multimodal sentiment analysis (MSA) has quickly become a current research hotspot. The main task of multimodal sentiment analysis is to achieve accurate emotion recognition by comprehensively analyzing information from different modalities (such as text, audio, video, image, etc.). Multimodal sentiment analysis is of great significance in many fields. It is widely used in human-computer interaction, public opinion surveys, news video analysis, and healthcare.

[0003] From the perspective of whether the training dataset covers unlabeled data, existing methods can be divided into semi-supervised learning and fully supervised learning methods. The semi-supervised learning method aims to improve the performance of the model by assisting model training with unlabeled data. Liang et al. believed that there is emotional consistency between modalities at the sentence level, so they proposed a semi-supervised emotion recognition model based on a cross-modal feature matching strategy to solve the problem of high cost of emotion labeling. However, this method did not take into account the contextual information in the conversation. Therefore, Lian et al. used a recurrent network and a hybrid fusion strategy to model the contextual information between modalities, and achieved good results on multiple datasets.

[0004] Different from semi-supervised learning methods, fully supervised learning methods focus on solving the difficulties of heterogeneous modal fusion and how to explicitly and implicitly align modalities. In terms of modal alignment, Tsai et al. use a cross-modal attention mechanism to achieve information interaction from the source modality to the target modality. This method does not require sequence alignment operations for each modality in advance, solving the problem of difficult manual alignment between modalities caused by different sampling rates. Yu et al. use an implicit cross-modal alignment strategy (SPECTRA) to pre-train speech and text dialogues, and predict the start and end time of each text word in the speech waveform by designing a time position prediction task, thereby capturing the explicit alignment between speech and text. In terms of modal fusion, Pham et al. proposed a multimodal recurrent transformation network (MCTN) to learn a robust joint multimodal intermediate representation through conversion between modalities. The advantage of this method is that it is not affected by modality loss during the test phase. Rahman et al. embed a multimodal threshold network into a pre-trained text encoder to introduce emotional knowledge of audio and video modalities into the pre-trained model. Li et al. proposed an enhanced modality awareness network (AMOA), which incorporates global acoustic information knowledge that reflects the overall changes in pitch into the model by constructing a contrastive learning loss function.

[0005] Although existing methods have made significant progress in fully supervised learning or semi-supervised learning, they either fail to effectively filter out the specificity within irrelevant modalities or the complementary knowledge between modalities, or fail to utilize the significant features of emotion in unlabeled data. Therefore, the present invention aims to improve the training method and overall architecture of existing methods to achieve higher accuracy of emotion recognition.

[0006] The shortcomings of existing technologies: The semi-supervised learning method used by SMIN requires more unlabeled data and the performance improvement is not obvious. At the same time, this method can only mine the specificity within the modality and the complementary information between modalities at the sentence level. However, some studies have shown that (Tsai YHH, Bai S, Liang PP, et al. Multimodal transformer for unaligned multimodal language sequences [C] / / Proceedings of the conference. Association for computational linguistics. Meeting. NIH Public Access, 2019, 2019: 6558. and Yu T, Gao H, Lin TE, et al. Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment [C] / / Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).2023:7900-7913.), capturing interactive information at a fine-grained level (between words within the text modality, between frames of video or speech within the visual or speech modality, and between words and video frames between text and visual modalities) helps improve the model's ability to recognize emotions. Summary of the invention

[0007] The purpose of the present invention is to propose a multimodal interactive sentiment analysis method based on semi-supervised learning. In terms of training methods, this method generates pseudo labels from unlabeled data sets through semi-supervised learning. Further, it selects high-confidence samples from pseudo labels to assist in model retraining, thereby reducing the dependence on label data. In terms of model structure, this method constructs intra-modal and inter-modal encoders through the proposed attention mask mechanism, mines effective interactive information within the modality and between different modalities, and then selectively filters through a gating mechanism to achieve accurate sentiment prediction.

[0008] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0009] A multimodal interactive sentiment analysis method based on semi-supervised learning includes the following steps:

[0010] S1. Build the training set used in the supervised learning phase based on the open source dataset Validation set and test set At the same time, based on the AMI dataset, an unlabeled dataset D used in the semi-supervised learning stage is constructed. u ;

[0011] S2, extracting features from speech and text through speech encoder and text encoder; visual modality obtains face area through frame-level face detection to reduce the influence of irrelevant background noise, and then uses visual encoder to extract visual features from face area;

[0012] S3, performing a one-dimensional convolution operation on the extracted speech, text and visual features to obtain local features of speech, text and vision;

[0013] S4, modality encoding is performed on the local features, and then they are concatenated with the sentiment polarity classification head and the sentiment intensity classification head along the dimension of feature length, and finally position encoding is performed to obtain the input sequence of the intra-modality / inter-modality encoder;

[0014] S5, performing feature fusion on the input sequence through the intra-modality encoder and the inter-modality encoder to obtain intra-modality and inter-modality fusion representation;

[0015] S6. The intra-modal and inter-modal fusion representations are selectively filtered through the gating mechanism, and then mapped through the projection layer to obtain the sentiment intensity and polarity predictions. The two are combined with the sentiment polarity label and intensity label to calculate the loss L total , after gradient optimization, select The best performing one gets the pre-trained model;

[0016] S7: Using the pre-trained model obtained in step S6, the unlabeled dataset D constructed in step S1 is trained. u Perform inference, obtain predictions and their confidence, adopt a top-k screening strategy for the predictions, and obtain a semi-supervised dataset including sentiment polarity labels

[0017]

[0018] S8. Integrated Dataset D ′ u With labeled training set Repeat steps S2-S6 to obtain the final emotion recognition model;

[0019] S9: The emotion recognition model obtained in step S8 is used to infer the test set constructed in step S1 This is used to evaluate the model performance.

[0020] Furthermore, in step S2, the speech, text and visual modality data are obtained from an open source dataset, and for the visual modality data, the face region is extracted using OpenFace;

[0021] The text encoder Roberta-large, speech encoder HuBERT, and visual encoder Fabnet were pre-trained on the datasets in their respective fields, and then the features of their respective modalities were obtained based on the three encoders:

[0022] in, represents the matrix dimension, h t 、h a 、h v Represents text, speech and visual features, l t , l a , l v Represent the length of text, speech and visual features respectively, d t ,d a ,d v Represent the dimensions of text, speech and visual features respectively.

[0023] Furthermore, in step S3, the one-dimensional convolution layer corresponding to the text and visual modalities includes a single layer of one-dimensional convolution with a convolution kernel size of 3×3, and the step size is 1; the one-dimensional convolution layer corresponding to the speech modality includes two layers of one-dimensional convolution with a convolution kernel size of 4×4, and the step sizes are 2 and 4 respectively;

[0024] After one-dimensional convolution operation, the local features of the text are obtained Local features of speech Visual local features Among them l ′ t , l ′ a , l ′ v Represents the length of local features of text, speech and visual modalities, d h Represents the feature dimension of the latent space of the subsequent intra-modal and inter-modal encoders.

[0025] Furthermore, in step S4, the local features of speech, text and visual modalities are modally encoded, that is, they are added to the feature vectors representing the corresponding modal types to indicate which modality the local features belong to. Then, the encoded local features are concatenated with the emotion intensity classification head and the emotion polarity classification head along the dimension of feature length, and the concatenated result is added to the position vector to encode the position information, and finally the input sequence Z of the intra-modal and inter-modal encoders is obtained. intra and Z inter ;

[0026] The feature vectors, position vectors, sentiment intensity and sentiment polarity classification heads representing the corresponding modal types are all learnable parameter matrices initialized with normal distribution.

[0027] Furthermore, the intra-modal encoder and the inter-modal encoder are obtained by stacking n layers of intra-modal sub-encoders and inter-modal sub-encoders respectively;

[0028] The intra-modality sub-encoder includes layer normalization (LN), intra-modality attention mask mechanism (IntraMA), layer normalization and fully connected layer (FFN). The input and output flows of the intra-modality sub-encoder are as follows:

[0029]

[0030] in, is the input sequence of the first-layer intramodal sub-encoder, is the output sequence of the sub-encoder within the l+1th modality, l∈[1,n], n represents the total number of layers;

[0031] The inter-modality sub-encoder includes layer normalization, inter-modality attention mask mechanism (InterMA), layer normalization and fully connected layers. The input and output flows of the inter-modality sub-encoder are as follows:

[0032]

[0033] in, is the input sequence of the first layer inter-modal sub-encoder, is the output sequence of the l+1th inter-modal sub-encoder, l∈[1,n], and n represents the total number of layers.

[0034] Furthermore, in step S5, the intra-modality / inter-modality attention mask mechanism constructs masks for attention at different positions to capture different types of modal interactions;

[0035] The intra-modality attention mask mechanism is implemented as follows:

[0036]

[0037] in, represents the output sequence of the sub-encoder in the lth layer after layer normalization, that is, represents the output sequence of the intra-modality attention mask mechanism of the intra-modality sub-encoder at the lth layer, W Q , W K , W V represents the learnable projection matrix, Q, K, V represent the query vector, key vector, and value vector obtained after projecting the input; d krepresents the latent space dimension; is the intra-modal mask, which represents the masking operation of the modal interaction attention at a specific position. For the element IntraMASK in the i-th row and j-th column in IntraMASK ij , which is implemented as follows:

[0038]

[0039] Among them, Intra pos Represents the position where attention needs to be masked in the intra-modal attention masking mechanism;

[0040] The inter-modal attention mask mechanism is implemented as follows:

[0041]

[0042] in, represents the output sequence of the inter-modal sub-encoder of the lth layer after layer normalization, that is, represents the output sequence of the inter-modal attention mask mechanism of the inter-modal sub-encoder at the lth layer; for the element InterMASK in the i-th row and j-th column of the inter-modal mask InterMASK ij , which is implemented as follows

[0043]

[0044] InterMASK represents the masking operation of the modal interaction attention at a specific location. pos Indicates the position where attention needs to be masked in the inter-modal attention masking mechanism.

[0045] Furthermore, for the intra-modality and inter-modality fusion representation in step S5, in step S6, gated filtering and projection layer mapping are performed:

[0046]

[0047] Among them, sigmoid is the activation function, Respectively represent the output of the last layer, i.e., the nth layer, of the intra-modal and inter-modal sub-encoders, G v , G e represents the gate vector, Z v , Z e represents the fusion feature after the gating mechanism; v and e represent the final sentiment intensity and category prediction.

[0048] Furthermore, in step S6, the loss L total The calculation is as follows:

[0049] Ltotal =λ1L v +(1-λ1)L e

[0050]

[0051]

[0052] Among them, L v , L e represents the sentiment intensity loss and sentiment polarity classification loss, N l represents the number of labeled samples, represents the true label of sentiment intensity and sentiment polarity, and λ1 is a hyperparameter that balances the two different losses.

[0053] Furthermore, in step S7, the top-k screening strategy is: classify the sentiment categories through sample prediction, and select the k samples with the highest confidence from each category, where k is a hyperparameter.

[0054] Further, in step S8, when repeating steps S2-S6, the sentiment polarity classification loss of the pseudo-labeled sample is increased in step S6, and the loss used in the repeated step S6 stage is as follows:

[0055]

[0056]

[0057] in, Represents the semi-supervised dataset D ′ u The obtained sentiment polarity classification loss, N u represents the number of samples in the semi-supervised dataset, and λ2 is the sentiment classification loss added in the semi-supervised learning stage.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1) In order to solve the problem that existing multimodal sentiment analysis methods are difficult to mine effective modal interaction knowledge at a fine-grained level, the present invention proposes intra-modal and inter-modal attention mask mechanisms that can capture the complementary and differential information between modalities, thereby realizing the mining of effective interaction information.

[0060] 2) This paper introduces a semi-supervised learning method, which further improves the recognition performance (precision and recall) of the existing model by introducing the sentiment knowledge of unlabeled samples, and achieves the best results on relevant open source datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1This is an overall network framework diagram of a multimodal interactive sentiment analysis method based on semi-supervised learning in an embodiment of the present invention;

[0062] Figure 2 is a diagram of an intra-modal / inter-modal encoder architecture in an embodiment of the present invention;

[0063] Figure 3 Implementation diagram of intra-modal and inter-modal masking mechanisms in an embodiment of the present invention;

[0064] Figure 4 This is a graph of emotion recognition results of an embodiment of the present invention on the MOSI dataset;

[0065] Figure 5 This is a graph of emotion recognition results of an embodiment of the present invention on the MOSEI dataset. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical methods in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Embodiment 1:

[0068] A multimodal interactive sentiment analysis method based on semi-supervised learning includes the following steps:

[0069] S1. Based on the open source dataset MOSI, build the training set used in the supervised learning stage Validation set and test set The dataset consists of video clips and emotional labels (including emotional polarity and emotional intensity labels). Each video clip includes an individual's evaluation of a movie or product. At the same time, based on the AMI dataset, an unlabeled dataset D used in the semi-supervised learning stage is constructed. u ;

[0070] S2, extracting features from speech and text through speech encoder and text encoder; visual modality obtains face area through frame-level face detection to reduce the influence of irrelevant background noise, and then uses visual encoder to extract visual features from face area;

[0071] The speech, text and visual modality data are obtained from open source datasets, and for the visual modality data, OpenFace is used to extract the face area.

[0072] The text encoder Roberta-large, speech encoder HuBERT, and visual encoder Fabnet were pre-trained on the datasets in their respective fields, and then the features of their respective modalities were obtained based on the three encoders:

[0073] in, represents the matrix dimension, h t 、h a 、h v Represents text, speech and visual features, l t , l a , l v Represent the length of text, speech and visual features respectively, d t ,d a ,d v Represent the dimensions of text, speech and visual features respectively.

[0074] S3, performing a one-dimensional convolution operation on the extracted speech, text and visual features to obtain local features of speech, text and vision;

[0075] The one-dimensional convolutional layer corresponding to the text and visual modalities includes a single layer of one-dimensional convolution with a kernel size of 3×3 and a step size of 1; the one-dimensional convolutional layer corresponding to the speech modality includes two layers of one-dimensional convolution with a kernel size of 4×4 and a step size of 2 and 4 respectively;

[0076] After one-dimensional convolution operation, the local features of the text are obtained Local features of speech Visual local features The details are as follows:

[0077]

[0078] Among them, k t =3×3, k v =3×3, k a1 =4×4, k a2 =4×4 represents the convolution kernel size of text, vision and speech respectively; l ′ t , l ′ a , l ′ v Represents the length of local features of text, speech and visual modalities, d h Represents the feature dimension of the latent space of the subsequent intra-modal and inter-modal encoders.

[0079] S4, modally encode the local features, then concatenate them with the sentiment polarity classification head and the sentiment intensity classification head along the dimension of feature length, and finally perform position encoding to obtain the input sequence of the intra-modal / inter-modal encoder, as follows:

[0080]

[0081] Among them, Z represents the final input sequence, val,cls represents the sentiment intensity and sentiment polarity classification heads, PE is the position encoding, and all three are learnable parameter matrices, initialized to a truncated normal distribution with a mean of 0 and a variance of 0.02. type , a type , v type are the encoding vectors of text, speech, and visual modalities, initialized to a normal distribution with a mean of 0 and a variance of 0.05.

[0082] The local features of speech, text and visual modalities are modally encoded, that is, they are added to the feature vectors representing the corresponding modal types to indicate which modality the local features belong to. Then, the encoded local features are concatenated with the sentiment intensity classification head and the sentiment polarity classification head along the dimension of feature length. The concatenated result is added to the position vector to encode the position information, and finally the input sequence Z of the intra-modal and inter-modal encoders is obtained. intra and Z inter ;

[0083] The feature vectors, position vectors, sentiment intensity and sentiment polarity classification heads representing the corresponding modal types are all learnable parameter matrices initialized with normal distribution.

[0084] S5, performing feature fusion on the input sequence through the intra-modality encoder and the inter-modality encoder to obtain intra-modality and inter-modality fusion representation;

[0085] like Figure 2 As shown, the intra-modal encoder and the inter-modal encoder are obtained by stacking n layers of intra-modal sub-encoders and inter-modal sub-encoders respectively;

[0086] The intra-modality sub-encoder includes layer normalization (LN), intra-modality attention mask mechanism (IntraMA), layer normalization and fully connected layer (FFN). The input and output flows of the intra-modality sub-encoder are as follows:

[0087]

[0088] in, is the input sequence of the first-layer intramodal sub-encoder, is the output sequence of the sub-encoder within the l+1th modality, l∈[1,n], n represents the total number of layers;

[0089] The inter-modality sub-encoder includes layer normalization, inter-modality attention mask mechanism (InterMA), layer normalization and fully connected layers. The input and output flows of the inter-modality sub-encoder are as follows:

[0090]

[0091]

[0092] in, is the input sequence of the first layer inter-modal sub-encoder, is the output sequence of the l+1th inter-modal sub-encoder, l∈[1,n], and n represents the total number of layers.

[0093] like Figure 3 As shown, the intra-modal / inter-modal attention mask mechanism captures different types of modal interactions by constructing masks for attention at different positions;

[0094] The intra-modality attention mask mechanism is implemented as follows:

[0095]

[0096] in, represents the output sequence of the sub-encoder in the lth layer after layer normalization, that is, represents the output sequence of the intra-modality attention mask mechanism of the intra-modality sub-encoder at the lth layer, W Q , W K , W V represents the learnable projection matrix, Q, K, V represent the query vector, key vector, and value vector obtained after projecting the input; d k represents the latent space dimension; is the intra-modal mask, which represents the masking operation of the modal interaction attention at a specific position. For the element IntraMASK in the i-th row and j-th column in IntraMASK ij , which is implemented as follows:

[0097]

[0098] Among them, Intra pos Represents the position where attention needs to be masked in the intra-modal attention masking mechanism;

[0099] The inter-modal attention mask mechanism is implemented as follows:

[0100]

[0101] in, represents the output sequence of the inter-modal sub-encoder of the lth layer after layer normalization, that is, represents the output sequence of the inter-modal attention mask mechanism of the inter-modal sub-encoder at the lth layer; for the element InterMASK in the i-th row and j-th column of the inter-modal mask InterMASK ij , which is implemented as follows

[0102]

[0103] InterMASK represents the masking operation of the modal interaction attention at a specific location. pos Indicates the position where attention needs to be masked in the inter-modal attention masking mechanism.

[0104] S6. The intra-modal and inter-modal fusion representations are selectively filtered through the gating mechanism, and then mapped through the projection layer to obtain the sentiment intensity and polarity predictions. The two are combined with the sentiment polarity label and intensity label to calculate the loss L total , after gradient optimization, select The best performing one gets the pre-trained model;

[0105] For the intra-modality and inter-modality fusion representation in step S5, in step S6, gated filtering and projection layer mapping are performed:

[0106]

[0107] v=FC v (MLP(Z v ))

[0108] e=FC e (MLP(Z e ))

[0109] Among them, sigmoid is the activation function, Respectively represent the output of the last layer, i.e., the nth layer, of the intra-modal and inter-modal sub-encoders, G v , G e represents the gate vector, Z v , Z e represents the fusion feature after the gating mechanism; v and e represent the final sentiment intensity and category prediction.

[0110] The loss L total The calculation is as follows:

[0111] L total =λ1L v +(1-λ1)L e

[0112]

[0113] Among them, L v , L e represents the sentiment intensity loss and sentiment polarity classification loss, N l represents the number of labeled samples, represents the true label of sentiment intensity and sentiment polarity, and λ1 is a hyperparameter that balances the two different losses.

[0114] S7: Using the pre-trained model obtained in step S6, the unlabeled dataset D constructed in step S1 is trained. u Perform inference, obtain predictions and their confidence, adopt a top-k screening strategy for the predictions, and obtain a semi-supervised dataset including sentiment polarity labels

[0115]

[0116] The top-k screening strategy is: divide the sentiment categories through sample prediction, and select the k samples with the highest confidence from each category. k is a hyperparameter and is set to 40 for the MOSI dataset.

[0117] S8. Integrated Dataset D ′ u With labeled training set Repeat steps S2-S6 to obtain the final emotion recognition model;

[0118] When repeating steps S2-S6, the sentiment polarity classification loss of the pseudo-labeled sample is increased in step S6, and the loss used in the repeated step S6 stage is as follows:

[0119]

[0120]

[0121] in, Represents the semi-supervised dataset D ′ u The obtained sentiment polarity classification loss, N u represents the number of samples in the semi-supervised dataset, and λ2 is the sentiment classification loss added in the semi-supervised learning stage.

[0122] S9: The emotion recognition model obtained in step S8 is used to infer the test set constructed in step S1 This is used to evaluate the model performance.

[0123] In this embodiment, an evaluation is performed on the MOSI dataset to demonstrate the effectiveness of the present invention; the dataset is randomly divided into a training set, a validation set, and a test set.

[0124] In this embodiment, the feature extraction networks of each modality are pre-trained Roberta-large, HuBERT, and Fabnet, and the input batch-size is set to 32. The Adam gradient optimization algorithm is used for optimization in the model, the weight decay is set to 0.0001, the learning rate of the network is set to 0.0001, λ1 and λ2 are set to 0.6 and 0.2, and the unlabeled data set comes from AMI.

[0125] In this embodiment, in order to prove the effectiveness of the multiple methods proposed in the present invention, a variety of different multimodal sentiment analysis methods are compared on the same data set. The main comparison methods are described as follows:

[0126] LMF comes from the literature 1 (see: Liu Z, Shen Y, Lakshminarasimhan VB, et al. Efficient low-rank multimodal fusion with modality-specific factors [J]. arXiv preprintarXiv:1806.00064, 2018;

[0127] TFN comes from literature 2 (see: Zadeh A, Chen M, Poria S, et al. Tensor fusion network for multimodal sentiment analysis [J]. arXiv preprint arXiv:1707.07250, 2017.);

[0128] MFM comes from reference 3 (see: Tsai YHH, Liang PP, Zadeh A, et al. Learning factorized multimodal representations [J]. arXiv preprint arXiv:1806.06176, 2018.);

[0129] Mult is from Reference 4 (see details: Tsai Y H H, Bai S, Liang P P, et al. Multimodal transformer for unaligned multimodal language sequences[C] / / Proceedings of the conference. Association for computational linguistics. Meeting. NIH Public Access, 2019, 2019: 6558.).

[0130] ICCN is from Reference 5 (see details: Sun Z, Sarma P, Sethares W, et al. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(05): 8992 - 8999.).

[0131] MISA is from Reference 6 (see details: Hazarika D, Zimmermann R, Poria S. Misa: Modality - invariant and - specific representations for multimodal sentiment analysis[C] / / Proceedings of the 28th ACM international conference on multimedia. 2020: 1122 - 1131.).

[0132] Self-MM is from Reference 7 (see details: Yu W, Xu H, Yuan Z, et al. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C] / / Proceedings of the AAAI conference on artificial intelligence. 2021, 35(12): 10790-10797.).

[0133] MAG-BERT is from Reference 8 (see details: Rahman W, Hasan M K, Lee S, et al. Integrating multimodal information in large pretrained transformers[C] / / Proceedings of the conference. Association for Computational Linguistics. Meeting. NIH Public Access, 2020, 2020: 2359.).

[0134] MMIM is from Reference 9 (see details: Han W, Chen H, Poria S. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis[J]. arXiv preprint arXiv:2109.00412, 2021.).

[0135] ConFEDE is from Reference 10 (see details: Yang J, Yu Y, Niu D, et al. ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis[C] / / Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers). 2023: 7617-7630.).

[0136] SMIN comes from reference 11 (see: Lian Z, Liu B, Tao J. Smin: Semi-supervised multi-modal interaction network for conversational emotion recognition [J]. IEEE Transactions on Affective Computing, 2022.).

[0137] TCDN comes from literature 12 (see: Chen C, Hong H, Guo J, et al. Inter-intra modal representation augmentation with trimodal collaborative disentanglement network for multimodal sentiment analysis [J]. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2023, 31: 1476-1488.).

[0138] Embodiment 2:

[0139] This embodiment is different from Embodiment 1 in that an evaluation is performed on the MOSEI dataset to demonstrate the effectiveness of the present invention.

[0140] In step S7, the hyperparameter k is set to the size of the unlabeled dataset for the MOSEI dataset.

[0141] The present invention is compared with the prior art based on the benchmark datasets MOSI and MOSEI. MOSI has a total of 2199 videos (1284 videos for training, 229 videos for verification, and 686 videos for testing). The MOSEI dataset is an extended version of the MOSI dataset, containing 22856 video clips (16326 videos for training, 1871 videos for verification, and 4659 videos for testing).

[0142] In order to further verify the technical effects of each module of the present invention, the present invention conducted an ablation experiment on the designed semi-supervised learning and attention mask method to demonstrate the practicality and effectiveness of Semi-IIN.

[0143] The mean absolute error (MAE) is used as an evaluation metric, which takes into account the gap between the sentiment intensity prediction and the true label. The smaller the MAE, the smaller the gap and the more accurate the prediction.

[0144] The accuracy of binary classification (Acc-2) is that the sentiment intensity prediction is divided into seven categories, and the accuracy and F1-score are calculated with the true label. The higher the better.

[0145] Seven-category accuracy (Acc-7), which divides the sentiment intensity prediction into seven categories and calculates the accuracy with the true label.

[0146] Table 1 Comparison results of the present invention with other algorithms on the MOSI dataset

[0147] Methods Embedding MAE Corr ACC-2 F1 Acc-7 LMF Glove 0.917 0.695 - / 82.50 - / 82.40 33.20 TFN Glove 0.901 0.698 - / 80.80 - / 80.70 34.90 MFM Glove 0.877 0.706 - / 81.7 - / 81.6 35.40 Mult Glove 0.861 0.711 81.50 / 84.10 80.60 / 83.90 - ICCN Bert-base 0.862 0.714 - / 83.00 - / 83.00 39.00 MISA Bert-base 0.804 0.764 80.79 / 82.10 80.77 / 82.03 - Self-MM Bert-base 0.713 0.798 84.00 / 85.98 84.42 / 85.95 - MAG-BERT Bert-base 0.712 0.796 84.20 / 86.10 84.10 / 86.00 - MMIM Bert-base 0.700 0.800 84.14 / 86.06 84.00 / 85.98 46.65 ConFEDE Bert-base 0.742 0.784 84.17 / 85.52 84.13 / 85.52 42.27 SMIN Roberta-large - - - / 81.55 - / 81.45 - TCDN Roberta-large 0.697 0.805 - / 87.10 - / 87.20 - Semi-IIN Roberta-large 0.679 0.822 85.28 / 87.04 85.19 / 87.00 46.50

[0148] Table 2 Comparison results of the present invention with other algorithms on the MOSEI dataset

[0149] Methods Embedding MAE Corr ACC-2 F1 Acc-7 LMF Glove 0.623 0.700 - / 82.00 - 48.00 TFN Glove 0.593 0.677 - / 82.50 - / 82.10 50.20 MFM Glove 0.568 0.703 - / 84.40 - / 84.30 51.30 Mult Glove 0.580 0.713 82.50 / 84.23 82.67 / 83.97 - ICCN Bert-base 0.565 0.704 - / 84.20 - / 84.20 51.60 MISA Bert-base 0.568 0.717 82.59 / 84.23 82.67 / 83.97 - Self-MM Bert-base 0.530 0.765 82.81 / 85.17 82.53 / 85.30 - MAG-BERT Bert-base - - 84.70 / - 84.50 / - - MMIM Bert-base 0.526 0.772 82.24 / 85.97 82.66 / 85.94 54.24 ConFEDE Bert-base 0.522 0.780 81.65 / 85.82 82.17 / 85.83 54.86 SMIN Roberta-large - - - / 86.82 - / 86.81 - TCDN Roberta-large 0.521 0.782 - / 87.50 - / 87.20 - Semi-IIN Roberta-large 0.497 0.804 84.98 / 87.70 85.27 / 87.65 55.89

[0150] Table 3 Ablation experiment results of the present invention on the MOSEI dataset

[0151]

[0152] Tables 1 and 2 show the comparison results of the method of the present invention with other methods on the MOSI and MOSEI datasets. It can be seen that the method of the present invention (Semi-IIN) is higher in accuracy, recall and Pearson correlation coefficient than other methods, which shows that the method of the present invention is superior to other methods. Table 3 shows the ablation experiment results of the present invention on the MOSEI dataset, where MA represents the attention mask mechanism, Semi represents the semi-supervised learning method, and Params represents the model parameter quantity. It can be seen from the table that after the introduction of semi-supervised learning and attention mask mechanisms, compared with the baseline (transformer-based method), the model has been improved to a certain extent in all aspects.

[0153] Figure 4 The prediction results of an embodiment of the present invention on the MOSI dataset are shown. Figure 5 The prediction results of an embodiment of the present invention on the MOSEI dataset are shown. From the results of the two embodiments, it can be seen that compared with using only intra-modal or inter-modal interaction information for prediction, the present invention achieves more accurate sentiment intensity prediction results by balancing the weights of the two, further verifying the superiority of the present invention.

[0154] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multimodal interactive sentiment analysis method based on semi-supervised learning, characterized in that: The following steps are involved: S1. Build the training set used in the supervised learning phase based on the open source dataset Validation set and test set At the same time, based on the AMI dataset, an unlabeled dataset D used in the semi-supervised learning stage is constructed. u ; S2, extracting features from speech and text through speech encoder and text encoder; visual modality obtains face area through frame-level face detection, and then uses visual encoder to extract visual features from face area; S3, performing a one-dimensional convolution operation on the extracted speech, text and visual features to obtain local features of speech, text and vision; S4, modality encoding is performed on the local features, and then they are concatenated with the sentiment polarity classification head and the sentiment intensity classification head along the dimension of feature length, and finally position encoding is performed to obtain the input sequence of the intra-modality / inter-modality encoder; S5, performing feature fusion on the input sequence through the intra-modality encoder and the inter-modality encoder to obtain intra-modality and inter-modality fusion representation; S6. The intra-modal and inter-modal fusion representations are selectively filtered through the gating mechanism, and then mapped through the projection layer to obtain the sentiment intensity and polarity predictions. The two are combined with the sentiment polarity label and intensity label to calculate the loss L total , after gradient optimization, select The best performing one gets the pre-trained model; S7: Using the pre-trained model obtained in step S6, the unlabeled dataset D constructed in step S1 is trained. u Perform inference, obtain predictions and their confidence, adopt a top-k screening strategy for the predictions, and obtain a semi-supervised dataset including sentiment polarity labels S8. Integrated Dataset D ′ u With labeled training set Repeat steps S2-S6 to obtain the final emotion recognition model; S9: The emotion recognition model obtained in step S8 is used to infer the test set constructed in step S1 This is used to evaluate the model performance.

2. According to claim 1, a multimodal interactive sentiment analysis method based on semi-supervised learning is characterized in that: In step S2, the speech, text and visual modality data are obtained from open source datasets, and the face area is extracted using OpenFace for the visual modality data; The text encoder Roberta-large, speech encoder HuBERT, and visual encoder Fabnet were pre-trained on the datasets in their respective fields, and then the features of their respective modalities were obtained based on the three encoders: in, represents the matrix dimension, h t 、h a 、h v Represents text, speech and visual features, l t , l a , l v Represent the length of text, speech and visual features respectively, d t d a d v Represent the dimensions of text, speech and visual features respectively.

3. According to the method of multimodal interactive sentiment analysis based on semi-supervised learning in claim 1, it is characterized in that: In step S3, the one-dimensional convolution layer corresponding to the text and visual modalities includes a single layer of one-dimensional convolution with a convolution kernel size of 3×3, and the step size is 1; the one-dimensional convolution layer corresponding to the speech modality includes two layers of one-dimensional convolution with a convolution kernel size of 4×4, and the step sizes are 2 and 4 respectively; After one-dimensional convolution operation, the local features of the text are obtained Local features of speech Visual local features Among them l ′ t , l ′ a , l ′ v Represents the length of local features of text, speech and visual modalities, d h Represents the feature dimension of the latent space of the subsequent intra-modal and inter-modal encoders.

4. According to claim 3, a multimodal interactive sentiment analysis method based on semi-supervised learning is characterized in that: In step S4, the local features of speech, text and visual modalities are modally encoded, that is, they are added to the feature vectors representing the corresponding modal types to indicate which modality the local features belong to. Then, the encoded local features are concatenated with the emotion intensity classification head and the emotion polarity classification head along the dimension of feature length, and the concatenated result is added to the position vector to encode the position information, and finally the input sequence Z of the intra-modal and inter-modal encoders is obtained. intra and Z inter ; The feature vectors, position vectors, sentiment intensity and sentiment polarity classification heads representing the corresponding modal types are all learnable parameter matrices initialized with normal distribution.

5. According to claim 4, a multimodal interactive sentiment analysis method based on semi-supervised learning is characterized in that: The intra-modal encoder and the inter-modal encoder are obtained by stacking n layers of intra-modal sub-encoders and inter-modal sub-encoders respectively; The intra-modality sub-encoder includes layer normalization, intra-modality attention mask mechanism, layer normalization and fully connected layers. The input and output flows of the intra-modality sub-encoder are as follows: in, is the input sequence of the first-layer intra-modal sub-encoder, is the output sequence of the sub-encoder within the l+1th modality, l∈[1,n], n represents the total number of layers; The inter-modal sub-encoder includes layer normalization, inter-modal attention mask mechanism, layer normalization and fully connected layers. The input and output flows of the inter-modal sub-encoder are as follows: in, is the input sequence of the first layer inter-modal sub-encoder, is the output sequence of the l+1th inter-modal sub-encoder, l∈[1,n], and n represents the total number of layers.

6. According to claim 5, a multimodal interactive sentiment analysis method based on semi-supervised learning is characterized in that: In step S5, the intra-modality / inter-modality attention mask mechanism constructs masks for attention at different positions to capture different types of modal interactions; The intra-modality attention mask mechanism is implemented as follows: in, represents the output sequence of the sub-encoder in the lth layer after layer normalization, that is, represents the output sequence of the intra-modality attention mask mechanism of the intra-modality sub-encoder at the lth layer, W Q , W K , W V represents the learnable projection matrix, Q, K, V represent the query vector, key vector, and value vector obtained after projecting the input; d k represents the latent space dimension; is the intra-modal mask, which represents the masking operation of the modal interaction attention at a specific position. For the element IntraMASK in the i-th row and j-th column in IntraMASK ij , which is implemented as follows: Among them, Intra pos Represents the position where attention needs to be masked in the intra-modal attention masking mechanism; The inter-modal attention mask mechanism is implemented as follows: in, represents the output sequence of the inter-modal sub-encoder of the lth layer after layer normalization, that is, represents the output sequence of the inter-modal attention mask mechanism of the inter-modal sub-encoder at the lth layer; for the element InterMASK in the i-th row and j-th column of the inter-modal mask InterMASK ij , which is implemented as follows InterMASK represents the masking operation of the modal interaction attention at a specific location. pos Indicates the position where attention needs to be masked in the inter-modal attention masking mechanism.

7. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 1, characterized in that: For the intra-modality and inter-modality fusion representation in step S5, in step S6, gated filtering and projection layer mapping are performed: v=FC v (MLP(Z v )) e=FC e (MLP(Z e )) Among them, sigmoid is the activation function, Respectively represent the output of the last layer, i.e., the nth layer, of the intra-modal and inter-modal sub-encoders, G v , G e represents the gate vector, Z v , Z e represents the fusion feature after the gating mechanism; v and e represent the final sentiment intensity and category prediction.

8. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 1, characterized in that: In step S6, the loss L total The calculation is as follows: L total =λ1L v +(1-λ1)L e Among them, L v , L e represents the sentiment intensity loss and sentiment polarity classification loss, N l represents the number of labeled samples, represents the true label of sentiment intensity and sentiment polarity, and λ1 is a hyperparameter that balances the two different losses.

9. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 1, characterized in that: In step S7, the top-k screening strategy is: divide the sentiment categories through sample prediction, and select the k samples with the highest confidence from each category, where k is a hyperparameter.

10. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 1, characterized in that: In step S8, when repeating steps S2-S6, the sentiment polarity classification loss of the pseudo-labeled sample is increased in step S6, and the loss used in the repeated step S6 stage is as follows: in, Represents the semi-supervised dataset D ′ u The obtained sentiment polarity classification loss, N u represents the number of samples in the semi-supervised dataset, and λ2 is the sentiment classification loss added in the semi-supervised learning stage.

Images