A multi-modal interactive sentiment analysis method based on semi-supervised learning

Abstract

The application discloses a multimodal interactive sentiment analysis method based on semi-supervised learning. The method first extracts features of voice, text and visual modalities, then fuses the features through two interactive branch networks of intra-modal and inter-modal, and finally adaptively extracts fusion information of the two interactive branches based on a gating mechanism. The intra-modal and inter-modal interactive networks are based on the encoder structure of the transformer, except that the multi-head self-attention layer introduces the proposed attention mask mechanism. In terms of training method, the method generates pseudo-labels to assist model training in a semi-supervised learning manner, thereby reducing the dependence on labeled data. The application solves the problems of high artificial labeling cost and how to mine effective interactive information in the field of multimodal sentiment analysis, and improves the sentiment recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of emotion computing in multimodal learning, relates to the fields of human-computer interaction and artificial intelligence, and specifically relates to a multimodal interaction emotion analysis method based on semi-supervised learning. Background Art

[0002] With the rapid development of social media, people can express their personal opinions and feelings through text and video. Sentiment analysis, a subtask of natural language processing, aims to understand the subject's attitudes or opinions. Traditional sentiment analysis methods primarily focus on the text modality, mining the text's underlying semantic information to analyze the subject's emotional state. Due to the correlation and complementarity between data from different modalities, multimodal sentiment analysis (MSA) has rapidly become a research hotspot. The main task of multimodal sentiment analysis is to accurately identify emotions by comprehensively analyzing information from different modalities (such as text, audio, video, and images). Multimodal sentiment analysis is extremely important in many fields and is widely used in human-computer interaction, public opinion surveys, news video analysis, and healthcare.

[0003] Existing methods can be divided into semi-supervised learning and fully supervised learning methods based on whether the training dataset includes unlabeled data. Semi-supervised learning methods aim to improve model performance by assisting model training with unlabeled data. Liang et al. believed that there is emotional consistency between modalities at the sentence level, and therefore proposed a semi-supervised emotion recognition model based on a cross-modal feature matching strategy to address the high cost of emotion labeling. However, this method does not take into account the contextual information in the conversation. Therefore, Lian et al. used a recurrent network and a hybrid fusion strategy to model the contextual information between modalities, achieving good results on multiple datasets.

[0004] Unlike semi-supervised learning methods, fully supervised learning approaches focus on addressing the difficulties of heterogeneous modal fusion and the explicit and implicit alignment between modalities. Regarding modal alignment, Tsai et al. implemented a cross-modal attention mechanism to enable information exchange from the source modality to the target modality. This method eliminates the need for prior sequence alignment of each modality, addressing the difficulty of manual alignment between modalities due to different sampling rates. Yu et al. used an implicit cross-modal alignment strategy (SPECTRA) to pre-train speech and text conversations. By designing a temporal position prediction task to predict the start and end times of each text word in the speech waveform, they captured the explicit alignment between speech and text. Regarding modal fusion, Pham et al. proposed a multimodal recurrent transformation network (MCTN) that learns a robust joint multimodal intermediate representation through transformations between modalities. This method is advantageous in that it is not affected by modality loss during testing. Rahman et al. embedded a multimodal threshold network into a pre-trained text encoder, incorporating sentiment knowledge from both audio and video modalities into the pre-trained model. Li et al. proposed an enhanced modality awareness network (AMOA), which incorporates global acoustic information knowledge reflecting the overall changes in pitch into the model by constructing a contrastive learning loss function.

[0005] While existing methods have made significant progress in fully supervised or semi-supervised learning, they either fail to effectively filter out irrelevant modality specificity or inter-modality complementary knowledge, or fail to exploit the emotionally significant features in unlabeled data. Therefore, this paper aims to improve the training methods and overall architecture of existing methods to achieve higher emotion recognition accuracy.

[0006] The shortcomings of existing technologies: The semi-supervised learning method used by SMIN requires more unlabeled data and the performance improvement is not obvious. At the same time, this method can only mine the specificity within the modality and the complementary information between modalities at the sentence level. However, studies have shown that (Tsai YHH, Bai S, Liang PP, et al. Multimodal transformer for unaligned multimodal language sequences [C] / / Proceedings of the conference. Association for computational linguistics. Meeting. NIH Public Access, 2019, 2019: 6558. and Yu T, Gao H, Lin TE, et al. Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment [C] / / Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).2023:7900-7913.), capturing interactive information at a fine-grained level (between words within the text modality; between frames of video or speech within the visual or speech modality; between words and video frames between text and visual modalities) helps improve the model's emotion recognition capabilities. Summary of the Invention

[0007] The purpose of this invention is to propose a multimodal interactive sentiment analysis method based on semi-supervised learning. In terms of training, this method generates pseudo-labels from unlabeled datasets through semi-supervised learning. Furthermore, high-confidence samples are screened from the pseudo-labels to assist in model retraining, reducing reliance on labeled data. In terms of model structure, this method constructs intra-modal and inter-modal encoders through the proposed attention mask mechanism, mining effective interaction information within the modality and between different modalities. Subsequently, selective filtering is performed through a gating mechanism to achieve accurate sentiment prediction.

[0008] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0009] A multimodal interactive sentiment analysis method based on semi-supervised learning includes the following steps:

[0010] S1. Build the training set used in the supervised learning phase based on the open source dataset Validation set and test set At the same time, based on the AMI dataset, we construct the unlabeled dataset D used in the semi-supervised learning stage. u ;

[0011] S2. The speech encoder and text encoder are used to extract features from speech and text. The visual modality uses frame-level face detection to obtain the face area, thereby reducing the influence of irrelevant background noise. The visual encoder is then used to extract visual features from the face area.

[0012] S3, performing a one-dimensional convolution operation on the extracted speech, text, and visual features to obtain local features of speech, text, and vision;

[0013] S4. Modal encoding is performed on the local features. Then, along the dimension of feature length, they are concatenated with the sentiment polarity classification head and the sentiment intensity classification head. Finally, position encoding is performed to obtain the input sequence of the intra-modal / inter-modal encoder.

[0014] S5. Perform feature fusion on the input sequence through the intra-modal encoder and the inter-modal encoder to obtain intra-modal and inter-modal fusion representations;

[0015] S6. The intra-modal and inter-modal fusion representations are selectively filtered through the gating mechanism, and then mapped through the projection layer to obtain the emotion intensity and polarity predictions. The two are combined with the emotion polarity label and intensity label to calculate the loss L total , after gradient optimization, select The best performing one gets the pre-trained model;

[0016] S7, using the pre-trained model obtained in step S6, to train the unlabeled dataset D obtained in step S1 u Perform inference, obtain predictions and their confidence, adopt a top-k screening strategy for the predictions, and obtain a semi-supervised dataset including sentiment polarity labels

[0017]

[0018] S8. Integrated Dataset D ′ u With labeled training set Repeat steps S2-S6 to obtain the final emotion recognition model;

[0019] S9: The emotion recognition model obtained in step S8 is used to infer the test set constructed in step S1. This is used to evaluate the model performance.

[0020] Furthermore, in step S2, the speech, text and visual modality data are obtained from open source datasets, and for the visual modality data, the face region is extracted using OpenFace;

[0021] The text encoder Roberta-large, speech encoder HuBERT, and visual encoder Fabnet were pre-trained on datasets from their respective fields. The following features were then derived from the three encoders:

[0022] in, Indicates the matrix dimension, h t 、h a 、h v Represents text, speech and visual features, l t 、l a 、l v Represent the length of text, speech and visual features respectively, d t d a d v Represent the dimensions of text, speech and visual features respectively.

[0023] Furthermore, in step S3, the one-dimensional convolution layer corresponding to the text and visual modalities includes a single layer of one-dimensional convolution with a convolution kernel size of 3×3, and a step size of 1; the one-dimensional convolution layer corresponding to the speech modality includes two layers of one-dimensional convolution with a convolution kernel size of 4×4, and a step size of 2 and 4 respectively;

[0024] After one-dimensional convolution operation, the local features of the text are obtained Local features of speech Visual local features where l ′ t 、l ′ a 、l ′ v Represents the length of local features of text, speech and visual modalities, d h Represents the feature dimension of the latent space of the subsequent intra-modal and inter-modal encoders.

[0025] Furthermore, in step S4, the local features of speech, text and visual modalities are modally encoded respectively, that is, they are added to the feature vectors representing the corresponding modal types to indicate which modality the local features belong to. Then, the encoded local features are concatenated with the emotion intensity classification head and the emotion polarity classification head along the dimension of feature length, and the concatenated result is added to the position vector to encode the position information, and finally the input sequence Z of the intra-modal and inter-modal encoders is obtained. intra and Z inter ;

[0026] The feature vectors, position vectors, sentiment intensity, and sentiment polarity classification heads representing the corresponding modal types are all learnable parameter matrices initialized with a normal distribution.

[0027] Furthermore, the intra-modal encoder and the inter-modal encoder are obtained by stacking n layers of intra-modal sub-encoders and inter-modal sub-encoders respectively;

[0028] The intra-modality sub-encoder includes layer normalization (LN), intra-modality attention mask mechanism (IntraMA), layer normalization and fully connected layer (FFN). The input and output flow of the intra-modality sub-encoder is as follows:

[0029]

[0030] in, is the input sequence of the first-layer intramodal sub-encoder, is the output sequence of the modal sub-encoder in the l+1th layer, l∈[1,n], n represents the total number of layers;

[0031] The inter-modality sub-encoder includes layer normalization, inter-modality attention mask mechanism (InterMA), layer normalization, and fully connected layers. The input and output flows of the inter-modality sub-encoder are as follows:

[0032]

[0033] in, is the input sequence of the first layer inter-modal sub-encoder, is the output sequence of the l+1th inter-modal sub-encoder, l∈[1,n], and n represents the total number of layers.

[0034] Furthermore, in step S5, the intra-modality / inter-modality attention masking mechanism captures different modality interaction types by constructing masks for attention at different positions;

[0035] The intra-modality attention mask mechanism is implemented as follows:

[0036]

[0037] in, represents the output sequence of the modal sub-encoder in the lth layer after layer normalization, that is, represents the output sequence of the intra-modality attention mask mechanism of the l-th layer intra-modality sub-encoder, W Q , W K , W V represents the learnable projection matrix, Q, K, and V represent the query vector, key vector, and value vector obtained after projecting the input; d krepresents the latent space dimension; It is the intra-modal mask, which represents the mask operation on the modal interaction attention at a specific position. For the element IntraMASK in the i-th row and j-th column in IntraMASK, ij , which is implemented as follows:

[0038]

[0039] Among them, Intra pos Represents the location where attention needs to be masked in the intra-modal attention masking mechanism;

[0040] The inter-modal attention mask mechanism is implemented as follows:

[0041]

[0042] in, represents the output sequence of the inter-modal sub-encoder of the lth layer after layer normalization, that is, Represents the output sequence of the inter-modal attention mask mechanism of the inter-modal sub-encoder of the lth layer; for the element InterMASK in the i-th row and j-th column of the inter-modal mask InterMASK ij , which is implemented as follows

[0043]

[0044] InterMASK represents the mask operation of the modal interaction attention at a specific location. pos Indicates the position where attention needs to be masked in the inter-modal attention mask mechanism.

[0045] Furthermore, for the intra-modal and inter-modal fusion representation in step S5, gated filtering and projection layer mapping are performed in step S6:

[0046]

[0047] Among them, sigmoid is the activation function, Represent the output of the last layer, i.e., the nth layer, of the intra-modal and inter-modal sub-encoders, G v , G e represents the gate vector, Z v , Z e represents the fusion feature after the gating mechanism; v and e represent the final sentiment intensity and category prediction.

[0048] Furthermore, in step S6, the loss L total The calculation is as follows:

[0049] Ltotal =λ1L v +(1-λ1)L e

[0050]

[0051]

[0052] Among them, L v , L e represents the sentiment intensity loss and sentiment polarity classification loss, N l represents the number of labeled samples, represents the true label of sentiment intensity and sentiment polarity, and λ1 is a hyperparameter that balances the two different losses.

[0053] Furthermore, in step S7, the top-k screening strategy is: classify the emotion categories through sample prediction, and select the k samples with the highest confidence from each category, where k is a hyperparameter.

[0054] Furthermore, in step S8, when repeating steps S2-S6, the sentiment polarity classification loss of the pseudo-labeled sample is added in step S6, and the loss used in the repeated step S6 stage is as follows:

[0055]

[0056]

[0057] in, Representative based on the semi-supervised dataset D ′ u The obtained sentiment polarity classification loss, N u represents the number of samples in the semi-supervised dataset, and λ2 is the sentiment classification loss added in the semi-supervised learning stage.

[0058] Compared with the existing technology, the beneficial effects of the present invention are:

[0059] 1) In order to address the problem that existing multimodal sentiment analysis methods are difficult to mine effective modal interaction knowledge at a fine-grained level, this paper proposes an intra-modal and inter-modal attention masking mechanism that can capture the complementary and differential information between modalities, thereby realizing the mining of effective interaction information.

[0060] 2) This paper introduces a semi-supervised learning method. By introducing the sentiment knowledge of unlabeled samples, it further improves the recognition performance (precision and recall) of the existing model and achieves the best results on relevant open source datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1This is an overall network framework diagram of a multimodal interactive sentiment analysis method based on semi-supervised learning in an embodiment of the present invention;

[0062] Figure 2 2 is a diagram of an intra-modal / inter-modal encoder architecture in an embodiment of the present invention;

[0063] Figure 3 Implementation diagram of the intra-modal and inter-modal masking mechanisms in an embodiment of the present invention;

[0064] Figure 4 This is a graph showing the emotion recognition results of an embodiment of the present invention on the MOSI dataset;

[0065] Figure 5 This is a diagram of the emotion recognition results of an embodiment of the present invention on the MOSEI dataset. DETAILED DESCRIPTION

[0066] The following will provide a clear and complete description of the technical methods in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0067] Example 1:

[0068] A multimodal interactive sentiment analysis method based on semi-supervised learning includes the following steps:

[0069] S1. Based on the open source dataset MOSI, build the training set used in the supervised learning stage Validation set and test set The dataset consists of video clips and emotional labels (including emotional polarity and emotional intensity labels). Each video clip includes an individual's evaluation of a movie or product. At the same time, based on the AMI dataset, an unlabeled dataset D is constructed for the semi-supervised learning stage. u ;

[0070] S2. The speech encoder and text encoder are used to extract features from speech and text. The visual modality uses frame-level face detection to obtain the face area, thereby reducing the influence of irrelevant background noise. The visual encoder is then used to extract visual features from the face area.

[0071] The speech, text and visual modality data are obtained from open source datasets. For the visual modality data, OpenFace is used to extract the facial area.

[0072] The text encoder Roberta-large, speech encoder HuBERT, and visual encoder Fabnet were pre-trained on datasets from their respective fields. The following features were then derived from the three encoders:

[0073] in, Indicates the matrix dimension, h t 、h a 、h v Represents text, speech and visual features, l t 、l a 、l v Represent the length of text, speech and visual features respectively, d t d a d v Represent the dimensions of text, speech and visual features respectively.

[0074] S3, performing a one-dimensional convolution operation on the extracted speech, text, and visual features to obtain local features of speech, text, and vision;

[0075] The one-dimensional convolution layer corresponding to the text and visual modalities includes a single layer of one-dimensional convolution with a kernel size of 3×3 and a stride of 1. The one-dimensional convolution layer corresponding to the speech modality includes two layers of one-dimensional convolution with a kernel size of 4×4 and strides of 2 and 4 respectively.

[0076] After one-dimensional convolution operation, the local features of the text are obtained Local features of speech Visual local features The details are as follows:

[0077]

[0078] Among them, k t =3×3, k v =3×3, k a1 =4×4, k a2 =4×4 represents the convolution kernel size of text, vision and speech respectively; l ′ t 、l ′ a 、l ′ v Represents the length of local features of text, speech and visual modalities, d h Represents the feature dimension of the latent space of the subsequent intra-modal and inter-modal encoders.

[0079] S4. Modal encoding is performed on the local features. Then, along the dimension of feature length, they are concatenated with the sentiment polarity classification head and the sentiment intensity classification head. Finally, position encoding is performed to obtain the input sequence of the intra-modal / inter-modal encoder, as follows:

[0080]

[0081] Among them, Z represents the final input sequence, val, cls represents the emotion intensity and emotion polarity classification head, PE is the position encoding, and all three are learnable parameter matrices, initialized to a truncated normal distribution with a mean of 0 and a variance of 0.02. type , a type , v type The encoding vectors for text, speech, and visual modalities are initialized to a normal distribution with a mean of 0 and a variance of 0.05.

[0082] The local features of speech, text and visual modalities are modally encoded respectively, that is, they are added to the feature vectors representing the corresponding modal types to indicate which modality the local features belong to. Then, the encoded local features are concatenated with the emotion intensity classification head and the emotion polarity classification head along the dimension of feature length. The concatenated result is added to the position vector to encode the position information, and finally the input sequence Z of the intra-modal and inter-modal encoders is obtained. intra and Z inter ;

[0083] The feature vectors, position vectors, sentiment intensity, and sentiment polarity classification heads representing the corresponding modal types are all learnable parameter matrices initialized with a normal distribution.

[0084] S5. Perform feature fusion on the input sequence through the intra-modal encoder and the inter-modal encoder to obtain intra-modal and inter-modal fusion representations;

[0085] like Figure 2 As shown, the intra-modal encoder and the inter-modal encoder are obtained by stacking n layers of intra-modal sub-encoders and inter-modal sub-encoders respectively;

[0086] The intra-modality sub-encoder includes layer normalization (LN), intra-modality attention mask mechanism (IntraMA), layer normalization and fully connected layer (FFN). The input and output flow of the intra-modality sub-encoder is as follows:

[0087]

[0088] in, is the input sequence of the first-layer intramodal sub-encoder, is the output sequence of the modal sub-encoder in the l+1th layer, l∈[1,n], n represents the total number of layers;

[0089] The inter-modality sub-encoder includes layer normalization, inter-modality attention mask mechanism (InterMA), layer normalization, and fully connected layers. The input and output flows of the inter-modality sub-encoder are as follows:

[0090]

[0091]

[0092] in, is the input sequence of the first layer intermodal sub-encoder, is the output sequence of the l+1th inter-modal sub-encoder, l∈[1,n], and n represents the total number of layers.

[0093] like Figure 3 As shown in Figure 2, the intra-modal / inter-modal attention mask mechanism captures different types of modal interactions by constructing masks for attention at different positions.

[0094] The intra-modality attention mask mechanism is implemented as follows:

[0095]

[0096] in, represents the output sequence of the modal sub-encoder in the lth layer after layer normalization, that is, represents the output sequence of the intra-modality attention mask mechanism of the l-th layer intra-modality sub-encoder, W Q , W K , W V represents the learnable projection matrix, Q, K, and V represent the query vector, key vector, and value vector obtained after projecting the input; d k represents the latent space dimension; It is the intra-modal mask, which represents the mask operation on the modal interaction attention at a specific position. For the element IntraMASK in the i-th row and j-th column in IntraMASK, ij , which is implemented as follows:

[0097]

[0098] Among them, Intra pos Represents the location where attention needs to be masked in the intra-modal attention masking mechanism;

[0099] The inter-modal attention mask mechanism is implemented as follows:

[0100]

[0101] in, represents the output sequence of the inter-modal sub-encoder of the lth layer after layer normalization, that is, Represents the output sequence of the inter-modal attention mask mechanism of the inter-modal sub-encoder of the lth layer; for the element InterMASK in the i-th row and j-th column of the inter-modal mask InterMASK ij , which is implemented as follows

[0102]

[0103] InterMASK represents the mask operation of the modal interaction attention at a specific location. pos Indicates the position where attention needs to be masked in the inter-modal attention mask mechanism.

[0104] S6. The intra-modal and inter-modal fusion representations are selectively filtered through the gating mechanism, and then mapped through the projection layer to obtain the emotion intensity and polarity predictions. The two are combined with the emotion polarity label and intensity label to calculate the loss L total , after gradient optimization, select The best performing one gets the pre-trained model;

[0105] For the intra-modal and inter-modal fusion representation in step S5, gated filtering and projection layer mapping are performed in step S6:

[0106]

[0107] v=FC v (MLP(Z v ))

[0108] e=FC e (MLP(Z e ))

[0109] Among them, sigmoid is the activation function, Represent the output of the last layer, i.e., the nth layer, of the intra-modal and inter-modal sub-encoders, G v , G e represents the gate vector, Z v , Z e represents the fusion feature after the gating mechanism; v and e represent the final sentiment intensity and category prediction.

[0110] The loss L total The calculation is as follows:

[0111] L total =λ1L v +(1-λ1)L e

[0112]

[0113] Among them, L v , L e represents the sentiment intensity loss and sentiment polarity classification loss, N l represents the number of labeled samples, represents the true label of sentiment intensity and sentiment polarity, and λ1 is a hyperparameter that balances the two different losses.

[0114] S7, using the pre-trained model obtained in step S6, to train the unlabeled dataset D constructed in step S1 u Perform inference, obtain predictions and their confidence, adopt a top-k screening strategy for the predictions, and obtain a semi-supervised dataset including sentiment polarity labels

[0115]

[0116] The top-k screening strategy is to classify sentiment categories through sample prediction and select the k samples with the highest confidence from each category. k is a hyperparameter and is set to 40 for the MOSI dataset.

[0117] S8. Integrated Dataset D ′ u With labeled training set Repeat steps S2-S6 to obtain the final emotion recognition model;

[0118] When repeating steps S2-S6, the sentiment polarity classification loss of the pseudo-labeled sample is increased in step S6, and the loss used in the repeated step S6 stage is as follows:

[0119]

[0120]

[0121] in, Representative based on the semi-supervised dataset D ′ u The obtained sentiment polarity classification loss, N u represents the number of samples in the semi-supervised dataset, and λ2 is the sentiment classification loss added in the semi-supervised learning stage.

[0122] S9: The emotion recognition model obtained in step S8 is used to infer the test set constructed in step S1. This is used to evaluate the model performance.

[0123] In this embodiment, an evaluation is performed on the MOSI dataset to demonstrate the effectiveness of the present invention; the dataset is randomly divided into a training set, a validation set, and a test set.

[0124] In this embodiment, the feature extraction networks of each modality are pre-trained Roberta-large, HuBERT, and Fabnet, and the input batch-size is set to 32. The Adam gradient optimization algorithm is used for optimization in the model, the weight decay is set to 0.0001, the network learning rate is set to 0.0001, λ1 and λ2 are set to 0.6 and 0.2, and the unlabeled dataset comes from AMI.

[0125] In this example, to demonstrate the effectiveness of the proposed method, a variety of multimodal sentiment analysis methods were compared on the same dataset. The main comparison methods are described as follows:

[0126] LMF comes from the literature 1 (see: Liu Z, Shen Y, Lakshminarasimhan VB, et al. Efficient low-rank multimodal fusion with modality-specific factors [J]. arXiv preprint arXiv:1806.00064, 2018;

[0127] TFN comes from literature 2 (see: Zadeh A, Chen M, Poria S, et al. Tensor fusion network for multimodal sentiment analysis [J]. arXiv preprint arXiv:1707.07250, 2017.);

[0128] MFM comes from reference 3 (see: Tsai YHH, Liang PP, Zadeh A, et al. Learning factorized multimodal representations [J]. arXiv preprint arXiv:1806.06176, 2018.);

[0129] Mult is from Reference 4 (see details: Tsai Y H H, Bai S, Liang P P, et al. Multimodal transformer for unaligned multimodal language sequences[C] / / Proceedings of the conference. Association for computational linguistics. Meeting. NIH Public Access, 2019, 2019: 6558.).

[0130] ICCN is from Reference 5 (see details: Sun Z, Sarma P, Sethares W, et al. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(05): 8992 - 8999.).

[0131] MISA is from Reference 6 (see details: Hazarika D, Zimmermann R, Poria S. Misa: Modality - invariant and - specific representations for multimodal sentiment analysis[C] / / Proceedings of the 28th ACM international conference on multimedia. 2020: 1122 - 1131.).

[0132] Self-MM is from Reference 7 (see details: Yu W, Xu H, Yuan Z, et al. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C] / / Proceedings of the AAAI conference on artificial intelligence. 2021, 35(12): 10790-10797.).

[0133] MAG-BERT is from Reference 8 (see details: Rahman W, Hasan M K, Lee S, et al. Integrating multimodal information in large pretrained transformers[C] / / Proceedings of the conference. Association for Computational Linguistics. Meeting. NIH Public Access, 2020, 2020: 2359.).

[0134] MMIM is from Reference 9 (see details: Han W, Chen H, Poria S. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis[J]. arXiv preprint arXiv:2109.00412, 2021.).

[0135] ConFEDE is from Reference 10 (see details: Yang J, Yu Y, Niu D, et al. ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis[C] / / Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers). 2023: 7617-7630.).

[0136] SMIN comes from reference 11 (see: Lian Z, Liu B, Tao J. Smin: Semi-supervised multi-modal interaction network for conversational emotion recognition [J]. IEEE Transactions on Affective Computing, 2022.).

[0137] TCDN comes from literature 12 (for details, see: Chen C, Hong H, Guo J, et al. Inter-intra modal representation augmentation with trimodal collaborative disentanglement network for multimodal sentiment analysis [J]. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2023, 31: 1476-1488.).

[0138] Example 2:

[0139] This embodiment is different from Embodiment 1 in that an evaluation is performed on the MOSEI dataset to demonstrate the effectiveness of the present invention.

[0140] In step S7, the hyperparameter k is set to the size of the unlabeled dataset for the MOSEI dataset.

[0141] The present invention is compared with previous techniques based on the benchmark datasets MOSI and MOSEI. MOSI contains 2,199 videos (1,284 for training, 229 for validation, and 686 for testing). The MOSEI dataset is an expanded version of the MOSI dataset, containing 22,856 video clips (16,326 for training, 1,871 for validation, and 4,659 for testing).

[0142] In order to further verify the technical effects of the various modules of the present invention, the present invention conducted ablation experiments on the designed semi-supervised learning and attention mask methods to demonstrate the practicality and effectiveness of Semi-IIN.

[0143] The mean absolute error (MAE) is used as an evaluation metric, which takes into account the gap between the sentiment intensity prediction and the true label. The smaller the MAE, the smaller the gap and the more accurate the prediction.

[0144] The accuracy of the second classification (Acc-2) is calculated by dividing the sentiment intensity prediction into seven categories and comparing it with the true label. The higher the accuracy, the better.

[0145] Seven-category accuracy (Acc-7) is to divide the sentiment intensity prediction into seven categories and calculate the accuracy with the true label.

[0146] Table 1 Comparison results of the present invention with other algorithms on the MOSI dataset

[0147] Methods Embedding MAE Corr ACC-2 F1 Acc-7 LMF Glove 0.917 0.695 - / 82.50 - / 82.40 33.20 TFN Glove 0.901 0.698 - / 80.80 - / 80.70 34.90 MFM Glove 0.877 0.706 - / 81.7 - / 81.6 35.40 Mult Glove 0.861 0.711 81.50 / 84.10 80.60 / 83.90 - ICCN Bert-base 0.862 0.714 - / 83.00 - / 83.00 39.00 MISA Bert-base 0.804 0.764 80.79 / 82.10 80.77 / 82.03 - Self-MM Bert-base 0.713 0.798 84.00 / 85.98 84.42 / 85.95 - MAG-BERT Bert-base 0.712 0.796 84.20 / 86.10 84.10 / 86.00 - MMIM Bert-base 0.700 0.800 84.14 / 86.06 84.00 / 85.98 46.65 ConFEDE Bert-base 0.742 0.784 84.17 / 85.52 84.13 / 85.52 42.27 SMIN Roberta-large - - - / 81.55 - / 81.45 - TCDN Roberta-large 0.697 0.805 - / 87.10 - / 87.20 - Semi-IIN Roberta-large 0.679 0.822 85.28 / 87.04 85.19 / 87.00 46.50

[0148] Table 2 Comparison results of the present invention with other algorithms on the MOSEI dataset

[0149] Methods Embedding MAE Corr ACC-2 F1 Acc-7 LMF Glove 0.623 0.700 - / 82.00 - 48.00 TFN Glove 0.593 0.677 - / 82.50 - / 82.10 50.20 MFM Glove 0.568 0.703 - / 84.40 - / 84.30 51.30 Mult Glove 0.580 0.713 82.50 / 84.23 82.67 / 83.97 - ICCN Bert-base 0.565 0.704 - / 84.20 - / 84.20 51.60 MISA Bert-base 0.568 0.717 82.59 / 84.23 82.67 / 83.97 - Self-MM Bert-base 0.530 0.765 82.81 / 85.17 82.53 / 85.30 - MAG-BERT Bert-base - - 84.70 / - 84.50 / - - MMIM Bert-base 0.526 0.772 82.24 / 85.97 82.66 / 85.94 54.24 ConFEDE Bert-base 0.522 0.780 81.65 / 85.82 82.17 / 85.83 54.86 SMIN Roberta-large - - - / 86.82 - / 86.81 - TCDN Roberta-large 0.521 0.782 - / 87.50 - / 87.20 - Semi-IIN Roberta-large 0.497 0.804 84.98 / 87.70 85.27 / 87.65 55.89

[0150] Table 3 Ablation experiment results of the present invention on the MOSEI dataset

[0151]

[0152] Tables 1 and 2 show the comparison results of the proposed method with other methods on the MOSI and MOSEI datasets. It can be seen that the proposed method (Semi-IIN) has higher precision, recall, and Pearson correlation coefficient than other methods, which shows that the proposed method is superior to other methods. Table 3 shows the ablation experiment results of the proposed method on the MOSEI dataset, where MA represents the attention mask mechanism, Semi represents the semi-supervised learning method, and Params represents the number of model parameters. As can be seen from the table, after introducing semi-supervised learning and the attention mask mechanism, the model has improved to a certain extent in all aspects compared to the baseline (transformer-based method).

[0153] Figure 4 The prediction results of an embodiment of the present invention on the MOSI dataset are shown. Figure 5 The prediction results of an embodiment of the present invention on the MOSEI dataset are demonstrated. From the results of these two embodiments, it can be seen that compared with using only intra-modal or inter-modal interaction information for prediction, the present invention achieves more accurate sentiment intensity prediction results by balancing the weights of the two, further verifying the superiority of the present invention.

[0154] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multimodal interactive sentiment analysis method based on semi-supervised learning, characterized in that: The following steps are involved: S1. Build the training set used in the supervised learning phase based on the open source dataset Validation set and test set At the same time, based on the AMI dataset, we construct the unlabeled dataset D used in the semi-supervised learning stage. u ; S2: The speech encoder and text encoder are used to extract features from speech and text. The visual modality uses frame-level face detection to obtain the face area, and then uses the visual encoder to extract visual features from the face area. S3, performing a one-dimensional convolution operation on the extracted speech, text, and visual features to obtain local features of speech, text, and vision; S4. Modal encoding is performed on the local features. Then, along the dimension of feature length, they are concatenated with the sentiment polarity classification head and the sentiment intensity classification head. Finally, position encoding is performed to obtain the input sequence of the intra-modal / inter-modal encoder. S5. Perform feature fusion on the input sequence through the intra-modal encoder and the inter-modal encoder to obtain intra-modal and inter-modal fusion representations; S6. The intra-modal and inter-modal fusion representations are selectively filtered through the gating mechanism, and then mapped through the projection layer to obtain the emotion intensity and polarity predictions. The two are combined with the emotion polarity label and intensity label to calculate the loss L total , after gradient optimization, select The best performing one gets the pre-trained model; S7, using the pre-trained model obtained in step S6, to train the unlabeled dataset D obtained in step S1 u Perform inference, obtain predictions and their confidence, adopt a top-k screening strategy for the predictions, and obtain a semi-supervised dataset including sentiment polarity labels S8. Integrated Dataset D ′ u With labeled training set Repeat steps S2-S6 to obtain the final emotion recognition model; S9: The emotion recognition model obtained in step S8 is used to infer the test set constructed in step S1. This is used to evaluate the model performance.

2. A multimodal interactive sentiment analysis method based on semi-supervised learning according to claim 1, characterized in that: In step S2, the speech, text and visual modality data are obtained from open source datasets, and the facial region is extracted using OpenFace for the visual modality data; The text encoder Roberta-large, speech encoder HuBERT, and visual encoder Fabnet were pre-trained on datasets from their respective fields. The following features were then derived from the three encoders: in, Indicates the matrix dimension, h t 、h a 、h v Represents text, speech and visual features, l t 、l a 、l v Represent the length of text, speech and visual features respectively, d t d a d v Represent the dimensions of text, speech and visual features respectively.

3. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 1 is characterized in that: In step S3, the one-dimensional convolution layer corresponding to the text and visual modalities includes a single layer of one-dimensional convolution with a convolution kernel size of 3×3 and a stride of 1; the one-dimensional convolution layer corresponding to the speech modality includes two layers of one-dimensional convolution with a convolution kernel size of 4×4 and a stride of 2 and 4 respectively; After one-dimensional convolution operation, the local features of the text are obtained Local features of speech Visual local features where l ′ t 、l ′ a 、l ′ v Represents the length of local features of text, speech and visual modalities, d h Represents the feature dimension of the latent space of the subsequent intra-modal and inter-modal encoders.

4. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 3 is characterized in that: In step S4, the local features of speech, text and visual modalities are modally encoded respectively, that is, they are added to the feature vectors representing the corresponding modal types to indicate which modality the local features belong to. Then, the encoded local features are concatenated with the emotion intensity classification head and the emotion polarity classification head along the dimension of feature length. The concatenated result is added to the position vector to encode the position information, and finally the input sequence Z of the intra-modal and inter-modal encoders is obtained. intra and Z inter ; The feature vectors, position vectors, sentiment intensity, and sentiment polarity classification heads representing the corresponding modal types are all learnable parameter matrices initialized with a normal distribution.

5. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 4 is characterized in that: The intra-modal encoder and the inter-modal encoder are obtained by stacking n layers of intra-modal sub-encoders and inter-modal sub-encoders respectively; The intra-modality sub-encoder includes layer normalization, intra-modality attention mask mechanism, layer normalization and fully connected layers. The input and output flow of the intra-modality sub-encoder is as follows: in, is the input sequence of the first-layer modal intra-encoder, is the output sequence of the modal sub-encoder in the l+1th layer, l∈[1,n], n represents the total number of layers; The inter-modal sub-encoder includes layer normalization, inter-modal attention mask mechanism, layer normalization and fully connected layers. The input and output flow of the inter-modal sub-encoder is as follows: in, is the input sequence of the first layer inter-modal sub-encoder, is the output sequence of the l+1th inter-modal sub-encoder, l∈[1,n], and n represents the total number of layers.

6. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 5 is characterized in that: In step S5, the intra-modal / inter-modal attention masking mechanism captures different modal interaction types by constructing masks for attention at different positions; The intra-modality attention mask mechanism is implemented as follows: in, represents the output sequence of the modal sub-encoder in the lth layer after layer normalization, that is, represents the output sequence of the intra-modality attention mask mechanism of the l-th layer intra-modality sub-encoder, W Q , W K , W V represents the learnable projection matrix, Q, K, and V represent the query vector, key vector, and value vector obtained after projecting the input; d k represents the latent space dimension; It is the intra-modal mask, which represents the mask operation on the modal interaction attention at a specific position. For the element IntraMASK in the i-th row and j-th column in IntraMASK, ij , which is implemented as follows: Among them, Intra pos Represents the location where attention needs to be masked in the intra-modal attention masking mechanism; The inter-modal attention mask mechanism is implemented as follows: in, represents the output sequence of the inter-modal sub-encoder of the lth layer after layer normalization, that is, Represents the output sequence of the inter-modal attention mask mechanism of the inter-modal sub-encoder of the lth layer; for the element InterMASK in the i-th row and j-th column of the inter-modal mask InterMASK ij , which is implemented as follows InterMASK represents the mask operation of the modal interaction attention at a specific location. pos Indicates the position where attention needs to be masked in the inter-modal attention mask mechanism.

7. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 1, characterized in that: For the intra-modal and inter-modal fusion representation in step S5, gated filtering and projection layer mapping are performed in step S6: v=FC v (MLP(Z v )) e=FC e (MLP(Z e )) Among them, sigmoid is the activation function, Represent the output of the last layer, i.e., the nth layer, of the intra-modal and inter-modal sub-encoders, G v , G e represents the gate vector, Z v , Z e represents the fusion feature after the gating mechanism; v and e represent the final sentiment intensity and category prediction.

8. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 1, characterized in that: In step S6, the loss L total The calculation is as follows: L total =λ1L v +(1-λ1)L e Among them, L v , L e represents the sentiment intensity loss and sentiment polarity classification loss, N l represents the number of labeled samples, represents the true label of sentiment intensity and sentiment polarity, and λ1 is a hyperparameter that balances the two different losses.

9. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 1, characterized in that: In step S7, the top-k screening strategy is: divide the emotion categories through sample prediction, and select the k samples with the highest confidence from each category, where k is a hyperparameter.

10. The method for multimodal interactive sentiment analysis based on semi-supervised learning according to claim 1, characterized in that: In step S8, when repeating steps S2-S6, the sentiment polarity classification loss of the pseudo-labeled sample is added in step S6, and the loss used in the repeated step S6 stage is as follows: in, Representative based on the semi-supervised dataset D ′ u The obtained sentiment polarity classification loss, N u represents the number of samples in the semi-supervised dataset, and λ2 is the sentiment classification loss added in the semi-supervised learning stage.

Images