Multimodal sentiment analysis method, system, device and medium based on multivariate loss function

Abstract

The application discloses a multi-modal sentiment analysis method, system, device and medium based on a multi-element loss function, and relates to the technical field of sentiment analysis. The method comprises the following steps: data preprocessing, model construction, model training and optimization, and task migration. The system, device and medium are used for realizing the multi-modal sentiment analysis method based on the multi-element loss function. The application has the characteristics of improving data processing efficiency and fusion efficiency, maximizing the use of correlation information, reducing noise interference, and improving model accuracy, migration and robustness by using an encoder based on a cross-attention mechanism to perform modal fusion, using a contrast learning framework to perform auxiliary training of a multi-element loss function, and setting different migration functions to migrate different downstream tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sentiment analysis, in particular to a multi-modal sentiment analysis method, system, device and medium based on a multi-element loss function. BACKGROUND

[0002] With the rapid development of artificial intelligence and machine learning technology, sentiment analysis has gradually become an important research direction in the field of natural language processing and computer vision. Sentiment analysis can provide valuable help in many fields, for example, improving intelligent interaction capability, sentiment analysis can better help understand human emotional needs, thereby improving intelligent interaction effect and enhancing intelligent system customization capability. By performing sentiment analysis on social media life data, artificial intelligence systems can provide personalized customization and point-to-point service recommendations in different situations, thereby improving user experience and enhancing medical and health management efficiency. Sentiment analysis can help doctors better understand the emotional state of patients in order to better diagnose and treat psychological problems and emotional disorders and improve the quality of life of patients. However, existing sentiment analysis technology still has many challenges, first, how to process multi-modal data, considering different modalities such as text, speech and image, and fuse them together for sentiment analysis; second, sentiment analysis also faces the problems of difficult data labeling, insufficient model robustness, poor model interpretability, etc.; finally, with the diversification of sentiment analysis application scenarios, higher requirements are put forward for the generalization of the algorithm.

[0003] The patent application with the title of "An intelligent sentiment recognition system based on multi-modal and context", application number [CN202211695431.5], discloses an intelligent sentiment recognition system based on multi-modal and context, the specific steps of the method are as follows: collecting the features of multiple modalities through a multi-modal feature collection module, generating a sentiment vector according to the collected features, performing fusion, and then modeling according to the context to obtain a prediction result.

[0004] The patent application with the title of "Multi-task learning and improved multi-modal sentiment analysis method based on contrast learning", application number [CN202211584630.9], discloses a multi-modal sentiment analysis method based on multi-task learning and improved contrast learning. The steps of sentiment recognition in the method mainly include: first, single-modal representation is performed on MOSI data, and single-modal sentiment polarity prediction is performed according to the obtained representation vector; then, multiple single-modal representations are spliced, and double-modal sentiment polarity prediction is obtained through a gating module; finally, the mean square error of any two outputs is taken as the contrast learning loss, the loss of the main task and the two groups of subtasks is taken as the multi-task learning loss, the loss weight coefficients of different tasks in the multi-task learning loss are adaptively adjusted through homoscedastic uncertainty, the weighted sum of the multi-task learning loss and the contrast learning loss is taken as the final loss function of the model for training, and the final network model is obtained.

[0005] However, the prior art has the following problems:

[0006] (1) In the prior art, the multi-modal network structure is complex, resulting in an overly bloated model and overly complex extracted information, which cannot establish effective timing and correlation. Image features and word labels are embedded in their own space, which makes it challenging for multi-modal encoders to learn to model their interactions;

[0007] (2) The multi-modal sentiment recognition scheme used in the prior art mainly focuses on the feature extraction stage. Whether it is decision-level, feature-level or data-level fusion, the proportion of its fusion framework in the overall model structure is very low. Although increasing the proportion of the feature extraction network maximizes the retention of sentiment-related information to some extent, the heterogeneity between modalities makes simple fusion schemes insufficient to extract multi-modal sentiment information, resulting in poor fusion results;

[0008] (3) The multi-modal fusion scheme used in the prior art requires both data sets with bounding box annotations and high resolution data. Because the model needs boundary box annotations during pre-training and high-resolution images during training, the current mainstream multi-modal data sets come from the network, and the data itself contains different levels of noise. The existing pre-training scheme, such as MLM, overfits to noisy text, resulting in poor generalization performance of the model;

[0009] (4) The multi-modal sentiment recognition scheme used in the prior art mainly identifies a single task according to the data set, which cannot adapt to complex downstream tasks. At the same time, the model's learning scheme mainly trains joint visual labels and word labels, but due to the different processing processes of the two, the model is prone to local optimal solution, thus failing to achieve effective training. SUMMARY

[0010] In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a multimodal sentiment analysis method, system, device and medium based on a multi-element loss function, which has the characteristics of improving data processing efficiency and fusion efficiency, maximizing the use of correlation information, reducing noise interference, and improving model accuracy, transferability and robustness by using a cross-attention mechanism-based encoder for modal fusion and using a contrast learning framework for auxiliary training of a multi-element loss function while setting different transfer functions to transfer different downstream tasks.

[0011] In order to achieve the above-mentioned purpose, the technical solutions adopted by the present application are as follows:

[0012] The multimodal sentiment analysis method based on a multi-element loss function comprises the following steps:

[0013] Step 1, data preprocessing: obtaining data of multiple modalities, preprocessing the data of multiple modalities, and dividing the preprocessed data into a training set, a test set and a validation set, wherein the data of multiple modalities includes text data, audio data and video data;

[0014] Step 2, model construction: constructing a multimodal sentiment analysis model based on a multi-element loss function, wherein the multimodal sentiment analysis model based on a multi-element loss function comprises a feature extraction encoder and a data fusion encoder;

[0015] Step 3, model training and optimization: combining the data in the training set of step 1 with a multi-element loss function and a contrast learning mechanism module to generate corresponding positive and negative sample pairs, inputting the positive and negative sample pairs into the multimodal sentiment analysis model based on a multi-element loss function constructed in step 2, training and optimizing the feature extraction encoder and the data fusion encoder, and obtaining an optimal multimodal sentiment analysis model based on a multi-element loss function;

[0016] Step 4, task transfer: using a contrast learning to construct two different transfer functions to process the multimodal sentiment feature values obtained in step 3, obtain different sentiment values, and transfer different downstream tasks.

[0017] The specific process of step 1 is as follows:

[0018] Step 1.1: dividing the data according to a predetermined time sequence, taking the text data as the alignment reference, and obtaining corresponding audio segments and video frames;

[0019] Step 1.2: dividing the text data in step 1.1 into word vectors and adding the corresponding position code to the end of the vector; converting the audio data of the audio segment into corresponding mel spectrum; converting the video data of the video frame into corresponding size and format;

[0020] Step 1.3: Align the processed audio data, video data, and text data, and divide them into a training set, a test set, and a validation set.

[0021] The feature extraction encoder in step 2 includes a text sentiment feature extraction encoder, a video sentiment feature extraction encoder, and an audio sentiment feature extraction encoder.

[0022] The text sentiment feature extraction encoder is a multi-layer multi-head attention mechanism encoder, based on a module of a stacked multi-layer Transformer encoder architecture.

[0023] The video sentiment feature extraction encoder is a multi-layer self-attention mechanism encoder, based on a module of a stacked multi-layer Vision Transformer encoder architecture.

[0024] The audio sentiment feature extraction encoder is a multi-layer self-attention mechanism encoder, based on a module of a stacked multi-layer self-attention mechanism forward network encoder.

[0025] The data fusion encoder in step 2 is a multi-layer cross-attention mechanism-based encoder, based on a module of a stacked cross-attention mechanism encoder of a specific number of layers.

[0026] The specific process of step 2 is as follows:

[0027] Step 2.1: Input the preprocessed text data from step 1 into the text sentiment feature extraction encoder to extract text sentiment features.

[0028] Step 2.2: Input the preprocessed video data from step 1 into the video sentiment feature extraction encoder to extract video sentiment features.

[0029] Step 2.3: Input the preprocessed audio data from step 1 into the audio sentiment feature extraction encoder to extract audio sentiment features.

[0030] Step 2.4: Input the extracted text sentiment features, video sentiment features, and audio sentiment features into the data fusion encoder for feature fusion to obtain a multi-modal sentiment analysis model based on a multi-element loss function.

[0031] The specific process of step 3 is as follows:

[0032] Step 3.1: Perform data augmentation on the calculation sequence generated during the data preprocessing process in step 1, specifically, match the data of the three modalities in the training set two by two between samples to form positive and negative sample pairs.

[0033] Step 3.2: Shuffle the data packets after data augmentation in step 3.1 to form different positive and negative sample pairs;

[0034] Step 3.3: With the text data as the alignment reference, input the positive and negative sample pairs obtained in step 3.2 into the corresponding feature extraction encoder constructed in step 2 to extract the corresponding features, specifically:

[0035] Use the contrast learning mechanism combined with the ITC_LOSS loss function to extract video emotional features and text emotional features;

[0036] Use the contrast learning mechanism combined with the InfoNCE_LOSS loss function to extract audio emotional features and text emotional features;

[0037] Step 3.4: Optimize the feature extraction encoder using the two contrast loss functions of the positive and negative sample pairs in step 3.3;

[0038] Step 3.5: Fuse the text emotional features, video emotional features, and audio emotional features extracted in step 3.3 through the data fusion encoder to obtain the fused multi-modal emotional feature values, and use the InfoNCE_LOSS loss function for regression training;

[0039] Step 3.6: Repeat steps 3.3-3.5 to train and optimize the multi-modal emotional analysis model based on the multi-element loss function, and obtain the optimal multi-modal emotional analysis model based on the multi-element loss function.

[0040] The specific process of step 4 is as follows:

[0041] Step 4.1: Divide the emotional state into absolute emotion and relative emotion;

[0042] Step 4.2: Set the absolute emotion transfer function and the relative emotion transfer function according to the two emotional states divided in step 4.1 to process the fused multi-modal emotional feature values obtained by the optimal multi-modal emotional analysis model based on the multi-element loss function in step 3;

[0043] The absolute emotion transfer function is:

[0044]

[0045] where n is the number of sequence samples, v i is the i-th sample, e is the bias threshold, A n is the absolute emotional binary state;

[0046] The relative emotion transfer function is:

[0047]

[0048] where n is the number of sequence samples, v i is the i-th sample, v i-1 is the (i-1)-th sample, R n is the relative emotional binary state.

[0049] Step 4.3: Normalize the result obtained in step 4.2 through a softmax layer to obtain the discrete value of the absolute emotion and the continuous value of the relative emotion.

[0050] The ITC_LOSS loss function in the step 3.3 is specifically as follows:

[0051] L ITC = aL triplet + (1-a)L contrastive

[0052] where L triplet is a Triplet loss function, L contrastive is a Contrastive loss function, and a is a weight coefficient of the loss function, used to balance the influence of the two loss functions.

[0053] The L triplet loss function and the L contrastive loss function are as follows:

[0054]

[0055] where N is the number of categories in the training set, P i is the number of samples in the i-th category, x i a is the a-th sample in the i-th category, (x i a ,x i p ) is the similarity between the samples x i a and x i p , S(x i a ,x i n ) is the similarity between the sample x i n and the sample x i a , and m is a hyperparameter used to control the distance between samples of the same category.

[0056]

[0057]

[0058] where d(x i a ,x i p ) is the distance between samples x i a and x i p , y i a is the class label to which sample x i a belongs, y different is the loss value of different types of samples, and m is a hyperparameter used to control the distance between samples of the same class;

[0059] The ITC_LOSS loss function in step 3.3 is specifically:

[0060]

[0061] where N is the number of samples in a training batch, z i is the feature extracted from sample i, z j is the feature extracted from sample j, sim(z i ,z k ) is the calculation formula of cosine similarity (sim(u,v)=u T v / ||u||·||v||), 1[k≠i], and τ is the temperature coefficient.

[0062] The multi-modal sentiment analysis system based on a multi-element loss function comprises:

[0063] A data preprocessing module: obtaining data of multiple modalities and preprocessing the data of multiple modalities;

[0064] A model construction module: constructing a multi-modal sentiment analysis model based on a multi-element loss function, wherein the multi-modal sentiment analysis model based on a multi-element loss function comprises a feature extraction encoder and a data fusion encoder;

[0065] A model training and optimization module: inputting the preprocessed data into the multi-modal sentiment analysis model based on a multi-element loss function, and generating corresponding positive and negative samples by using a multi-element loss function in combination with a contrastive learning mechanism module, training and optimizing the feature extraction encoder and the data fusion encoder to obtain fused multi-modal features;

[0066] A task migration module: using contrastive learning to construct two different transfer functions to process the fused multi-modal feature values to obtain different sentiment values, and migrating different downstream tasks.

[0067] A multi-modal sentiment analysis device based on a multi-element loss function comprises:

[0068] A memory for storing a computer program implementing the multi-modal sentiment analysis method based on the multi-element loss function;

[0069] A processor for implementing the multi-modal sentiment analysis method based on the multi-element loss function when executing the computer program.

[0070] A computer-readable storage medium comprising:

[0071] The computer-readable storage medium stores a computer program, which can implement the multi-modal sentiment recognition method based on the self-attention mechanism fusion when executed by a processor.

[0072] Compared with the prior art, the beneficial effects of the present application are:

[0073] 1、The present application maximizes the reduction of model redundancy by comparing the influence of model depth on convergence, and enhances the information correlation degree between modalities compared with the prior art.

[0074] 2、In step 3, the multi-element loss function is used to limit feature extraction and modal fusion, which improves the accuracy of model recognition compared with the prior art.

[0075] 3、In step 3, the data augmentation method generates multiple data pairs, and combines the contrast learning theory to perform self-training on the model from the perspective of maximizing relevant information, which shows excellent results on multiple data sets compared with the prior art.

[0076] 4、In step 4, the transfer function of the contrast learning framework is combined to realize the transfer of different downstream tasks, which improves the generalization and robustness of the model compared with the prior art.

[0077] In summary, compared with the prior art, the multi-modal sentiment analysis method based on the multi-element loss function proposed in the present application fully considers text, speech and video information, uses an encoder based on cross-attention mechanism for fusion, and improves the processing and fusion efficiency of data; The present application combines the contrast learning framework for multi-element loss function assisted training, so that the correlation information can be maximized in the feature extraction and modal fusion process, and the interference of noise is reduced; At the same time, different transfer functions under the contrast learning framework are combined, which improves the accuracy of the sentiment analysis model while improving the transferability and robustness of the model, so that the application range of the model is wider. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 The flowchart of the method of the present application.

[0079] Figure 2 For the network model architecture diagram of the application.

[0080] Figure 3 For the embedded coding flow chart in the text feature extraction encoder.

[0081] Figure 4 For the multi-head attention mechanism encoder architecture diagram in the text feature extraction encoder.

[0082] Figure 5 For the self-attention mechanism encoder architecture diagram in the video feature extraction encoder. DETAILED DESCRIPTION

[0083] The technical solutions of the application will be described in detail below with reference to the drawings and examples.

[0084] Referring to Figure 1 , the multi-modal sentiment analysis method based on the multi-element loss function includes the following steps:

[0085] Step 1, data preprocessing: obtaining data of multiple modalities, preprocessing data of multiple modalities, dividing the preprocessed data into a training set, a test set and a validation set, the data of multiple modalities including text data, audio data and video data;

[0086] Step 2, model construction: constructing a multi-modal sentiment analysis model based on a multi-element loss function, the multi-modal sentiment analysis model based on the multi-element loss function including a feature extraction encoder and a data fusion encoder;

[0087] Step 3, model training and optimization: combining the data in the training set of step 1 with the multi-element loss function and the contrastive learning mechanism module to generate corresponding positive and negative sample pairs, inputting the positive and negative sample pairs into the multi-modal sentiment analysis model based on the multi-element loss function constructed in step 2, training and optimizing the feature extraction encoder and the data fusion encoder, and obtaining the optimal multi-modal sentiment analysis model based on the multi-element loss function;

[0088] Step 4, task migration: using contrastive learning to construct two different migration functions to process the multi-modal sentiment feature values obtained in step 3, to obtain different sentiment values, and to migrate different downstream tasks.

[0089] Referring to Figure 2 , Figure 3 , Figure 4 and Figure 5The current largest multi-modal sentiment annotation dataset CMU MOSEI and the MOSI dataset are selected. The CMU MOSEI dataset is significantly higher than other datasets in terms of the number of annotated data points, the number of speakers, the number of modalities, and the total video duration. In addition, the CMU MOSEI has two labels of sentiment and emotion. The sentiment has seven stages from negative to positive, that is, [-3, 3]. The emotion includes six categories of anger, happiness, sadness, surprise, fear and disgust. The text data in the MOSI dataset is mainly extracted from the speech and video of the actor's performance, including the actor's verbal expression, gestures and facial expressions and other text information. The speech data is mainly extracted from the actor's recording, including the frequency and tone of the audio signal. The video data is mainly extracted from the actor's performance video, including the actor's facial expression, gesture, posture and movement. Each video segment in the MOSI dataset has a sentiment label, including positive sentiment, negative sentiment and neutral sentiment. In addition, each sentiment label is accompanied by an intensity score, which represents the intensity of the emotion.

[0090] The specific process of data preprocessing on the two datasets in step 1 is as follows:

[0091] Step 1.1: Taking the text segment as the sequence reference, the dataset is divided according to the fixed time, and the video data and audio data are respectively aligned with the text to form the calculation sequence;

[0092] Step 1.2: The text data is divided into word vectors, and the corresponding position code is added to the end of the vector. For audio data, the frequency spectrum is obtained by FFT fast Fourier transform, and the mel frequency cepstrum data vector is converted by the spec conversion function. For video data, the above-mentioned data frame sequence aligned with the text is converted into a three-channel feature vector of RGB, and is subjected to adaptive scale cropping and scaling to convert it into a four-channel feature vector;

[0093] Step 1.3: Align the audio data, video data and text data processed in step 1.2, and divide them into training set, test set and validation set according to the proportion of 8:1:1.

[0094] The feature extraction encoder in step 2 includes a text emotion feature extraction encoder, a video emotion feature extraction encoder and an audio emotion feature extraction encoder. The data fusion encoder in step 2 is a multi-layer encoder based on the cross-attention mechanism, which is based on the encoder stacked with a specific number of cross-attention mechanisms.

[0095] (1) Text sentiment feature extraction encoder: the text sentiment feature extraction encoder is a multi-layer multi-head attention mechanism encoder, which is based on a stacked multi-layer Transformer encoder architecture module, specifically:

[0096] For the text feature extraction encoder, referring to the calculation process of BERT in natural language processing, the present application is built based on the Transformer encoder architecture. As shown in Figure 3 , first, the text is subjected to Token Embedding instruction encoding and Position Eembedding position encoding, and the encoding specific form is as follows:

[0097]

[0098] With the increase of the dimension of the sentence, the period gradually increases, p i Different dimensions from 1 to 768 are generated using different frequencies of sine and cosine functions, and then added to the corresponding position word vector.

[0099] In order to better focus on the context of the emotional text change relationship, a multi-head attention mechanism is used to obtain multiple transformation values, thereby maximizing the use of text information. As shown in Figure 4 , the network input is divided into multiple branches, and the results of each branch are spliced to obtain H' by mapping each branch through the parameter matrix of the attention mechanism, and the specific formula is:

[0100] Q i = HW i,Q

[0101] K i = HW i,K

[0102] V i = HW i,V

[0103]

[0104] H' = [H 1,' , H 2,' ,..., H N,' ] W O

[0105] Wherein, the input H is multiplied by the matrix W i,Q , W i,K , W i,V (all are trainable parameter matrices), Q i , K i , V iThe matrix, i.e., the query, key value and value matrix of the i-th "head", is subjected to a softmax function to obtain a probability value, and i outputs H i,' The concatenation is multiplied by a trainable mapping matrix W O to obtain the final output H'.

[0106] The residual network enables the network to be deepened while preserving the effective timing, and the non-linear transformation is performed through the full connection layer and the non-linear activation function according to the following formula:

[0107]

[0108] m i =MLP(output i )=W2*GELU(W1×output i +b1)+b2

[0109] wherein the smooth GELU is used instead of the RELU, so that the derivative remains non-zero state when the input x corresponds to the negative output. The MLP is a full connection layer, m i is a one-to-one correspondence, i.e., m i is only determined by output i . W1 and W2 are trainable parameters, and b1 and b2 are offsets.

[0110] In addition to the above replacement of the smooth activation function, the Layer norm performs variance scaling according to the position of Post-LN, so that the training of the subsequent model is more stable.

[0111] (2) Video emotion feature extraction encoder: the video emotion feature extraction encoder is a multi-layer self-attention mechanism encoder, which is based on a stacked multi-layer Vision Transformer encoder architecture module as a basic architecture, specifically:

[0112] The video feature extraction encoder takes the Vision Transformer module as the basic architecture and is composed of three modules. As Figure 5 shown, the Fattened patches, i.e., the embedding layer, transforms the video data in the format of [H, W, C] to divide the input picture into 16x16x3 patches, and then maps each three-channel patch to a one-dimensional vector to obtain a vector of length 768, i.e., token. Through the above sine and cosine encoding method for embedding, the self-attention mechanism architecture is Q i , K i , V i is no longer concatenated, but is calculated separately, and the MLP classification layer is consistent with the above.

[0113] (3) Audio emotion feature extraction encoder: the audio emotion feature extraction encoder is a multi-layer self-attention mechanism encoder, which is based on a stacked multi-layer self-attention mechanism forward network encoder as a basic architecture, specifically:

[0114] The audio feature extraction encoder is based on the Transformer architecture, the difference is that the input mel spectrum data is embedded before data processing and fusion, and the self-attention mechanism and the MLP classification layer are consistent with the above video feature extraction encoder.

[0115] (4) Data fusion encoder: the data fusion encoder is a multi-layer cross-attention mechanism-based encoder, which is based on a stacked cross-attention mechanism encoder with a specific number of layers as a basic architecture, specifically:

[0116] The fusion network is designed based on the cross-attention mechanism, and the data fusion of multiple modalities is performed according to the following formula.

[0117] First, the similarity score between samples is calculated, and the inner product of vectors is selected as the similarity score basis in the present application, and the similarity is calculated by the following formula:

[0118] e i,j =w T [UX i ;VX j ]

[0119] In the formula, X i and X j represent the feature vectors of the i-th sample and the j-th sample, respectively; [UX i ; VX j ] represents the concatenation of the two feature vectors and the linear transformation by matrix U and V; w is the weight vector obtained by training, and the product is the similarity score e i,j ;

[0120] Second, the similarity score is converted into attention weight according to the following formula, and the probability distribution is converted according to the following formula using softmax:

[0121]

[0122] In the formula, a i,j represents the attention weight of the i-th and j-th samples; n represents the total number of samples.

[0123] Finally, multiply the feature vector and perform weighted summation to obtain the weighted feature vector, and the formula is as follows:

[0124]

[0125] wherein h i is the difference attention feature vector of the i-th sample.

[0126] The specific process of step 2 is as follows:

[0127] Step 2.1: input the text data preprocessed in step 1 into a text sentiment feature extraction encoder to extract text sentiment features;

[0128] Step 2.2: input the video data preprocessed in step 1 into a video sentiment feature extraction encoder to extract video sentiment features;

[0129] Step 2.3: input the audio data preprocessed in step 1 into an audio sentiment feature extraction encoder to extract audio sentiment features;

[0130] Step 2.4: input the extracted text sentiment features, video sentiment features and audio sentiment features into a data fusion encoder for feature fusion to obtain a multi-modal sentiment analysis model based on a multi-element loss function.

[0131] The specific process of step 3 model training and optimization is as follows:

[0132] Step 3.1: data augmentation is performed on the calculation sequence generated in the data preprocessing process of step 1, specifically: a group of input calculation sequences is formed by text, audio and video data within a period of time, data augmentation is performed according to Table 1, samples of the same sequence are taken as positive samples, and samples of different sequences can be taken as negative samples of each other, to prepare for subsequent training of the model under the contrast learning mechanism, and Table 1 is as follows:

[0133] Table 1 contrast learning data scheme

[0134] Data scheme Text Audio Video Positive sample Same sequence Same sequence Same sequence Negative sample 1 Different sequence Same sequence Same sequence Negative sample 2 Same sequence Different sequence Same sequence Negative sample 3 Same sequence Same sequence Different sequence

[0135] Step 3.2: group the data after data augmentation in step 3.1 and shuffle to form different positive and negative sample pairs;

[0136] Step 3.3: taking text data as the alignment reference, input the positive and negative sample pairs obtained in step 3.2 into the corresponding feature extraction encoder constructed in step 2 to extract corresponding features, specifically:

[0137] Input the above processed data into the model, and perform 256 epoch iterations on the overall network architecture to gradually regress to a stable optimal solution, which can effectively extract the relevant features of the corresponding three sentiment dimensions. Then, through the cross-attention mechanism, different weights are adaptively assigned to different modalities to effectively fuse the multi-modal data, and the specific training process is as follows:

[0138] The video emotion feature and the text emotion feature are extracted by using the contrast learning mechanism and the ITC_LOSS loss function;

[0139] The audio emotion feature and the text emotion feature are extracted by using the contrast learning mechanism and the InfoNCE_LOSS loss function;

[0140] Step 3.4: The feature extraction encoder is optimized by using the two contrast loss functions of the positive and negative sample pairs in step 3.3, specifically:

[0141] For the video emotion feature extraction encoder and the text emotion feature extraction encoder, ITC_LOSS is used as the contrast learning loss function, as shown in the following formula:

[0142] L ITC =αL triplet +(1-α)L contrastive

[0143] In the formula, L triplet is the Triplet loss loss function, L contrastive is the Contrastive loss loss function, and α is the weight coefficient of the other loss function, used to balance the influence of the two loss functions;

[0144] Wherein, the loss functions of L triplet and L contrastive are as follows:

[0145]

[0146] In the formula, N is the number of categories in the training set, P i is the number of samples in the i-th category, x i a is the a-th sample in the i-th category, (x i a ,x i p ) is the similarity between samples x i a and x i p , S(x i a ,x i n ) is the similarity between sample x i n and sample x i a , and m is a hyperparameter used to control the distance between samples of the same category;

[0147]

[0148]

[0149] where d(x i a ,x i p ) is the distance between samples x i a and x i p , y i a is the class label of sample x i a , y different represents the loss value of different types of samples, and m is a hyperparameter used to control the distance between samples of the same class.

[0150] Through the weight distribution of the above loss function, samples of the same class can be mapped to a tight cluster, and samples from different classes are mapped to separate clusters. The TripletLoss loss function is mainly used to distinguish sample points between different classes, and the Contrastive Loss loss function is more used for learning similarity measurement. By weighting the loss function, it can better adapt to different learning tasks in different scenarios.

[0151] The following InfoNCE loss function is used for the audio emotion feature extraction encoder and the text emotion feature extraction encoder, which is used as the loss function of the cross-modal learning task of text and audio data for training. The specific formula is as follows:

[0152]

[0153] where N represents the number of samples in a training batch, that is, for N samples in a training batch, N pairs of positive samples are obtained through data augmentation, at this time there are 2N samples, and the negative samples are the remaining 2(N-1) samples, that is, the negative samples are generated based on the data of this training batch; z i is the feature extracted for sample i; z j is the feature extracted for sample j; sim(z i ,z k ) is the calculation formula of cosine similarity (sim(u,v) = u T v / ||u||·||v||), 1[k≠i] inputs 0 or 1, when k is not equal to i, the output result is 1, otherwise 0; τ is the temperature coefficient, used to control the amplitude of training;

[0154] Step 3.5: The text sentiment features, video sentiment features and audio sentiment features extracted in step 3.3 are fused by a data fusion encoder to obtain fused multi-modal sentiment feature values, and an InfoNCE_LOSS loss function is used for regression training, specifically:

[0155] According to the InfoNCE loss function, the distance of the positive sample pair is calculated, while the distance of the negative sample is superimposed in the denominator. The distance of the positive sample pair becomes closer, while the distance of the negative sample becomes farther, thereby improving the correlation between the features extracted from the modalities. The feature extraction function is fine-tuned according to the loss to complete the feature extraction training of the overall multi-modal network.

[0156] Step 3.6: Repeat steps 3.3-3.5 to train and optimize the multi-modal sentiment analysis model based on the multi-element loss function to obtain the optimal multi-modal sentiment analysis model based on the multi-element loss function.

[0157] Finally, considering that the network model of the present application does not involve a generation process, the fusion network uses the InfoNCE loss function to train the emotional labels in the data, taking the contrast relationship as the output result and the mapping of the fusion vector. The InfoNCE loss function formula is as shown above.

[0158] The specific process of the task migration of step 4 is as follows:

[0159] Step 4.1: The emotional state is divided into absolute emotion and relative emotion, and different migration strategies are set for the two directions of absolute emotion state and relative emotion tendency. It should be noted that the data targeted by the migration state is the data processed by the multi-modal fusion encoder described above. The probability can be converted by a softmax function, or the emotional state or the interval of the emotional tendency can be directly output. The present application explores two different positive / negative sample selection migration strategies;

[0160] Step 4.2: Different migration functions are set according to the two different emotional states divided in step 4.1 to process the fused multi-modal feature values obtained in step 3.5. The migration functions include absolute emotion migration functions and relative emotion migration functions;

[0161] Step 4.3: The results obtained in step 4.2 are normalized by a softmax layer to obtain discrete values of absolute emotion and continuous values of relative emotion, specifically:

[0162] (1) Absolute emotion transfer function: Comparative emotion can be constructed by matching windows with similar emotional states as positive counterparts and windows with different emotional states as negative counterparts. Specifically, the absolute emotional state is binary as "high" and "low", and windows with the same (different) state are considered similar (dissimilar). The binary standard is based on the average value of the emotional state labeled by CMU_MOSEI and MOSI. The weighted sum of the corresponding window samples can be used as the absolute emotional state of the sample. Different scenes can be manually regulated by a self-defined threshold e. The absolute emotion transfer function is:

[0163]

[0164] where n is the number of sequence samples, v i is the i-th sample, e is the bias threshold, A n is the absolute emotional binary state.

[0165] (2) Relative emotion transfer function: Relative measurement can also be performed using emotional transformation. When the emotional change value of the former and the latter is large, the average difference is calculated in the form of average difference. The label is selected to define the value, and the average difference of the corresponding time period is summed to obtain the change trend of emotion. The relative emotion transfer function is:

[0166]

[0167] where n is the number of sequence samples, v i is the i-th sample, v i-1 is the i-1-th sample, R n is the relative emotional state.

[0168] The main difference between the first contrast labeling strategy and the second strategy is that the former is direct, because the values of "high" and "low" are derived from the actual size of the impact annotation trace, while the second strategy is indirect, because "change" is a change trace, which expresses the average absolute change rate.

[0169] The multi-modal emotion analysis system based on the multi-element loss function comprises:

[0170] A data preprocessing module: obtaining data of multiple modalities, and preprocessing data of multiple modalities;

[0171] A model construction module: constructing a multi-modal emotion analysis model based on a multi-element loss function, the multi-modal emotion analysis model based on the multi-element loss function comprising a feature extraction encoder and a data fusion encoder;

[0172] The model training and optimization module inputs the preprocessed data into a multi-modal sentiment analysis model based on a multi-element loss function, and generates corresponding positive and negative samples by using the multi-element loss function in combination with a contrast learning mechanism module, trains and optimizes the feature extraction encoder and the data fusion encoder, and obtains the fused multi-modal features.

[0173] The task migration module uses contrast learning to construct two different migration functions to process the fused multi-modal feature values to obtain different sentiment values, thereby migrating different downstream tasks.

[0174] The multi-modal sentiment analysis device based on a multi-element loss function comprises:

[0175] The memory is used to store a computer program for implementing the multi-modal sentiment analysis method based on a multi-element loss function.

[0176] The processor is used to implement the multi-modal sentiment analysis method based on a multi-element loss function when the computer program is executed.

[0177] A computer readable storage medium comprises:

[0178] The computer readable storage medium stores a computer program, and the computer program can implement the multi-modal sentiment recognition method based on self-attention mechanism fusion when executed by a processor.

[0179] The application effect of the present application will be described in detail in combination with experiments.

[0180] (1) Experimental conditions:

[0181] All the following experiments are realized by using Pycharm software under the Ubuntu 20.04 operating system, and the specific experimental conditions are shown in Table 2:

[0182] Table 2 Experimental conditions

[0183] Experimental conditions Parameters Graphic card 4090(24g) Optimizer Adam Learning rate 1e-4 Batch size 64 Dropout in / out 0.2 / 0.1

[0184] In the table, batchsize represents the number of samples selected for training at a time, Dropout is a regularization method, Dropout in represents the proportion of random dropout of the input vector, and Dropout out represents the proportion of dropout during forward propagation.

[0185] (2) Analysis of experimental results:

[0186] First, an evaluation experiment of the overall network scheme was conducted, as shown in Tables 3 and 4. Compared with small sentiment analysis networks such as LMF, LSTM and RAVEN, the present invention significantly outperforms in both overall efficiency and accuracy of sentiment analysis. Compared with large fusion networks such as TFN, the present invention achieves better results in the accuracy of single regression sentiment analysis and also improves the accuracy of multi-regression sentiment analysis.

[0187] Table 3. Results of MOSEI comparison experiment (%)

[0188]

[0189]

[0190] Table 4. Results of MOSI Comparison Experiment (%)

[0191] Name ACC-2 F1 ACC-7 TFN 73.90 73.42 32.10 LMF 80.10 82.15 32.81 LF-LSTM 76.84 76.71 35.30 RAVEN 78.02 76.60 33.20 PMR 83.66 83.47 40.62 The invention 88.52 86.40 44.25

[0192] In the table, TFN, LMF, LF-LSTM, RAVEN, MCTN, MulT, PMR, MISA, and FDMER are different fusion networks. ACC-2 represents the model's performance on different sentiment categories. F1 is a comprehensive metric based on recall and classifier accuracy, used to evaluate the performance of binary classifiers. ACC-7 refers to the proportion of each category in the total number of samples in that category.

[0193] Secondly, experiments were conducted on the fusion scheme. By closing the channel of a certain modality, the analysis results of the trained model were verified. As shown in Table 5, closing the text channel resulted in a significant decrease in sentiment analysis accuracy; closing the video channel resulted in a certain degree of accuracy decrease; and closing the audio channel also resulted in some loss of accuracy. All of these losses were significant, thus verifying the effectiveness of the multimodal fusion mechanism of this invention.

[0194] Table 5. Results of the fusion experiment (%)

[0195] Method ACC-2 F1 ACC-7 Text + video 85.66 86.02 52.54 Text + speech 83.22 81.95 51.54 Speech + video 72.12 71.56 34.77

[0196] In summary, the multimodal sentiment analysis method based on multivariate loss functions proposed in this invention fully considers text, speech, and video information, and utilizes an encoder based on a cross-attention mechanism for fusion, thereby improving data processing and fusion efficiency. This invention combines a contrastive learning framework with multivariate loss function-assisted training, maximizing the use of relevance information and reducing noise interference during feature extraction and modality fusion. Furthermore, by incorporating different transfer functions within the contrastive learning framework, the accuracy of the sentiment analysis model is improved, along with its transferability and robustness, thus broadening its application scope.

Claims

1. A multi-modal sentiment analysis method based on a multi-element loss function, characterized in that, Comprising the following steps: Step 1, data preprocessing: obtaining data of multiple modalities, preprocessing data of multiple modalities, dividing the preprocessed data into a training set, a test set and a validation set, wherein the data of multiple modalities includes text data, audio data and video data; Step 2, model construction: constructing a multi-modal sentiment analysis model based on a multi-element loss function, wherein the multi-modal sentiment analysis model based on the multi-element loss function comprises a feature extraction encoder and a data fusion encoder; Step 3, model training and optimization: combining the data in the training set in step 1 with a multi-element loss function and a contrastive learning mechanism module to generate corresponding positive and negative sample pairs, inputting the positive and negative sample pairs into the multi-modal sentiment analysis model based on the multi-element loss function constructed in step 2, training and optimizing the feature extraction encoder and the data fusion encoder, and obtaining an optimal multi-modal sentiment analysis model based on the multi-element loss function; The specific process of step 3 is: Step 3.1: data augmentation is performed on the calculation sequence generated in the data preprocessing process of step 1, specifically, the data of the three modalities in the training set are matched with each other to form positive and negative sample pairs; Step 3.2: the data after data augmentation in step 3.1 is grouped and shuffled to form different positive and negative sample pairs; Step 3.3: taking the text data as the alignment reference, inputting the positive and negative sample pairs obtained in step 3.2 into the corresponding feature extraction encoder constructed in step 2 to extract corresponding features, specifically: Video emotion features and text emotion features are extracted using a contrastive learning mechanism combined with an ITC_LOSS loss function; Audio emotion features and text emotion features are extracted using a contrastive learning mechanism combined with an InfoNCE_LOSS loss function; Step 3.4: the feature extraction encoder is optimized using the two contrastive loss functions of the positive and negative sample pairs in step 3.3; Step 3.5: the text emotion features, video emotion features and audio emotion features extracted in step 3.3 are fused through a data fusion encoder to obtain fused multi-modal emotion feature values, and an InfoNCE_LOSS loss function is used for regression training; Step 3.6: repeating steps 3.3-3.5, training and optimizing the multi-modal sentiment analysis model based on the multi-element loss function to obtain an optimal multi-modal sentiment analysis model based on the multi-element loss function; Step 4, task migration: constructing two different migration functions using contrastive learning to process the multi-modal emotion feature values obtained in step 3 to obtain different emotion values, and migrating different downstream tasks. 2.The multi-modal sentiment analysis method based on multi-element loss function according to claim 1, characterized in that, The specific process of step 1 is: Step 1.1: dividing the data according to a predetermined time sequence, taking the text data as the alignment reference to obtain corresponding audio segments and video frames; Step 1.2: dividing the text data in step 1.1 into word vectors and adding the corresponding position code to the end of the vector; converting the audio data of the audio segment into corresponding mel spectrum; converting the video data of the video frame into corresponding size and format; Step 1.3: Align the processed audio data, video data, and text data, and divide them into a training set, a test set, and a validation set. 3.The multi-modal sentiment analysis method based on multi-element loss function of claim 1, wherein, The feature extraction encoder in step 2 includes a text sentiment feature extraction encoder, a video sentiment feature extraction encoder, and an audio sentiment feature extraction encoder; The text sentiment feature extraction encoder is a multi-layer multi-head attention mechanism encoder based on a module of a stacked multi-layer Transformer encoder architecture; The video sentiment feature extraction encoder is a multi-layer self-attention mechanism encoder based on a module of a stacked multi-layer Vision Transformer encoder architecture; The audio sentiment feature extraction encoder is a multi-layer self-attention mechanism encoder based on a module of a stacked multi-layer self-attention mechanism forward network encoder; The data fusion encoder in step 2 is a multi-layer cross-attention mechanism-based encoder based on a module of a stacked cross-attention mechanism encoder of a specific number of layers. 4.The multi-modal sentiment analysis method based on multi-element loss function of claim 1, wherein, The specific process of step 2 is as follows: Step 2.1: Input the preprocessed text data in step 1 into the text sentiment feature extraction encoder to extract text sentiment features; Step 2.2: Input the preprocessed video data in step 1 into the video sentiment feature extraction encoder to extract video sentiment features; Step 2.3: Input the preprocessed audio data in step 1 into the audio sentiment feature extraction encoder to extract audio sentiment features; Step 2.4: Input the extracted text sentiment features, video sentiment features, and audio sentiment features into the data fusion encoder for feature fusion to obtain a multi-modal sentiment analysis model based on a multi-element loss function. 5.The multi-modal sentiment analysis method based on multi-element loss function of claim 1, wherein, The specific process of step 4 is as follows: Step 4.1: Divide the sentiment state into two sentiment states: absolute sentiment and relative sentiment; Step 4.2: Set an absolute sentiment transfer function and a relative sentiment transfer function according to the two sentiment states divided in step 4.1 to process the fused multi-modal sentiment feature values obtained by the optimal multi-modal sentiment analysis model based on a multi-element loss function in step 3; The absolute sentiment transfer function is: where n is the number of sequence samples, v i is the i-th sample, e is the bias threshold, A n is the absolute emotional binary state; The relative sentiment transfer function is: where n is the number of sequential samples, v i is the i-th sample, v i-1 is the i-1-th sample, R n is the relative emotional binary state; Step 4.3: Normalize the results obtained in step 4.2 through a softmax layer to obtain discrete values of absolute sentiment and continuous values of relative sentiment. 6.The multi-modal sentiment analysis method based on multi-element loss function of claim 1, wherein, The ITC_LOSS loss function in step 3.3 is as follows: L ITC = aL triplet + (1 - a)L contrastive In the formula, L triplet is a Tripletloss loss function, L contrastive is a Contrastive loss loss function, and a is a weight coefficient of the loss function, used to balance the influence of the two loss functions. The L triplet Loss function and L contrastive The loss function is as follows: where N is the number of classes in the training set, P i is the number of samples in the i-th class, x i a is the a-th sample in the i-th class, (x i a ,x i p ) is the similarity between sample x i a and x i p , S(x i a ,x i n ) is the similarity between sample x i n and sample x i a , and m is a hyper-parameter used to control the distance between samples of the same class. where d(x i a ,x i p ) is the distance between samples x i a and x i p , y i a is the class label to which sample x i a belongs, y different is the loss value of different types of samples, and m is a hyperparameter used to control the distance between samples of the same class. The ITC_LOSS loss function in step 3.3 is as follows: where N is the number of samples in a training batch, z i is the feature extracted for sample i, z j is the feature extracted for sample j, sim(z i ,z k ) is the cosine similarity formula (sim(u, v) = u Τ v / ||u||·||v||), 1[k≠i], and τ is the temperature coefficient.

7. A multi-modal sentiment analysis system based on a multi-variate loss function characterized in that, It includes: A data preprocessing module that obtains multiple modal data and preprocesses the multiple modal data; A model construction module that constructs a multi-modal sentiment analysis model based on a multi-element loss function, which includes a feature extraction encoder and a data fusion encoder; The model training and optimization module inputs the preprocessed data into a multi-modal sentiment analysis model based on a multi-element loss function, and generates corresponding positive and negative samples by using the multi-element loss function in combination with a contrast learning mechanism module, trains and optimizes the feature extraction encoder and the data fusion encoder, and obtains the fused multi-modal features. Specifically, the method comprises the following steps: Data augmentation is performed on the calculation sequence generated in the preprocessing process, specifically, the data of the three modalities in the training set are matched with each other to form positive and negative sample pairs; The data after data augmentation is grouped and shuffled to form different positive and negative sample pairs; The text data is used as the alignment benchmark, and the positive and negative sample pairs are input into the corresponding feature extraction encoder for corresponding feature extraction, specifically: Video sentiment features and text sentiment features are extracted by using the contrast learning mechanism in combination with the ITC_LOSS loss function; Audio sentiment features and text sentiment features are extracted by using the contrast learning mechanism in combination with the InfoNCE_LOSS loss function; The feature extraction encoder is optimized by using the two contrast loss functions of the positive and negative sample pairs; The extracted text sentiment features, video sentiment features and audio sentiment features are fused by the data fusion encoder to obtain the fused multi-modal sentiment feature values, and the InfoNCE_LOSS loss function is used for regression training; The grouped data in the training set is processed in batches according to the above feature extraction encoder data processing process, and the multi-modal sentiment analysis model based on the multi-element loss function is trained and optimized to obtain the optimal multi-modal sentiment analysis model based on the multi-element loss function. The task transfer module uses contrast learning to construct two different transfer functions to process the fused multi-modal feature values to obtain different sentiment values, which are used to transfer different downstream tasks.

8. A multi-modal sentiment analysis device based on a multi-variate loss function, characterized in that, It comprises: a memory for storing a computer program for implementing the multi-modal sentiment analysis method based on the multi-element loss function of any one of claims 1-6; a processor for executing the computer program to implement the multi-modal sentiment analysis method based on the multi-element loss function of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It comprises: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the multi-modal sentiment recognition method based on the self-attention mechanism fusion of any one of claims 1-6.

Images