A multi-modal sentiment prediction method based on transductive contrastive learning

Abstract

The application discloses a kind of multi-modal sentiment prediction methods based on interoception contrast learning, including obtaining multi-modal data containing emotional state;Interoception conversion network model based on contrast learning is constructed;Dictionary structure is used to construct negative sample queue, and the sample of each iteration is used to update negative sample queue;Multi-modal data and negative sample queue are used to train interoception conversion network model based on contrast learning;The interoception conversion network model based on contrast learning after training is used for multi-modal sentiment prediction.The application enables multi-modal data to be more natural and effective fusion, so as to improve the sentiment prediction accuracy and model calculation efficiency of interoception conversion network model to multi-modal data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal sentiment analysis, and particularly relates to a multi-modal sentiment prediction method based on synesthesia contrast learning. BACKGROUND

[0002] With more and more researches on multi-modal data, the natural fusion and understanding of multi-modal information have been widely studied in the field of emotional artificial intelligence. The initial sentiment analysis task is carried out in a single-modal environment, which has aroused strong interest in the research community. The multi-modal sentiment analysis (MSA) task aims to use multi-modal information such as video, text and audio to analyze the emotion in people's speech. Compared with single-modal data, multi-modal data provides more clues for richer perception. The purpose is to endow machines with the ability to discover and understand human emotional states and make corresponding responses, that is, through the learning of various modal data such as human facial expressions, speech intonation, text content or body physiological signals, the emotional state of human beings is analyzed. The research on sentiment analysis is of great benefit to many applications from different fields, such as intelligent human-computer interaction systems, intelligent business and customer service systems and remote medical systems.

[0003] In order to better utilize multi-sensory data, people have made many attempts on the advanced fusion strategy of different modalities. From simple modal connection to aggregation-based modal fusion, the mainstream method is generally to process each modality first and then combine them. However, these methods cannot effectively simulate the feature correlation between different modalities, so they cannot fully represent the close correlation between different modalities and cannot contain enough feature information in the same high-dimensional space. These fusion methods often rely on artificial models spliced by artificial methods, without trying to embody the rules of modal fusion in reality, resulting in poor effect of modal fusion, so the accuracy is often low when applied to sentiment prediction of multi-modal data. SUMMARY

[0004] In view of the above deficiencies in the prior art, the present application provides a multi-modal sentiment prediction method based on synesthesia contrast learning.

[0005] In order to achieve the above-mentioned application purposes, the technical scheme adopted by the present application is:

[0006] A multi-modal sentiment prediction method based on synesthesia contrast learning, comprising the following steps:

[0007] Obtaining multi-modal data containing emotional states, the multi-modal data comprising at least two of video modal information, text modal information and audio modal information;

[0008] Constructing a synesthesia conversion network model based on contrast learning;

[0009] The negative sample queue is constructed in a dictionary structure, and the negative sample queue is updated by using the sample of each iteration;

[0010] The multi-modal data and the negative sample queue are used to train the cross-modal conversion network model based on contrastive learning.

[0011] The trained cross-modal conversion network model based on contrastive learning is used for multi-modal sentiment prediction.

[0012] Optionally, the cross-modal conversion network model based on contrastive learning comprises a cross-modal attention framework, an auto-encoder and a conversion network encoder connected in sequence.

[0013] The cross-modal attention framework is used to fuse the multi-modal data to generate a cross-modal unit and a positive sample cross-modal unit.

[0014] The auto-encoder is used to perform modal fusion on the cross-modal unit and the positive sample cross-modal unit respectively to obtain a strengthened cross-modal unit and a positive sample strengthened cross-modal unit.

[0015] The conversion network encoder encodes the strengthened cross-modal unit and the positive sample strengthened cross-modal unit respectively to obtain an encoding result.

[0016] Optionally, the cross-modal attention framework is used to fuse the multi-modal data to generate a cross-modal unit and a positive sample cross-modal unit, and specifically comprises:

[0017] Sequentially selecting one kind of modal information from the multi-modal data as a main modal, and other kinds of modal information as additional attention modal.

[0018] Using a self-attention mechanism to fuse the features of the main modal and the additional attention modal respectively to obtain a first feature fusion vector.

[0019] Using a self-attention mechanism to fuse the features of the main modal and the first feature fusion vector to obtain a second feature fusion vector.

[0020] Splicing all the second feature fusion vectors to generate a sample output vector labeled by three kinds of modalities at the same time.

[0021] Using different dropout mask probabilities to perform a dropout mask operation on the sample output vector to generate a cross-modal unit and a positive sample cross-modal unit.

[0022] Optionally, the cross-modal unit is represented as:

[0023]

[0024] Wherein, Q α is the Q value of the main modal a as a self-attention mechanism. To add an attention modality as the K value in the self-attention mechanism, T represents the transpose, and V... α The dominant mode α is used as the V value in the self-attention mechanism, d k K represents the dimension of the value, η represents the dropout mask probability, softmax represents the activation function, and concat represents the concatenation operation.

[0025] Optionally, the positive sample sensing unit is represented as:

[0026]

[0027] Among them, Q α The master mode α is used as the Q-value in the self-attention mechanism. To add an attention modality as the K value in the self-attention mechanism, T represents the transpose, and V... α The dominant mode α is used as the V value in the self-attention mechanism, d k η is the dimension of the K value. + is the dropout mask probability of positive samples, softmax is the activation function, and concat is the concatenation operation.

[0028] Optionally, the autoencoder is used to perform modal fusion on the synesthetic unit and the positive sample synesthetic unit respectively to obtain the enhanced synesthetic unit and the positive sample enhanced synesthetic unit, specifically including:

[0029] The autoencoder includes an encoder and a decoder, each constructed using a fully connected neural network.

[0030] The encoder is used to map the sample output vector of the input synesthetic unit or positive sample synesthetic unit to a low-dimensional space for encoding, so as to obtain the low-dimensional vector unit of the synesthetic unit or positive sample synesthetic unit.

[0031] The decoder is used to reproject the low-dimensional vector units of the synesthetic unit or the positive sample synesthetic unit into a high-dimensional space for feature fusion, so as to obtain the enhanced synesthetic unit and the positive sample enhanced synesthetic unit.

[0032] Optionally, the conversion network encoder encodes the enhanced synesthetic unit and the positive sample enhanced synesthetic unit respectively to obtain the encoding result, specifically including:

[0033] The transformation network encoder includes a raw encoder and a positive sample encoder constructed using a multi-head self-attention structure and a feedforward neural network structure, respectively.

[0034] The original encoder is configured to perform hierarchical normalization on the enhanced transduction unit, then pass through a multi-head self-attention structure, and perform residual connection between the input of the normalization layer and the output of the multi-head self-attention structure; then perform hierarchical normalization and pass through a feedforward neural network structure, and perform residual connection between the input of the normalization layer and the output of the feedforward neural network structure, to obtain the encoding result of the enhanced transduction unit.

[0035] The positive sample encoder is configured to perform hierarchical normalization on the positive sample enhanced transduction unit, then pass through a multi-head self-attention structure, and perform residual connection between the input of the normalization layer and the output of the multi-head self-attention structure; then perform hierarchical normalization and pass through a feedforward neural network structure, and perform residual connection between the input of the normalization layer and the output of the feedforward neural network structure, to obtain the encoding result of the positive sample enhanced transduction unit.

[0036] Optionally, the conversion network encoder adopts a momentum optimization method, and the parameters of the original encoder are used to update the parameters of the positive sample encoder, and the update formula is:

[0037] ω simi ← kω simi + (1-k)ω nor

[0038] wherein ω simi ′ is the updated parameter of the positive sample encoder, ω simi is the current parameter of the positive sample encoder, ω nor is the current parameter of the original encoder, and k is a hyperparameter of parameter updating.

[0039] Optionally, the negative sample queue is constructed by using a dictionary structure, and the negative sample queue is updated by using the samples of each iteration, and specifically includes:

[0040] The negative sample queue is constructed by using a dictionary structure, and after each model iteration is completed, the small batch of samples of the current input multi-modal data is encoded to replace the small batch of samples with the earliest iteration time in the negative sample queue.

[0041] Optionally, the training of the transduction conversion network model based on contrast learning is specifically:

[0042] The encoding result of the enhanced transduction unit is taken as an input sample, the encoding result of the positive sample enhanced transduction unit is taken as a positive sample, and a semi-supervised learning loss function is used to train the transduction conversion network model based on contrast learning;

[0043] The semi-supervised learning loss function is specifically:

[0044]

[0045] Wherein, a is an anchor point, K is the encoding result of the enhanced transduction unit, tau is a hyperparameter, K + is the encoding result of the enhanced transduction unit for positive samples, P is the length of the negative sample queue plus the positive sample, exp(a K / tau) is the similarity score between the output result of the input sample and the anchor point, exp(K K + / tau) is the similarity score between the output result of the input sample and the positive sample result.

[0046] The present application has the following beneficial effects:

[0047] The present application can make other modalities except the input main modality also add attention labels to the input modality training by applying a transduction attention framework and an auto-encoder framework, so that the multi-modal data can be more naturally and effectively fused, thereby improving the sentiment prediction accuracy and model calculation efficiency of the transduction conversion network model for multi-modal data. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a flowchart of a multi-modal sentiment prediction method based on transduction contrast learning in an embodiment of the present application;

[0049] Figure 2 is a structural diagram of a transduction conversion network model based on contrast learning in an embodiment of the present application;

[0050] Figure 3 is a structural diagram of a transduction attention framework in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The specific embodiments of the present application are described below to facilitate understanding of the present application by those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all applications utilizing the concept of the present application are within the scope of protection.

[0052] As Figure 1 shown, the present application provides a multi-modal sentiment prediction method based on transduction contrast learning, comprising the following steps S1 to S5:

[0053] S1, obtaining multi-modal data containing emotional state, the multi-modal data comprising at least two of video modality information, text modality information, and audio modality information;

[0054] S2, constructing a transduction conversion network model based on contrast learning;

[0055] S3, a negative sample queue is constructed in a dictionary structure, and the negative sample queue is updated by using the sample of each iteration;

[0056] S4, training a cross-modal conversion network model based on contrastive learning by using the multi-modal data and the negative sample queue;

[0057] S5, using the trained cross-modal conversion network model based on contrastive learning for multi-modal sentiment prediction.

[0058] In the field, synesthesia, i.e. people with synesthesia will "hear" sounds with their eyes and "smell" music with their noses. People with synesthesia have more neural synapses from other modalities when receiving one modality than normal people. This means that when a certain mode is processed, the neural units of other sensory organs will also participate in this activity and jointly process the content. These additional labels can enable people with synesthesia to receive sensory information across sensory organs, so the present application believes that the MSA task can be improved by using this phenomenon to improve the training results. In this work, inspired by synesthesia, the present application proposes a contrastive learning mechanism that mimics human processing of multi-modal information, which is called a cross-modal converter. The model combines each basic modality with other modalities and merges them into a cross-modal unit through a cross-modal attention mechanism, and the cross-modal unit and the subsequent autoencoder will work together to mimic synesthesia. In the cross-modal unit, one modality is selected as the main body of the cross-modal unit, and the other modalities will be connected together through a neural network of an attention framework. Compared with the traditional method of taking a single modality as a basic unit, the cross-modal unit allows the model to mimic the emotional labels from other modalities like synesthesia, and such emotional labels are stable and effective in all modalities. The present application constructs an anchor sample, a positive sample and a negative sample queue when using contrastive learning, which has been proven to achieve good performance in the MSA task. The anchor sample can be constructed by obtaining the labeled dataset.

[0059] Such a method does not rely on human operation, but allows the model to generate positive samples by itself, which meets the natural attribute pursued by the present application.

[0060] In an optional embodiment of the present application, the cross-modal conversion network model based on contrastive learning constructed by the present application includes a cross-modal attention framework, an autoencoder and a conversion network encoder connected in sequence, as shown in Figure 2 .

[0061] The cross-modal attention framework is used to fuse the multi-modal data to generate a cross-modal unit and a positive sample cross-modal unit;

[0062] The automatic encoder is used for modal fusion on the transception unit and the positive sample transception unit respectively to obtain a strengthened transception unit and a positive sample strengthened transception unit.

[0063] The conversion network encoder encodes the strengthened transception unit and the positive sample strengthened transception unit respectively to obtain an encoding result.

[0064] The following will be described respectively for each part in the transception conversion network model.

[0065] As shown in the prior art, Figure 3 The transception attention framework in the application is used for fusing multi-modal data to generate a transception unit and a positive sample transception unit, and specifically comprises:

[0066] Sequentially, one kind of modal information is selected from the multi-modal data as a main modal, and other kinds of modal information are selected as additional attention modals;

[0067] The main modal and the additional attention modals are respectively fused by using a self-attention mechanism to obtain a first feature fusion vector;

[0068] The main modal and the first feature fusion vector are fused by using a self-attention mechanism to obtain a second feature fusion vector;

[0069] All the second feature fusion vectors are spliced to generate a sample output vector labeled by three kinds of modals simultaneously;

[0070] The sample output vector is subjected to a dropout mask operation using different dropout mask probabilities to generate a transception unit and a positive sample transception unit.

[0071] Specifically, as shown in the prior art, Figure 3 Taking multi-modal data composed of video modal information, text modal information and audio modal information as an example, one kind of modal is sequentially selected as a main modal alpha, and the other two modals beta and gamma are respectively fused by using a self-attention mechanism to obtain first feature fusion vectors and The main modal alpha and the first feature fusion vectors and are fused by using a self-attention mechanism to obtain second feature fusion vectors and

[0072] All the second feature fusion vectors and are spliced, and the values alpha β and alpha γ labeled by two basic modals are combined together to generate a sample output alpha labeled by three kinds of modals simultaneously(β，γ) The result output by the transduction attention framework is called a transduction unit, which is the basic unit of the model constructed by the application; then the sample output is output by using different dropout probabilities η and η + A dropout mask is performed to construct a positive sample. After the mask operation, the model successfully constructs a positive sample pair α by selecting different intermediate hidden layer features of dropout (β，γ) And α + (β，γ) , which is expressed as:

[0073]

[0074]

[0075] Wherein, Q α is the Q value of the main mode α as a self-attention mechanism, is the K value of the additional attention mode as a self-attention mechanism, T represents transposition, V α is the V value of the main mode α as a self-attention mechanism, d k is the dimension of the K value, η is the dropout mask probability, η + is the dropout mask probability of the positive sample, and softmax is an activation function, and concat is a concatenation operation.

[0076] The automatic encoder of the application is used for modal fusion of the transduction unit and the positive sample transduction unit respectively, to obtain the enhanced transduction unit and the positive sample enhanced transduction unit, and specifically comprises:

[0077] The automatic encoder comprises an encoder and a decoder constructed by using a full connection neural network respectively;

[0078] The encoder is used for mapping the sample output vector of the input transduction unit or positive sample transduction unit to a low-dimensional space for encoding, to obtain a low-dimensional vector unit of the transduction unit or positive sample transduction unit;

[0079] The decoder is used for projecting the low-dimensional vector unit of the transduction unit or positive sample transduction unit to a high-dimensional space for feature fusion, to obtain the enhanced transduction unit and the positive sample enhanced transduction unit.

[0080] Specifically, as Figure 2As shown, in order to enable good integration between modalities, the attention mechanism of the conversion network can be used to enable other modalities to add annotations to the main modality. Therefore, the application simulates the structure of neural synaptic connections in the synesthesia phenomenon, and adds an autoencoder to the constructed model to ensure that the output features are consistent with the input features through back propagation of the model, and also enables good integration between different modalities.

[0081] When processing each main modality, people with synesthesia will allow the nerves to have more synaptic connections to the nerve cells of different modalities. This neural connection enables other sensory organs to access nerve signals that do not belong to themselves, thereby enabling multiple modalities to share a neural network to process information, which effectively integrates each modality. Since the synesthesia neural synaptic connection belongs to a fully connected neural structure, the application simulates this multi-sensory participation through a fully connected neural network to construct an autoencoder, where the autoencoder is divided into an encoder and a decoder, and the basic modalities are video (V), audio (A), and text (T). Here, the main modality is denoted as alpha, and the two modalities with added attention are denoted as beta and gamma.

[0082] Assuming that the original single modality has a dimension of L α , L β , L γ , when they are processed into synesthesia units with alpha as the main modality, the dimension of the synesthesia unit becomes 2L α , in order to process the subsequent operations, when they are modality integrated through the autoencoder, the dimension of the output modality will be set to L α . The enhanced synesthesia unit alpha enhance(β，γ) and When passing through the autoencoder, the synesthesia units connected by simply splicing x together will be reduced in dimension by data, and after being reduced to the same low-dimensional space, the synesthesia units combined from the two annotated basic modalities can further integrate features, and then the decoder of the autoencoder will project them back to the high-dimensional space with a dimension of L α , at this time, the synesthesia unit with alpha as the main modality and the other basic modalities beta and gamma as the auxiliary is fully integrated, so that different modalities are naturally integrated together.

[0083] The conversion network encoder of the application encodes the enhanced synesthesia unit and the positive sample enhanced synesthesia unit respectively to obtain an encoding result, which specifically includes:

[0084] The conversion network encoder includes an original encoder and a positive sample encoder constructed by using a multi-head self-attention structure and a feedforward neural network structure, respectively;

[0085] The original encoder is used for performing hierarchical normalization operation on the enhanced transductive unit first, then passing through the multi-head self-attention structure, and performing residual connection between the input of the normalization layer and the output of the multi-head self-attention structure; after performing hierarchical normalization operation, passing through the feedforward neural network structure, and performing residual connection between the input of the normalization layer and the output of the feedforward neural network structure, the encoding result of the enhanced transductive unit is obtained.

[0086] The positive sample encoder is used for performing hierarchical normalization operation on the positive sample enhanced transductive unit first, then passing through the multi-head self-attention structure, and performing residual connection between the input of the normalization layer and the output of the multi-head self-attention structure; after performing hierarchical normalization operation, passing through the feedforward neural network structure, and performing residual connection between the input of the normalization layer and the output of the feedforward neural network structure, the encoding result of the positive sample enhanced transductive unit is obtained.

[0087] Specifically, the enhanced transductive unit α enhance(β，γ) and is sent into the original encoder e nor and the positive sample encoder e simi for operation processing. In order to make the two enhanced transductive units converge faster during training, the input enhanced transductive unit α enhance(β，γ) and in the model will first undergo hierarchical normalization operation to normalize the feature vector, and then pass through the main part of the multi-head self-attention structure. Unlike the previous transductive attention structure, in the transition network encoder, the input Q, K and V values are all the enhanced homophilic units output by the autoencoder, after the enhanced transductive unit is processed by the multi-head self-attention structure, residual connection is performed to ensure the stability of the parameters, and then the feedforward neural network is used for result prediction.

[0088] In this process, the input modalities pass through the transductive attention mechanism, and although the long queue negative sample has better robustness than the limited negative sample, it also brings problems to the optimization of the positive sample encoder. The main problem is that: assuming that the input sample and the positive sample have the original encoder e nor and the positive sample encoder e simi, the parameters of the known sample encoder are constantly optimized and improved, and the specific parameters of the samples in the queue are calculated by the encoder that has not been updated before, so there are differences in the numerical values between the previous encoder parameters and the current encoder parameters. When calculating the contrast loss in the current running period, the sample data in the queue will return the gradient generated by the previous sample encoder to the current sample encoder, which will cause problems in the optimization process of the model. In other models that select limited negative samples, the difference is small and does not have too much impact on model training, but in this model, the negative samples are a long queue, and the difference of a large number of negative samples is likely to affect the training of model parameters. It is known that the negative samples in the queue come from different sequence batches, and because all the samples in the queue will be calculated for gradient, a large amount of data from different batches will make it difficult for the encoder to update and converge in the process of back propagation, like a normal model.

[0089] To prevent the model output of the original sample and the positive sample from deviating due to the violent jitter of the encoder, so that the modal between the two loses feature consistency, the application sets the positive sample encoder not to participate in gradient calculation, but to change its parameters with the parameters of the original encoder. If the parameters of the original encoder are directly copied to the positive sample encoder, the parameters of the two will converge, and only the sample data will be different, which is easy to cause overfitting problem. Therefore, the application adopts the momentum optimization method to update the parameters of the positive sample encoder with the parameters of the original encoder, and the update formula is:

[0090] ω simi ′←kω simi +(1-k)ω nor

[0091] Wherein, ω simi ′ is the updated parameter of the positive sample encoder, ω simi is the current parameter of the positive sample encoder, ω nor is the current parameter of the original encoder, and k is the hyperparameter of parameter update. Only the parameter ω nor is updated by back propagation, and the momentum update makes the evolution of ω simi more stable than ω nor . A slow evolution momentum optimization structure well solves the problem of gradient decline failure caused by the violent fluctuation of the negative sample queue, successfully makes the shortcomings of the negative sample queue no longer affect the model, but become a kind of enhancement for model training.

[0092] The momentum structure makes the parameters of the positive sample only affected by the parameters of the original sample encoder each time instead of being directly replaced, so that the model of the positive sample can converge more stably and smoothly compared with the parameters of the original sample, and meanwhile, the distortion of the model parameters caused by the sudden and violent fluctuation of the negative sample can be avoided, and the final result of the experiment can not be affected by the difference of the negative sample.

[0093] In an optional embodiment of the present application, step S3 adopts a dictionary structure to construct the negative sample queue, and the negative sample queue is updated by using the sample of each iteration, specifically including:

[0094] The negative sample queue is constructed by using a dictionary structure, and after each model iteration is completed, the small batch of samples of the current input multi-modal data is encoded to replace the small batch of samples with the earliest iteration time in the negative sample queue.

[0095] Specifically, the present application selects a dictionary {N1, N2, N3,...} that can accommodate a long enough negative sample queue as a negative sample, assumes that each input sequence is a small batch, and arranges the encoding representation of the samples in the small batch participating in the training in the queue. It is assumed in the experiment that the samples in the small batch that have been trained previously are negative samples of the samples in the current small batch being processed. All negative samples are obtained from the small batch samples that have been processed previously, and the current small batch only contains positive sample data. Therefore, the updating method of the negative sample queue is to encode the samples in the current small batch to replace the oldest small batch in the queue after the current model processing iteration is completed. In principle, it is a typical first-in-first-out method. Using the queue method can make the size of the dictionary not affected by the capacity of the small batch, so that a larger dictionary can be used, more negative samples can be calculated in each contrast loss, and a long queue can be selected as a negative sample for calculation, which conforms to the natural properties of the model required in this paper. A large number of negative samples can also make the model more accurately move away from the wrong direction during the experiment, so as to improve the quality of the representation.

[0096] In an optional embodiment of the present application, step S4 trains the transmodal conversion network model based on the contrast learning, and adopts a semi-supervised learning loss function for model training, and the semi-supervised learning loss function is specifically:

[0097]

[0098] Wherein, a is an anchor point, K is an encoding result of the enhanced transmodal unit, τ is a hyperparameter, K + is an encoding result of the positive sample enhanced transmodal unit, P is the length of the negative sample queue plus the positive sample, exp(a·K / τ) is a similarity score between the output result of the input sample and the anchor point, and exp(K·K +is the similarity score between the output of the input sample and the positive sample result.

[0099] Specifically, the loss function of contrastive learning usually focuses on pushing away the distance of negative samples and pulling closer the distance of positive samples. Therefore, how to construct a function that can take into account both requirements is an important problem to be solved. The present application constructs a loss function for semi-supervised learning as follows:

[0100]

[0101] Where a is an anchor point representing the correct result stored in the database, and K represents the final output. Because the anchor points output by the database are arranged in the order of T, V, and A, the final output K is arranged in order to align with the experimental evaluation. The actual group is the result of concatenating the values of T final(V，A) , V final(A，T) , and A final(V，T) . K + is the concatenation of the corresponding positive sample results. Where τ is defined as a temperature coefficient, i.e. a hyperparameter, which can be adjusted by changing its value to adjust the speed of distance movement during gradient training. P is the length of the negative sample queue plus the positive sample. Where exp(a·K / τ) is the similarity score between the original sample output result and the anchor point, and exp(K·K + / τ) is the similarity score between the original sample output result and the positive sample result. The former allows the model to have a direction like the anchor point during model training, so that the model can normally perform gradient descent. The latter can be regarded as a clustering algorithm that allows similar original sample outputs and positive sample outputs to be closer, thereby improving experimental results. And is the set containing the anchor point and the negative sample queue, which is placed in the denominator to make the model move away from the negative sample during training. To make the model converge well, the similarity score between the original sample output result and the anchor point and the positive sample should be higher, while the similarity score between the anchor point and the negative sample should be lower.

[0102] The loss function constructed by this method can help the function to converge better, because the characteristics of the loss function allow the model to give a greater moving speed to the sample when it encounters a similar positive sample as a negative sample, as shown below:

[0103] Now find the gradient of the loss function with respect to the random negative sample s i，j (i≠k), the result is:

[0104]

[0105] As can be seen, the denominator of the public is always consistent, at this time the greater the numerator, i.e. exp(a·K / τ)+exp(K·K+ The larger the / r) is, the larger the gradient is, that is, the more similar the negative sample and the positive sample are, a larger gradient away from the positive sample will be obtained for the loss function, and this property can help the model to be better trained and optimized.

[0106] To better highlight the function of the model and reflect the actual effect of the model on sentiment analysis, we test and compare the unaligned version of CMU-MOSI and CMU-MOSEI with a series of state-of-the-art works, and the results are shown in Tables 1 and 2. In the evaluation list, we add the data results of the above baseline models to the table for better comparison of the results.

[0107] Table 1

[0108]

[0109] Table 2 Comparison of CMU-MOSEI results

[0110]

[0111] From the above table, it can be seen that the model proposed by us has obtained competitive results in the main indicators. Specifically, it can be seen that the model has obtained the best results in the F1 score of the two data sets and the accuracy of the CMU-MOSEI data set. The output data of the model on CMU-MOSI has increased by 2.1% in the Acc-7 standard compared with the MULT model which has the best result in the above baseline models. And the F1 score is 1.4% higher than the MULT model, and the F1 score of the output data of the model on CMU-MOSEI is 0.2% higher than the highest baseline model, which proves the effectiveness of the model constructed by us.

[0112] The ablation experiment is to explore the optimization effect of each basic structure on the experimental data in our model, and we conduct comprehensive ablation experiments on the unaligned version of the two data sets of CMU-MOSEI and CMU-MOSI, and judge whether each structure helps to improve the model by extracting each structure. All results are shown in Tables 3 and 4.

[0113] First, we test the performance of STC in the single modality data text, video, audio, represented as STC-Text, STC-Video and STC-Audio. We can note that when only the text modality is involved in training, its performance is better than the performance of the other two modalities trained alone. On the CMU-MOSI dataset, the F1 score of STC-Text is 6.7% higher than that of STC-Video, 9.9% higher than that of STC-Audio, the Acc-1 performance is 9.6% higher than that of STC-Video, 10.9% higher than that of STC-Audio, and the mean absolute error (MAE) is 0.107 lower than that of STC-Video, 0.112 lower than that of STC-Audio. In terms of correlation, it is 0.248 higher than that of STC-Video, 0.281 higher than that of STC-Audio.

[0114] On the CMU-MOSEI dataset, the F1 score of STC-Text is 7.4% and 9.1% higher than that of STC-Video and STC-Audio respectively, the Acc-1 performance is 8.9% higher than that of STC-Video, 10.2% higher than that of STC-Audio. In terms of correlation, STC-Text also performs better than other modes, 0.206 higher than that of STC-Video, 0.171 higher than that of STC-Audio. Such trends can also be found in previous work.

[0115] In addition, the gap between the text modality and other modalities is smaller than the previous research results, which to some extent proves that the transmodal attention framework and the autoencoder structure can help the single modality data in the form of transmodal unit to obtain more feature information than ordinary single modality to obtain better training results.

[0116] Table 3 CMU-MOSI ablation experiment

[0117]

[0118] Table 4 CMU-MOSEI ablation experiment

[0119]

[0120] We then evaluate the model without the autoencoder involved in the training (STC-WA). We observe that the performance drops significantly when the autoencoder structure is not included in the model. On the CMU-MOSEI dataset, the F1 score drops from 81.2% to 75.8%, and on the CMU-MOSI dataset, it drops by 10.1%. Meanwhile, the mean absolute error (MAE) of STC-WA is 9% higher than STC on the CMU-MOSEI dataset and nearly 18% higher on the CMU-MOSI dataset. The Acc-1 of STC-WA drops by 4.8% on the CMU-MOSI dataset and 2.8% on the CMU-MOSEI dataset compared to STC. Finally, the correlation of STC-WA is 0.006 lower than STC on the CMU-MOSI dataset and 0.035 lower on the CMU-MOSEI dataset.

[0121] We believe that the use of the autoencoder provides better fusion of different modalities with other modalities, which cannot be ignored compared to when the autoencoder is not used. The simple concatenation of the three modalities does not allow them to effectively fuse, resulting in less enhancement of the model.

[0122] Next, we study the enhancement effect of the cross-modal attention framework (STC-WS). According to Tables 3-3 and 3-4, we can see that when only using single-modal data as input, the results show that the accuracy-1 drops by 2% and the F1 score drops by 2% on the CMU-MOSEI dataset. On the CMU-MOSI, the acc-1 drops by 0.4% and the F1 score drops by 4%. Therefore, it proves the effectiveness of the proposed cross-modal attention framework through data.

[0123] Finally, we compare the difference between using supervised learning and semi-supervised learning (STC-SL). That is, when only using the anchor points and the original encoder output in the dataset for contrastive learning without considering positive samples, the output is not as good as when positive samples are included. At this time, the model becomes supervised learning. The process is as follows.

[0124]

[0125] Through testing, we found that the accuracy of the model without positive samples on CMU-MOSEI decreased to 76% and on CMU-MOSI decreased to 75.2% at 1. This shows that compared with supervised contrast learning, semi-supervised contrast learning provides anchor data, which helps the model converge in the correct direction to a certain extent, proving the effectiveness of semi-supervised contrast learning.

[0126] The present application is described in reference to the flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus for implementing functions specified in one or more flows and / or blocks.

[0127] These computer program instructions can also be stored in a computer-readable memory capable of guiding the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus for implementing functions specified in one or more flows and / or blocks.

[0128] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus for implementing functions specified in one or more flows and / or blocks.

[0129] The principles and implementation modes of the present application are described in the specific embodiments in the present application, and the above embodiment description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application.

[0130] Those skilled in the art will appreciate that the embodiments described herein are presented for purposes of illustration and that the inventive principles are not limited to these particular embodiments. Other variations and modifications can be made to the embodiments without departing from the spirit and scope of the inventive principles.

Claims

1. A multi-modal sentiment prediction method based on transductive contrastive learning, characterized in that, The method comprises the following steps: Obtaining multi-modal data containing emotional states, the multi-modal data comprising at least two of video modal information, text modal information and audio modal information; Building a transmodal conversion network model based on contrast learning, comprising a transmodal attention framework, an auto-encoder and a conversion network encoder connected in sequence; The transmodal attention framework is used to fuse the multi-modal data to generate a transmodal unit and a positive sample transmodal unit, and one kind of modal information is selected as a main modal from the multi-modal data in sequence, and other kinds of modal information are selected as additional attention modal; The main modal and the additional attention modal are fused by using a self-attention mechanism to obtain a first feature fusion vector; The main modal and the first feature fusion vector are fused by using a self-attention mechanism to obtain a second feature fusion vector; All the second feature fusion vectors are spliced to generate a sample output vector with three modalities participating in labeling at the same time; A dropout mask operation is performed on the sample output vector using different dropout mask probabilities to generate a transmodal unit and a positive sample transmodal unit; The auto-encoder is used to fuse the transmodal unit and the positive sample transmodal unit to obtain a strengthened transmodal unit and a positive sample strengthened transmodal unit; The conversion network encoder encodes the strengthened transmodal unit and the positive sample strengthened transmodal unit to obtain an encoding result; A negative sample queue is constructed using a dictionary structure, and the negative sample queue is updated using the samples of each iteration; The transmodal conversion network model based on contrast learning is trained using the multi-modal data and the negative sample queue; The trained transmodal conversion network model based on contrast learning is used for multi-modal emotion prediction.

2. The multi-modal sentiment prediction method based on common sense contrastive learning according to claim 1, characterized in that, The transmodal unit is represented as: where, is the main modality α as the value in the self-attention mechanism Q , is the additional attention modality as the value in the self-attention mechanism K , is the additional attention modality as the value in the self-attention mechanism K , T denotes the transpose, is the main modality α as the value in the self-attention mechanism V , is the K dimension of the value, is the dropout mask probability, is the activation function, is the concatenation operation, is the modality labeled value and are combined together to generate the sample output with all three modalities participating in the labeling.

3. The multi-modal sentiment prediction method based on common sense contrastive learning according to claim 2, characterized in that, The positive sample transmodal unit is represented as: wherein, is the output for positive samples, is the main modality α is the Q value, is the additional attention modality as the K value, T denotes the transpose, is the main modality α is the V value, is the K dimension of the value, is the dropout mask probability for positive samples, is the activation function, is the concatenation operation.

4. The multi-modal sentiment prediction method based on common sense contrastive learning according to claim 1, characterized in that, The auto-encoder is used to fuse the transmodal unit and the positive sample transmodal unit to obtain a strengthened transmodal unit and a positive sample strengthened transmodal unit, and specifically comprises: The auto-encoder comprises an encoder and a decoder constructed using a fully connected neural network respectively; The encoder is used to map the sample output vector of the input transmodal unit or positive sample transmodal unit to a low-dimensional space for encoding to obtain a low-dimensional vector unit of the transmodal unit or positive sample transmodal unit; The decoder is used to project the low-dimensional vector unit of the transmodal unit or positive sample transmodal unit to a high-dimensional space for feature fusion to obtain a strengthened transmodal unit and a positive sample strengthened transmodal unit.

5. The multi-modal sentiment prediction method based on common sense contrastive learning according to claim 1, characterized in that, The conversion network encoder encodes the strengthened transmodal unit and the positive sample strengthened transmodal unit to obtain an encoding result, and specifically comprises: The conversion network encoder comprises an original encoder and a positive sample encoder constructed using a multi-head self-attention structure and a feedforward neural network structure respectively; The original encoder is used to perform hierarchical normalization operation on the enhanced transduction unit first, then pass through the multi-head self-attention structure, and perform residual connection between the input of the normalization layer and the output of the multi-head self-attention structure; after performing hierarchical normalization operation, pass through the feedforward neural network structure, and perform residual connection between the input of the normalization layer and the output of the feedforward neural network structure, to obtain the encoding result of the enhanced transduction unit; The positive sample encoder is used to perform hierarchical normalization operation on the positive sample enhanced transduction unit first, then pass through the multi-head self-attention structure, and perform residual connection between the input of the normalization layer and the output of the multi-head self-attention structure; after performing hierarchical normalization operation, pass through the feedforward neural network structure, and perform residual connection between the input of the normalization layer and the output of the feedforward neural network structure, to obtain the encoding result of the positive sample enhanced transduction unit.

6. The multi-modal sentiment prediction method based on common sense contrastive learning according to claim 5, characterized in that, The conversion network encoder adopts a momentum optimization method, and updates the parameters of the positive sample encoder using the parameters of the original encoder, and the update formula is: wherein, are the parameters of the updated positive sample encoder, are the parameters of the current positive sample encoder, are the parameters of the current raw encoder, k are hyperparameters for the parameter update.

7. The multi-modal sentiment prediction method based on common sense contrastive learning according to claim 1, characterized in that, The negative sample queue is constructed by using a dictionary structure, and the negative sample queue is updated using the samples of each iteration, specifically including: The negative sample queue is constructed by using a dictionary structure, and after each model iteration is completed, the small batch of samples of the current input multi-modal data are encoded to replace the small batch of samples with the earliest iteration time in the negative sample queue.

8. The multi-modal sentiment prediction method based on common sense contrastive learning according to claim 1, characterized in that, Training the transduction conversion network model based on contrast learning is specifically: The encoding result of the enhanced transduction unit is used as an input sample, and the encoding result of the positive sample enhanced transduction unit is used as a positive sample, and a semi-supervised learning loss function is used to train the transduction conversion network model based on contrast learning; The semi-supervised learning loss function is specifically: wherein, a is an anchor point, K is an encoding result of the reinforced empathic unit, is a hyper-parameter, is an encoding result of the reinforced empathic unit for positive samples, P is the length of the positive samples added to the negative sample queue, is a similarity score between the output result of the input sample and the anchor point, is a similarity score between the output result of the input sample and the positive sample result; is a summation term, representing the encoding result of the reinforced empathic unit corresponding to the i-th input sample.

Images