Multimodal sentiment analysis method based on modal representation space and superimposed self-attention

Abstract

This paper proposes a multimodal sentiment analysis method based on modal representation space and superimposed self-attention to address technical issues such as the semantic gap faced by existing methods when processing multimodal data. The method comprises the following steps: first, extracting feature vectors of the three modalities, text, speech, and image, from the original video data; then, using a modal representation space segmentation method, mapping the feature vectors of the three modalities to the sentiment representation space and noise representation space, respectively, to obtain modal sentiment representations and modal noise representations; second, passing the obtained modal sentiment representations into a superimposed self-attention mechanism to mine the correlation information between the modal sentiment representations and perform concatenation; finally, under a joint loss constraint, passing the concatenated modal sentiment representations into the sentiment prediction classification layer for classification prediction, outputting the optimal sentiment prediction value. This method can bridge the semantic gap in multimodal sentiment analysis and obtain more reliable sentiment prediction results.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal sentiment analysis, and in particular to a multimodal sentiment analysis method based on modal representation space and superimposed self-attention. Background Art

[0002] With the growing trend toward information diversification, sentiment computing research has evolved from unimodal analysis, primarily focused on text, to multimodal analysis encompassing multiple sources of information, including text, speech, and images. However, due to the heterogeneity of multi-source data, traditional sentiment analysis methods struggle to bridge the semantic gap between modalities when processing multimodal data. A new approach is urgently needed to overcome the technical bottleneck in multimodal sentiment analysis.

[0003] At present, the main approach to solving the multimodal semantic gap is feature fusion. Through the fusion method, the independent feature vectors of multiple modalities are merged into a multimodal feature vector. Using this multimodal feature vector for sentiment prediction can narrow the semantic gap to a certain extent. However, this method has two obvious problems: (1) Multi-source heterogeneous data are distributed in different modal representation spaces. The barriers between different spaces make it difficult to achieve tight coupling of the feature vectors of each modality; (2) The information interaction within and outside the modality is not taken into account during the fusion process, which makes it easy to ignore some potential sentiment-related information during the sentiment analysis process.

[0004] Therefore, it is necessary to design a new method that can solve the problem of modal representation space barriers and multimodal information interaction problems in order to break the shackles that hinder the development of multimodal sentiment analysis technology. Summary of the Invention

[0005] To address the challenges of modal representation space barriers and multimodal information interaction in existing multimodal sentiment analysis methods, this paper proposes a multimodal sentiment analysis method based on modal representation space and superimposed self-attention. This method first splits each modal representation space into a sentiment representation space and a noise representation space to break down the barriers between the different spaces. Then, through a superimposed self-attention mechanism, connections are established between the various representation spaces, thereby learning modal representations that contain information about the interactions between the modalities. The multimodal sentiment analysis model constructed using this method can fully account for the interaction between multiple modal information, thereby bridging the semantic gap between multiple modalities and improving the model's sentiment classification performance.

[0006] The technical solution of the present invention is achieved as follows:

[0007] A multimodal sentiment analysis method based on modal representation space and superimposed self-attention, the steps are as follows:

[0008] Step 1: Extract feature vectors of the three modalities of text, speech, and image from the original video data;

[0009] Step 2: Use the modal representation space segmentation method to map the feature vectors of the three modalities of text, speech, and image to the emotion representation space and noise representation space respectively to obtain the modal emotion representation and modal noise representation;

[0010] Step 3: The obtained modal sentiment representation is passed into the superimposed self-attention mechanism to mine the correlation information between the modal sentiment representations and splice the modal sentiment representations containing the correlation information;

[0011] Step 4: Use the modal emotion representation loss, modal noise representation loss, and classification loss as joint loss constraints. Under the joint loss constraints, pass the concatenated modal emotion representation into the emotion prediction classification layer for classification prediction, and output the optimal emotion prediction value.

[0012] Preferably, in step 2, the specific implementation method is:

[0013] S21, use the sentiment encoder to respectively transform the text feature vector U t , speech feature vector U a and image feature vector U v Enter the sentiment representation space and obtain the text sentiment representation S with a size of 1*d t , Speech Emotion Representation S a and image sentiment representation S v , the specific calculation formula is:

[0014] S m =E s (U m θ s );

[0015] Where m∈{t,a,v}; E s represents the emotion encoder, θ s are the parameters of the emotion encoder;

[0016] S22, using the text noise encoder to transform the text feature vector U t Enter the text noise space and get the text noise representation P of size 1*d t , the specific calculation formula is:

[0017]

[0018] in, represents a text noise encoder, are the parameters of the text noise encoder;

[0019] S23, with the help of speech noise encoder, the speech feature vector U a Enter the speech noise space and get the speech noise representation P of size 1*d a , the specific calculation formula is:

[0020]

[0021] in, represents the speech noise encoder, are the parameters of the speech noise encoder;

[0022] S24, using the image noise encoder to transform the image feature vector U v Enter the image noise space and get the image noise representation P of size 1*d v , the specific calculation formula is:

[0023]

[0024] in, represents the image noise encoder, It is a parameter unique to the image noise encoder.

[0025] Preferably, in step three, the specific implementation method is:

[0026] S31. Given three text modality weight matrices W of size d*d t1 ,W t2 ,W t3 , respectively, with the text sentiment representation S t Multiply to get three initialized text sentiment representations S t1 ,S t2 ,S t3 , the specific calculation formula is:

[0027] S ti' =S t W ti' ;

[0028] Where, i'∈{1,2,3};

[0029] S for text sentiment representation t1 ,S t2 Do the dot multiplication and pass the Relu(·) activation function to obtain the self-attention score matrix A of size d*d t , the specific calculation formula is:

[0030]

[0031] in, For S t2 The transpose of

[0032] The text sentiment is represented by S t3 and the self-attention score matrix A t Multiplying them together yields the self-attention result S′, which contains the internal correlation information of the text modality. t , the specific calculation formula is:

[0033] S′ t =S t3 A t ;

[0034] S32, given three speech modality weight matrices W of size d*d a1 ,W a2 ,W a3 , respectively, with the speech emotion representation S a Multiply to get three initialized speech emotion representations S a1 ,S a2 ,S a3 , the specific calculation formula is:

[0035] S ai' =S a W ai' ;

[0036] Speech emotion representation S a1 ,S a2 Do a dot multiplication and pass it through the Relu(·) activation function to obtain the self-attention score matrix A of size d*d a , the specific calculation formula is:

[0037]

[0038] The speech emotion is represented by S a3 and the self-attention score matrix A a Multiplying them together yields the self-attention result S′, which contains the internal correlation information of the speech modality. a , the specific calculation formula is:

[0039] S′ a =S a3 A a ;

[0040] S33, given three image modality weight matrices W of size d*d v1 ,W v2 ,W v3 , respectively, with the image emotion representation S v Multiply to get three initialized image emotion representations S v1 ,S v2 ,S v3 , the specific calculation formula is:

[0041] Svi' =S v W vi' ;

[0042] The emotion representation S of the image v1 ,S v2 Do a dot multiplication and pass it through the Relu(·) activation function to obtain the self-attention score matrix A of size d*d v , the specific calculation formula is:

[0043]

[0044] The image emotion is represented by S v3 and the self-attention score matrix A v Multiply to get the self-attention result S′ which contains the internal correlation information of the image modality v , the specific calculation formula is:

[0045] S′ v =S v3 A v ;

[0046] S34. Given three multimodal weight matrices W of size 3d*3d m1 ,W m2 ,W m3 , respectively, and S′ t ,S′ a ,S′ v The concatenated multimodal representations are multiplied to obtain three initialized multimodal emotion representations S m1 ,S m2 ,S m3 , the specific calculation formula is:

[0047] S mi' =(S′ t ; S′ a ; S′ v )W mi' ;

[0048] For multimodal sentiment representation S m1 ,S m2 Do a dot multiplication and pass it through the Relu(·) activation function to obtain a self-attention score matrix A of size 3d*3d m , the specific calculation formula is:

[0049]

[0050] The multimodal sentiment representation S m3 and the self-attention score matrix A m Multiplying them together yields the self-attention result S′, which contains the correlation information between the modalities. m, the specific calculation formula is:

[0051] S′ m =S m3 ·A m .

[0052] Preferably, the modality emotion representation loss is:

[0053]

[0054] Among them, L s is the sentiment representation loss, (m,n)∈{(t,a),(t,v),(a,v)}, d represents the number of elements in the feature vector, represents the square of the 2-norm;

[0055] The modal noise represents the loss:

[0056]

[0057]

[0058] Among them, L p′ represents the intermediate variable, L p is the noise representation loss, i∈{l,a,v},(x,z)∈{(t,a),(t,v),(a,t),(a,v),(v,t),(v,a)}.

[0059] The classification loss is:

[0060]

[0061] Among them, L class is the cross entropy classification loss between the predicted value and the true value of the neural network, y represents the true value of the sample, represents the predicted value;

[0062] Joint cross entropy classification loss L class , emotional expression loss L s , noise representation loss L p Together as the total task loss L task To constrain the training process, the specific calculation formula is:

[0063] L task =αL class +βL s +γL p ;

[0064] Among them, α, β, and γ all represent the weight of the loss.

[0065] Preferably, the classification method of the emotion prediction classification layer is:

[0066] S′ m Input into a three-layer neural network, the input layer and hidden layer size of the three-layer neural network are both 3d, the output layer size is 2, and the sigmoid (·) activation function is used to implement emotion 2 classification; the specific calculation formula is:

[0067]

[0068] in, represents the predicted value, F(·) represents the function expression of the neural network, θ m Represents the parameter set between each neuron.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] (1) The present invention projects the feature vectors of the three modalities from the original representation space into an emotion representation space and three noise representation spaces through modal representation space segmentation. The emotion representation space breaks the spatial barriers between the original representation spaces and removes the redundant parts between the original feature vectors to the greatest extent, so that the obtained three modal emotion representations retain the most consistent information between the modalities.

[0071] (2) The present invention superimposes the self-attention mechanism and performs self-attention calculations on the three modal sentiment representations respectively, thereby mining the detailed interaction information within a single modality. The results are then concatenated and self-attention calculations are performed again to mine the global interaction information between different modalities. This allows the multimodal representation finally obtained to bridge the semantic gap brought about by multi-source heterogeneous data during sentiment analysis.

[0072] (3) The present invention combines the classification loss of the emotion prediction classification layer with the emotion representation loss and noise representation loss of spatial segmentation, optimizes the training process through joint loss constraints, and ultimately improves the effect of emotion prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0074] Figure 1 It is a schematic diagram of the overall model of the present invention.

[0075] Figure 2 Schematic diagram of the modal representation space segmentation according to the present invention.

[0076] Figure 3 Schematic diagram of the superposition attention mechanism described in the present invention. DETAILED DESCRIPTION

[0077] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0078] The hardware environment used to implement this invention is: AMD Ryzen 7 5800 3.4GHz CPU, 16GB memory, GTX3060ti 12G graphics card, and the software environment is: Windows 10, Python 3.6, CUDA 10.2. In addition, the present invention uses the MOSI dataset published by Carnegie Mellon University to test the specific performance of the embodiments of the present invention.

[0079] like Figure 1 As shown, an embodiment of the present invention provides a multimodal sentiment analysis method based on modal representation space and superimposed self-attention, wherein the modal representation space segmentation method breaks the spatial barriers by dividing each modal representation space into a sentiment representation space and a noise representation space, and can remove the redundant parts between the original feature vectors to the greatest extent, so that the obtained three modal sentiment representations retain the most consistent information between the modalities, providing a new research idea for achieving tight coupling of multimodal features; the proposed superimposed self-attention mechanism can not only mine the interactive information within a single modality, but also capture the global correlation between different modalities, providing a new research solution for bridging the semantic gap between multimodal data; the specific steps are as follows:

[0080] Step 1: Extract feature vectors of the three modalities of text, speech, and image from the original video data; split the complete video into three categories of data: text, sound, and visual images; extract feature vectors from the original data of the three modalities of text, speech, and image;

[0081] For raw data in text mode, we use the pre-trained BERT model to extract text features. This model consists of 12 stacked Transformer frameworks, which can capture contextual information in both directions, making the generated text features more representative. Finally, the raw text data is converted into multiple text feature vectors U of size 1*d. t .

[0082] For the raw data of the speech modality, the Librosa toolkit is used to extract speech features. This toolkit can sample and analyze the spectral features of audio data. The Mel-frequency cepstral coefficients contain rich emotional information, which is extracted as acoustic features using this toolkit. Finally, the raw audio data is converted into multiple speech feature vectors U of size 1*d. a .

[0083] For the original data of the image modality, the Openface 2.0 toolkit is used to identify the facial information in the image, and then a series of facial expression information such as facial action units, head orientation and eye direction are extracted as image features. Finally, the original image data is converted into multiple image feature vectors U of size 1*d v .

[0084] Step 2: Use the modal representation space segmentation method to map the feature vectors of the three modalities of text, speech and image to the emotion representation space and noise representation space respectively to obtain modal emotion representation and modal noise representation; map the feature vectors of the three modalities to one emotion representation space and three noise representation spaces respectively, and during the spatial segmentation, use the constraints of emotion representation loss and noise representation loss to make the emotion representations of the three modalities distributed in the emotion representation space as close as possible, while the noise representations in the text noise representation space, speech noise representation space and image noise representation space are as far as possible, thereby learning better modal emotion representation in the emotion space to break the spatial barriers.

[0085] In step 2, the specific implementation method is:

[0086] S21, use the sentiment encoder to respectively transform the text feature vector U t , speech feature vector U a and image feature vector U v Enter the sentiment representation space and obtain the text sentiment representation S with a size of 1*d t , Speech Emotion Representation S a and image sentiment representation S v , the specific calculation formula is:

[0087] S m =E s (U m θ s );

[0088] Where m∈{t,a,v}; E s θ represents the emotion encoder, which is composed of a feedforward neural network with an input size of d and an output size of d. s are the parameters of the emotion encoder, and the feature vectors of the three modalities use the same set of parameters when entering the emotion representation space.

[0089] S22, using the text noise encoder to transform the text feature vector U t Enter the text noise space and get the text noise representation P of size 1*d t , the specific calculation formula is:

[0090]

[0091] in, represents a text noise encoder, which is composed of a feedforward neural network with an input size of d and an output size of d. are the parameters of the text noise encoder;

[0092] S23, with the help of speech noise encoder, the speech feature vector U a Enter the speech noise space and get the speech noise representation P of size 1*d a , the specific calculation formula is:

[0093]

[0094] in, represents a speech noise encoder, which is composed of a feedforward neural network with an input size of d and an output size of d. are the parameters of the speech noise encoder;

[0095] S24, using the image noise encoder to transform the image feature vector U v Enter the image noise space and get the image noise representation P of size 1*d v , the specific calculation formula is:

[0096]

[0097] in, represents an image noise encoder, which is composed of a feedforward neural network with an input size of d and an output size of d. It is a parameter unique to the image noise encoder.

[0098] Calculate the sentiment representation loss L s By constraining the representation, the emotional representations of the three modalities distributed in the emotional representation space are made as close as possible to mine the common emotional information between the three modal data (the emotional representation loss will be used in the joint loss constraint in step 4). The specific calculation formula is:

[0099]

[0100] Among them, L sis the sentiment representation loss, (m,n)∈{(t,a),(t,v),(a,v)}, d represents the number of elements in the feature vector, represents the square of the 2-norm;

[0101] Calculate the noise representation loss L p By constraining the representation, the distance between the noise representations in the three noise representation spaces is made as far as possible, and the distance between the emotion representation of any modality and the noise representation is also made as far as possible, so as to maximize the separation of irrelevant noise information (the noise representation loss will be used in the joint loss constraint in step 4). The specific calculation formula is:

[0102]

[0103]

[0104] Among them, L p is the noise representation loss, L p′ Represents the intermediate variable, which represents the fusion of the three distance metrics: "distance metric between cross-modal emotion representations", "distance metric between same-modal emotion representation and noise representation", and "distance metric between cross-modal emotion representation and noise representation". The larger the distance metric, the greater the difference between the two types of representations (that is, the better the noise classification effect). In order to use it as a noise loss to constrain the training process, L is selected. p′ The reciprocal L p As the noise representation loss, the greater the difference between the two types of representations in training, the smaller the noise loss, and the better the training effect, i∈{l,a,v},(x,z)∈{(t,a),(t,v),(a,t),(a,v),(v,t),(v,a)}.

[0105] Step 3: The obtained modal sentiment representations are passed into the superimposed self-attention mechanism to mine the correlation information between the modal sentiment representations, bridge the semantic gap between the modalities, and splice the modal sentiment representations containing the correlation information;

[0106] In step three, the specific implementation method is:

[0107] S31. Given three text modality weight matrices W of size d*d t1 ,W t2 ,W t3 , respectively, with the text sentiment representation S t Multiply to get three initialized text sentiment representations S t1 ,S t2 ,S t3 , the specific calculation formula is:

[0108] S ti' =St W ti' ;

[0109] Among them, i'∈{1,2,3}; S ti' The size is 1*d.

[0110] S for text sentiment representation t1 ,S t2 Do the dot multiplication and pass the Relu(·) activation function to obtain the self-attention score matrix A of size d*d t , the specific calculation formula is:

[0111]

[0112] in, For S t2 The transpose of

[0113] The text sentiment is represented by S t3 and the self-attention score matrix A t Multiply to get the self-attention result S′ which contains the internal correlation information of the text modality t , the specific calculation formula is:

[0114] S′ t =S t3 A t ;

[0115] S32, given three speech modality weight matrices W of size d*d a1 ,W a2 ,W a3 , respectively, with the speech emotion representation S a Multiply to get three initialized speech emotion representations S a1 ,S a2 ,S a3 , the specific calculation formula is:

[0116] S ai' =S a W ai' ;

[0117] Among them, S ai' The size is 1*d.

[0118] Speech emotion representation S a1 ,S a2 Do a dot multiplication and pass it through the Relu(·) activation function to obtain the self-attention score matrix A of size d*d a , the specific calculation formula is:

[0119]

[0120] The speech emotion is represented by S a3 and the self-attention score matrix A a Multiplying them together yields the self-attention result S′, which contains the internal correlation information of the speech modality. a , the specific calculation formula is:

[0121] S′ a =S a3 A a ;

[0122] S33, given three image modality weight matrices W of size d*d v1 ,W v2 ,W v3 , respectively, with the image emotion representation S v Multiply to get three initialized image emotion representations S v1 ,S v2 ,S v3 , the specific calculation formula is:

[0123] S vi' =S v W vi' ;

[0124] Among them, S vi' The size is 1*d.

[0125] The emotion representation S of the image v1 ,S v2 Do a dot multiplication and pass it through the Relu(·) activation function to obtain the self-attention score matrix A of size d*d v , the specific calculation formula is:

[0126]

[0127] The image emotion is represented by S v3 and the self-attention score matrix A v Multiply to get the self-attention result S′ which contains the internal correlation information of the image modality v , the specific calculation formula is:

[0128] S′ v =S v3 A v ;

[0129] S34. Given three multimodal weight matrices W of size 3d*3d m1 ,W m2 ,W m3 , respectively, and S′ t ,S′ a ,S′ v The concatenated multimodal representations are multiplied to obtain three initialized multimodal emotion representations Sm1 ,S m2 ,S m3 , the specific calculation formula is:

[0130] S mi' =(S′ t ; S′ a ; S′ v )W mi' ;

[0131] For multimodal sentiment representation S m1 ,S m2 Do a dot multiplication and pass it through the Relu(·) activation function to obtain a self-attention score matrix A of size 3d*3d m , the specific calculation formula is:

[0132]

[0133] The multimodal sentiment representation S m3 and the self-attention score matrix A m Multiplying them together yields the self-attention result S′, which contains the correlation information between the modalities. m , the specific calculation formula is:

[0134] S′ m =S m3 ·A m .

[0135] Step 4: Use the modal emotion representation loss, modal noise representation loss, and classification loss as joint loss constraints. Under the joint loss constraints, pass the concatenated modal emotion representation into the emotion prediction classification layer for classification prediction, and output the optimal emotion prediction value.

[0136] The classification method of the sentiment prediction classification layer is:

[0137] S′ m Input into a three-layer neural network, the input layer and hidden layer size of the three-layer neural network are both 3d, the output layer size is 2, and the sigmoid (·) activation function is used to implement emotion 2 classification; the specific calculation formula is:

[0138]

[0139] in, represents the predicted value, F(·) represents the function expression of the neural network, θ m Represents the parameter set between each neuron.

[0140] Calculate the cross entropy classification loss L between the predicted value of the neural network and the true value class , the specific calculation formula is:

[0141]

[0142] Among them, L class is the cross entropy classification loss between the predicted value and the true value of the neural network, y represents the true value of the sample, represents the predicted value;

[0143] Joint cross entropy classification loss L class , emotional expression loss L s , noise representation loss L p Together as the total task loss L task To constrain the training process, the specific calculation formula is:

[0144] L task =αL class +βL s +γL p ;

[0145] Among them, α, β, and γ all represent the weight of the loss.

[0146] The implementation steps of the present invention are as follows: extracting the feature vectors of the three modalities from the original data; converting the feature vectors of the three modalities into three modal emotion representations and three modal noise representations through modal representation space segmentation; using the superimposed self-attention mechanism to mine the correlation information within and between the three modal emotion representations; passing the multimodal representation with correlation information into the emotion classification layer to obtain the emotion prediction result, and combining the classification loss, emotion representation loss and noise representation loss to optimize the training process, and finally obtain a reliable emotion prediction value.

[0147] The effectiveness of our method was evaluated through quantitative analysis on the MOSI dataset. Table 1 shows a comparison of the accuracy and F1 score of our method with several representative methods in the field on this dataset. Acc refers to the accuracy, which represents the ratio of the number of samples with correct sentiment classification to the total number of samples in the prediction results. The F1 score is the harmonic mean of precision and recall; higher values ​​of these two metrics indicate better performance. Our method achieved superior results in both binary classification accuracy and F1 score compared to BC-LSTM, LMF, MARN, and ARGF methods.

[0148] Table 1 Comparison results of the method of the present invention and various representative methods on the MOSI dataset

[0149]

[0150] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal sentiment analysis method based on modal representation space and superimposed self-attention, characterized by: The steps are as follows: Step 1: Extract feature vectors of the three modalities of text, speech, and image from the original video data; Step 2: Use the modal representation space segmentation method to map the feature vectors of the three modalities of text, speech, and image to the emotion representation space and noise representation space respectively to obtain the modal emotion representation and modal noise representation; Step 3: The obtained modal sentiment representation is passed into the superimposed self-attention mechanism to mine the correlation information between the modal sentiment representations and splice the modal sentiment representations containing the correlation information; Step 4: Use the modal emotion representation loss, modal noise representation loss, and classification loss as joint loss constraints. Under the joint loss constraints, pass the concatenated modal emotion representation to the emotion prediction classification layer for classification prediction, and output the optimal emotion prediction value. In step 2, the specific implementation method is: S21, use the sentiment encoder to respectively transform the text feature vector U t , speech feature vector U a and image feature vector U v Enter the sentiment representation space and obtain the text sentiment representation S with a size of 1*d t , Speech Emotion Representation S a and image sentiment representation S v , the specific calculation formula is: S m =E s (U m ;θ s ); Where m∈{t,a,v}; E s represents the emotion encoder, θ s are the parameters of the emotion encoder; S22, using the text noise encoder to transform the text feature vector U t Enter the text noise space and get the text noise representation P of size 1*d t , the specific calculation formula is: in, represents a text noise encoder, are the parameters of the text noise encoder; S23, with the help of speech noise encoder, the speech feature vector U a Enter the speech noise space and get the speech noise representation P of size 1*d a , the specific calculation formula is: in, represents the speech noise encoder, are the parameters of the speech noise encoder; S24, using the image noise encoder to transform the image feature vector U v Enter the image noise space and get the image noise representation P of size 1*d v , the specific calculation formula is: in, represents the image noise encoder, It is a parameter unique to the image noise encoder.

2. The multimodal sentiment analysis method based on modal representation space and superimposed self-attention according to claim 1 is characterized in that In step three, the specific implementation method is: S31. Given three text modality weight matrices W of size d*d t1 ,W t2 ,W t3 , respectively, with the text sentiment representation S t Multiply to get three initialized text sentiment representations S t1 ,S t2 ,S t3 , the specific calculation formula is: S ti' =S t W ti' ; Where, i'∈{1,2,3}; S for text sentiment representation t1 ,S t2 Do the dot multiplication and pass the Relu(·) activation function to obtain the self-attention score matrix A of size d*d t , the specific calculation formula is: in, For S t2 The transpose of The text sentiment is represented by S t3 and the self-attention score matrix A t Multiply them together to get the self-attention result S containing the internal correlation information of the text modality t ′, the specific calculation formula is: S t ′=S t3 A t ; S32, given three speech modality weight matrices W of size d*d a1 ,W a2 ,W a3 , respectively, with the speech emotion representation S a Multiply to get three initialized speech emotion representations S a1 ,S a2 ,S a3 , the specific calculation formula is: S ai' =S a W ai' ; Speech emotion representation S a1 ,S a2 Do a dot multiplication and pass it through the Relu(·) activation function to obtain the self-attention score matrix A of size d*d a , the specific calculation formula is: The speech emotion is represented by S a3 and the self-attention score matrix A a Multiplying them together yields the self-attention result S containing the internal correlation information of the speech modality. a ′, the specific calculation formula is: S a ′=S a3 A a ; S33, given three image modality weight matrices W of size d*d v1 ,W v2 ,W v3 , respectively, with the image emotion representation S v Multiply to get three initialized image emotion representations S v1 ,S v2 ,S v3 , the specific calculation formula is: S vi' =S v W vi' ; The emotion representation S of the image v1 ,S v2 Do a dot multiplication and pass it through the Relu(·) activation function to obtain the self-attention score matrix A of size d*d v , the specific calculation formula is: The image emotion is represented by S v3 and the self-attention score matrix A v Multiply to get the self-attention result S containing the internal correlation information of the image modality v ′, the specific calculation formula is: S v ′=S v3 A v ; S34. Given three multimodal weight matrices W of size 3d*3d m1 ,W m2 ,W m3 , respectively with S t ′,S a ′,S v The concatenated multimodal representations are multiplied to obtain three initialized multimodal emotion representations S m1 ,S m2 ,S m3 , the specific calculation formula is: S mi' =(S t ′;S a ′;S v ′)W mi' ; For multimodal sentiment representation S m1 ,S m2 Do a dot multiplication and pass it through the Relu(·) activation function to obtain a self-attention score matrix A of size 3d*3d m , the specific calculation formula is: The multimodal sentiment representation S m3 and the self-attention score matrix A m Multiplying them together yields the self-attention result S′, which contains the correlation information between the modalities. m , the specific calculation formula is: S′ m =S m3 ·A m 。 3. The multimodal sentiment analysis method based on modal representation space and superimposed self-attention according to claim 1 is characterized in that The modality sentiment representation loss is: Among them, L s is the sentiment representation loss, (m,n)∈{(t,a),(t,v),(a,v)}, d represents the number of elements in the feature vector, represents the square of the 2-norm; The modal noise represents the loss: Among them, L p′ represents the intermediate variable, L p is the noise representation loss, i∈{l,a,v},(x,z)∈{(t,a),(t,v),(a,t),(a,v),(v,t),(v,a)}; The classification loss is: Among them, L class is the cross entropy classification loss between the predicted value and the true value of the neural network, y represents the true value of the sample, represents the predicted value; Joint cross entropy classification loss L class , emotional expression loss L s , noise representation loss L p Together as the total task loss L task To constrain the training process, the specific calculation formula is: L task =αL class +βL s +γL p ; Among them, α, β, and γ all represent the weight of the loss.

4. The multimodal sentiment analysis method based on modal representation space and superimposed self-attention according to claim 2 is characterized in that The classification method of the sentiment prediction classification layer is: The self-attention result S′ m Input into a three-layer neural network, the input layer and hidden layer size of the three-layer neural network are both 3d, the output layer size is 2, and the sigmoid (·) activation function is used to implement emotion 2 classification; the specific calculation formula is: in, represents the predicted value, F(·) represents the function expression of the neural network, θ m Represents the parameter set between each neuron.

Images