Multi-modal sentiment analysis method based on cross-modal attention

Abstract

The invention relates to the technical field of intelligent emotion recognition, in particular to a multi-modal emotion analysis method based on cross-modal attention, and the method comprises the steps: obtaining a to-be-analyzed video segment; the to-be-analyzed video segment is input into a preset multi-mode sentiment analysis model, a sentiment analysis result is output, and the multi-mode sentiment analysis model comprises an input layer used for extracting features of the to-be-analyzed video segment; the single-mode feature extraction layer is used for performing single-mode context information extraction on the features to obtain single-mode context features; the cross-modal interaction layer is used for performing cross-modal feature interaction on the single-modal context features to obtain a plurality of bimodal joint features; the multi-modal fusion layer is used for carrying out fusion processing on the bimodal joint features to obtain cross-modal interaction features; and the output layer is used for splicing the single-modal context features and the cross-modal interaction features, obtaining multi-modal fusion features and outputting an emotion analysis result. According to the invention, the emotion tendency of the speaker can be reflected more accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent emotion recognition, and in particular to a multimodal sentiment analysis method based on cross-modal attention. Background Art

[0002] With the rapid development of industry and the Internet, people have gradually tended to use a combination of text, pictures and videos to express their opinions and insights. These users share and display their daily lives on various social media such as Twitter, YouTube, and Sina Weibo, and express their views and discussions on various social events. These social information contains rich emotional information. As an important data source for sentiment analysis, if these social information can be collected and analyzed, then the research on sentiment analysis can be widely applied to various fields, such as product recommendations, restaurant reviews, public opinion monitoring and control, and mental health guidance.

[0003] Previous sentiment analysis research has mainly focused on text modality. Given the existing sources of sentiment information, the sentiment expression presented by a single modality has defects and certain limitations. Ignoring the different inherent sampling rates of temporal information between different modalities and cross-modal contextual dependencies, multimodal sentiment analysis should not only consider the single modal feature information extraction technology of text, vision and speech, but also the effective fusion strategy between different modalities. Due to the different sampling methods of different modal features, there are large differences in time series and semantic information, and the visual and speech feature information contains a large amount of low-order signals and redundant information, making it impossible to effectively fuse the feature information between modalities.

[0004] Therefore, a multimodal sentiment analysis method based on cross-modal attention is urgently needed. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal sentiment analysis method based on cross-modal attention to address the defects of current multimodal sentiment analysis methods. With the supplement of visual and voice modalities, sentiment analysis can more accurately reflect the emotional tendency of the speaker at that time.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A multimodal sentiment analysis method based on cross-modal attention, including:

[0008] Get the video segment to be analyzed;

[0009] The video segment to be analyzed is input into a preset multimodal sentiment analysis model, and the sentiment analysis result is output, wherein the multimodal sentiment analysis model includes an input layer, a unimodal feature extraction layer, a cross-modal interaction layer, a multimodal fusion layer, and an output layer, wherein the input layer is used to extract the features of the video segment to be analyzed; the unimodal feature extraction layer is used to extract unimodal context information on the features to obtain unimodal context features; the cross-modal interaction layer is used to perform cross-modal feature interaction on the unimodal context features to obtain a number of bimodal joint features; the multimodal fusion layer is used to fuse the bimodal joint features to obtain cross-modal interaction features; the output layer is used to splice the unimodal context features and the cross-modal interaction features to obtain multimodal fusion features, and output the sentiment analysis result based on the multimodal fusion features.

[0010] Optionally, the input layer extracting the features of the video segment to be analyzed includes:

[0011] Obtaining a text sequence, a voice sequence, and a visual sequence in the video segment to be analyzed;

[0012] A text feature vector, a speech feature vector, and a visual feature vector are extracted based on the text sequence, the speech sequence, and the visual sequence, respectively.

[0013] Optionally, the unimodal feature extraction layer extracts unimodal context information from the feature, and obtaining the unimodal context feature includes:

[0014] Mapping the text feature vector, the speech feature vector, and the visual feature vector to the same dimension through a fixed-dimensional one-dimensional convolution to obtain local temporal features;

[0015] Input the local time series features into BiGRU, obtain the bidirectional hidden state and context features at each moment, then use the multi-head attention mechanism to enhance the context features, calculate the query index similarity of the context features, and obtain the query matrix;

[0016] The outputs of the multi-head attention mechanism are spliced ​​to obtain the output result, and the output result and the query matrix are processed by layer normalization, and then integrated through a fully connected layer to obtain a unimodal feature vector with contextual temporal information.

[0017] Optionally, the cross-modal interaction layer performs cross-modal feature interaction on the unimodal context feature to obtain a plurality of bimodal joint features, including:

[0018] Interacting the unimodal context features in pairs;

[0019] The two interacting unimodal context features are reconstructed using low-order signals, and then passed through several cross-modal converters to obtain two unidirectional cross-modal features;

[0020] The two unidirectional cross-modal features are concatenated through an activation function to obtain concatenated features;

[0021] The attention distribution of the splicing feature is calculated, and the attention distribution is multiplied by the splicing feature to obtain the bimodal joint feature.

[0022] Optionally, the working method of the cross-modal converter is:

[0023]

[0024] Among them, Y α Represents the cross-modal attention implementation operator, the enhanced query (Query) From target modality α, key and value From the auxiliary modal β, are the corresponding weight matrices, They represent the sequence features of the target modality α and the auxiliary modality β respectively, L and d represent the sequence length and feature dimension respectively.

[0025] Optionally, the multimodal fusion layer performs fusion processing on the bimodal joint features to obtain cross-modal interaction features, including:

[0026] The pairwise interaction features are concatenated with the corresponding bimodal joint features to obtain the joint features;

[0027] Based on the joint features, a self-attention mechanism is used to obtain the modal internal correlation features of the pairwise interactive features;

[0028] After filtering the relevant features within the modality, the modal attention features of the pairwise interactive features are obtained, that is, the cross-modal interaction features.

[0029] Optionally, the output layer concatenates the unimodal context features and the cross-modal interaction features to obtain multimodal fusion features, and the method for outputting the sentiment analysis result based on the multimodal fusion features is:

[0030]

[0031] F1=ReLU(W R Z2+b R )

[0032] ICA-MSA=Softmax(F1)

[0033] in, Represents the concatenation operation of the vector, Z2 is the feature vector obtained by concatenating the unimodal context feature and the modal attention feature, Self VT is the text-visual modality attention feature, Self AV is the visual-speech modality attention feature, Self TA is the speech-text modality attention feature, F1 is the multimodal fusion feature obtained by integrating the concatenated feature vectors through the fully connected layer, and W R and b R are the initialization weights and bias terms of the activation function ReLU, and ICA-MSA is the final classification result.

[0034] The beneficial effects of the present invention are:

[0035] The present invention aims at the problems of insufficient modal fusion and weak interactivity caused by the differences in semantic features between different modalities in multimodal sentiment analysis. By studying and analyzing the potential correlation between different modalities, a multimodal sentiment analysis model is built. The model first uses a bidirectional gated recurrent unit and a multi-head attention mechanism to extract text, visual and speech modal temporal features with contextual semantic information; then a cross-modal interaction layer is designed, and the target modality is continuously strengthened by the low-order signal of the auxiliary modality, so that the target modality learns the information of the auxiliary modality and captures the potential adaptability between modalities; then the enhanced features are input into the multimodal fusion layer, and the similarities between different modalities are further captured by the conditional vector, the correlation degree of important features is enhanced, and the deeper interactivity between modalities is excavated; finally, the composite cross-modal enhanced features are spliced ​​and fused with low-order signals by using a multi-head attention mechanism, the weight of important features inside the modality is increased, the unique feature information of the initial modality is retained, and the obtained multimodal fusion features are used for the final sentiment classification task. The model can more effectively mine the correlation between different modalities and make more accurate sentiment judgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A flow chart of a multimodal sentiment analysis method based on cross-modal attention according to an embodiment of the present invention;

[0038] Figure 2A schematic diagram of the working process and network structure of a cross-modal converter according to an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the cross-modal interaction layer workflow and network structure of an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of the multimodal sentiment analysis model workflow and network structure of an embodiment of the present invention;

[0041] Figure 5 The figures show the effects of different numbers of cross-modal attention layers on the model according to the embodiments of the present invention, where (a) is the Acc-2 indicator experimental result and (b) is the F1-Score indicator experimental result. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] This embodiment provides a multimodal sentiment analysis method based on cross-modal attention, including:

[0045] Get the video segment to be analyzed;

[0046] The video segment to be analyzed is input into a preset multimodal sentiment analysis model, and the sentiment analysis result is output, wherein the multimodal sentiment analysis model includes an input layer, a unimodal feature extraction layer, a cross-modal interaction layer, a multimodal fusion layer, and an output layer. The input layer is used to extract the features of the video segment to be analyzed; the unimodal feature extraction layer is used to extract unimodal context information from the features to obtain unimodal context features; the cross-modal interaction layer is used to perform cross-modal feature interaction on the unimodal context features to obtain several bimodal joint features; the multimodal fusion layer is used to fuse the bimodal joint features to obtain cross-modal interaction features; the output layer is used to splice the unimodal context features and the cross-modal interaction features to obtain multimodal fusion features, and output the sentiment analysis result based on the multimodal fusion features.

[0047] Specifically, Figure 4 As shown, the multimodal sentiment analysis model of this embodiment specifically includes:

[0048] (1) Input layer: Extracts primary modal features, including text features, data features, and audio features.

[0049] (2) Unimodal feature extraction layer: A bidirectional gated recurrent neural network is used to extract unimodal contextual feature vectors, mine unique features within the modality, and extract contextual information.

[0050] (3) Cross-modal interaction layer: A multimodal fusion method based on an improved cross-modal attention mechanism is designed. The low-order signals of the auxiliary modality are used to continuously strengthen the target modality, so that the target modality can learn the information of the auxiliary modality and capture the potential adaptability between modalities to achieve information enhancement between different modalities.

[0051] (4) Multimodal fusion layer: A multimodal feature fusion module is constructed through the self-attention mechanism and the fully connected layer to fuse the enhanced modalities and obtain cross-modal interaction features.

[0052] (5) Output layer: The contextual features and cross-modal joint features are concatenated and sent to Softmax after passing through the fully connected layer for the final sentiment analysis task.

[0053] This embodiment aims at the problems of insufficient modal fusion and weak interactivity caused by the differences in semantic features between different modalities in multimodal sentiment analysis. By studying and analyzing the potential correlation between different modalities, a multimodal sentiment analysis model is built. The model first uses a bidirectional gated recurrent unit and a multi-head attention mechanism to extract text, visual and speech modal temporal features with contextual semantic information; then a cross-modal interaction layer is designed, and the target modality is continuously strengthened by the low-order signal of the auxiliary modality, so that the target modality learns the information of the auxiliary modality and captures the potential adaptability between modalities; then the enhanced features are input into the multimodal fusion layer, and the similarities between different modalities are further captured through the conditional vector, the correlation degree of important features is enhanced, and the deeper interactivity between modalities is excavated; finally, the composite cross-modal enhanced features are spliced ​​and fused with low-order signals by using the multi-head attention mechanism, the weight of important features within the modality is increased, the unique feature information of the initial modality is retained, and the obtained multimodal fusion features are used for the final sentiment classification task. The model can more effectively explore the correlation between different modalities and make more accurate sentiment judgments.

[0054] Furthermore, the input layer extracts the features of the video segment to be analyzed including:

[0055] Obtaining text sequences, voice sequences, and visual sequences in the video segment to be analyzed;

[0056] The text feature vector, speech feature vector and visual feature vector are extracted based on the text sequence, speech sequence and visual sequence respectively.

[0057] Specifically, the input layer extracts modal primary features, including text data, using 300-dimensional Glove word embedding with global vectors as text features; for each emotional polarity audio extraction, the COVAREP speech analysis framework is used to extract 74-dimensional speech high-order statistical features in the speech signal; the Openface facial expression analysis framework is used to obtain visual features, detect the facial action units of the speaker in each frame, and extract 35 muscle movement units. The dimensions of text, audio, and visual features of each video sequence are d respectively. T =300,d A =74,d V =35.

[0058] Furthermore, the unimodal feature extraction layer extracts unimodal context information from the features, and obtaining unimodal context features includes:

[0059] The text feature vector, speech feature vector, and visual feature vector are mapped to the same dimension through a fixed-dimensional one-dimensional convolution to obtain local temporal features;

[0060] Input the local time series features into BiGRU to obtain the bidirectional hidden state and context features at each moment, then use the multi-head attention mechanism to enhance the context features, calculate the query index similarity of the context features, and obtain the query matrix;

[0061] The outputs of the multi-head attention mechanism are spliced ​​to obtain the output results, and the output results and query matrix are processed using layer normalization, and then integrated through the fully connected layer to obtain a unimodal feature vector with contextual temporal information.

[0062] Specifically, the feature vector is input into the unimodal feature extraction layer, and the temporal information of the text, speech, and visual feature sequences is obtained through the bidirectional gated recurrent unit and self-attention method. Respectively represent the final output text, visual and audio feature vectors with contextual temporal information.

[0063] Furthermore, the cross-modal interaction layer performs cross-modal feature interaction on the unimodal context features to obtain several bimodal joint features including:

[0064] Interact unimodal context features pairwise;

[0065] The two interacting unimodal context features are reconstructed using low-order signals, and then passed through several cross-modal converters to obtain two unidirectional cross-modal features;

[0066] The two unidirectional cross-modal features are concatenated through an activation function to obtain concatenated features;

[0067] Calculate the attention distribution of the concatenated features and multiply the attention distribution with the concatenated features to obtain the bimodal joint features.

[0068] Specifically, the three modes Z T ,Z V ,Z A Input to the cross-modal interaction layer, by combining different modalities in pairs, cross-modal feature fusion is performed, with text (Z T ) and vision (Z V ) as an example, the low-order signal of the auxiliary modality is used to continuously strengthen the target modality to enhance the semantic relevance between the two modalities, and the bimodal features after modal information enhancement are obtained. Then, the feature fusion method of the multimodal fusion layer is used to obtain the bimodal joint feature ICA rich in semantic relevance. VT .

[0069] Furthermore, the working method of the cross-modal converter is:

[0070]

[0071] Among them, Y α Represents the cross-modal attention implementation operator, the enhanced query (Query) From target modality α, key and value From the auxiliary modal β, are the corresponding weight matrices, They represent the sequence features of the target modality α and the auxiliary modality β respectively, L and d represent the sequence length and feature dimension respectively.

[0072] Furthermore, the multimodal fusion layer fuses the bimodal joint features to obtain cross-modal interaction features including:

[0073] The pairwise interaction features are concatenated with the corresponding bimodal joint features to obtain the joint features;

[0074] Based on the joint features, the self-attention mechanism is used to obtain the modal internal correlation features of the pairwise interactive features;

[0075] After filtering the relevant features within the modality, the modal attention features of the pairwise interactive features are obtained, namely the cross-modal interaction features.

[0076] Specifically, the text-visual features Visual-audio features Speech-text features Input to the multimodal fusion layer to obtain the text-visual modality attention feature Self VT, visual-speech modality attention feature Self AV , speech-text modality attention feature Self TA .

[0077] Furthermore, the output layer concatenates the unimodal context features and the cross-modal interaction features to obtain multimodal fusion features, and the method for outputting the sentiment analysis results based on the multimodal fusion features is as follows:

[0078]

[0079] F1=ReLU(W R Z2+b R )

[0080] ICA-MSA=Softmax(F1)

[0081] Among them, Z2 is the feature vector obtained by concatenating the unimodal context feature and the modal attention feature, Self VT is the text-visual modality attention feature, Self AV is the visual-speech modality attention feature, Self TA is the speech-text modality attention feature, F1 is the multimodal fusion feature obtained by integrating the concatenated feature vectors through the fully connected layer, and W R and b R are the initialization weights and bias terms of the activation function ReLU, and ICA-MSA is the final classification result.

[0082] Combine the following Figure 1-Figure 3 A multimodal sentiment analysis method based on cross-modal attention proposed in the present invention is described in detail, which specifically includes the following steps:

[0083] Step 1: Get the text, speech, and visual sequences of a video segment. Each sample data {x1,x2,···=,x L} is divided into a sequence of length L, each sample sequence is divided into three sequence modes: text (T), vision (V), and audio (A), and the feature vector X = [X T ,X V ,X A ]express.

[0084] Step 2: The dimensions of text, audio, and visual features of each video sequence obtained in step 1 are d T = = 300, d A = =74,d V = = 35. The feature vector is input into the unimodal feature extraction layer, and the temporal information of the text, speech, and visual feature sequences are obtained through the bidirectional gated recurrent unit and self-attention method. They represent the final output text, visual and audio feature vectors with contextual temporal information respectively.

[0085] Step 3: The three modes Z obtained in step 2 T , Z V , Z A Input to the cross-modal interaction layer, by combining different modalities in pairs, cross-modal feature fusion is performed, with text (Z T ) and vision (Z V ) as an example, the low-order signal of the auxiliary modality is used to continuously strengthen the target modality to enhance the semantic relevance between the two modalities, and the bimodal features after modal information enhancement are obtained. Then, the feature fusion method in the composite fusion module is used to obtain the bimodal joint feature ICA rich in semantic relevance. VT .

[0086] Step 4: Repeat step 3 to obtain the information-enhanced text-visual features Visual-audio features Speech-text features Then the output features of the improved cross-modal interaction are input into the multimodal fusion layer to obtain the relevant features within the modality Self VT Finally, the text, speech, and visual modalities are cross-modally interacted in pairs to obtain the text-visual modality attention feature Self VT , visual-speech modality attention feature Self AV , speech-text modality attention feature Self TA .

[0087] Step 5: Concatenate the obtained attention feature vectors, and use the fully connected layer to integrate the obtained inter-modal interaction features and modal internal features, and input them into the Softmax function to perform sentiment classification on the final multimodal fusion features.

[0088] Furthermore, step 1 specifically includes using the Multimodal SDK toolkit to obtain unimodal features in the video sequence:

[0089] Step 1.1: For text data, use 300-dimensional GloVe word embedding as text features;

[0090] Step 1.2: For each emotional polarity audio, the COVAREP speech analysis framework extracts 74-dimensional high-order statistical features of speech in the speech signal;

[0091] Step 1.3: Use the FACET facial expression analysis framework to obtain visual features, detect the facial action units of the speaker in each frame, and extract 35 muscle movement units;

[0092] The dimensions of text, audio, and visual features obtained for each video sequence are d T = = 300,d A = =74,d V =35.

[0093] Furthermore, step 2 specifically includes:

[0094] Step 2.1: Use a fixed set of feature dimensions d k The same convolution kernel (k∈{T,V,A}) extracts local temporal features and maps features of different modalities to the same dimension d:

[0095]

[0096] Where L represents the sequence length.

[0097] Step 2.2: After the one-dimensional convolutional layer processing, the obtained features are input into BiGRU to obtain the bidirectional hidden state of the features at each moment, H = BiGRU (X {T，V，A} ), k∈{T,V,A}, captures the long-term dependencies of the context, and then uses the multi-head attention mechanism to strengthen the weight of the context features within the modality and calculate the similarity of query indexes in different subspaces:

[0098]

[0099] Where M is the number of attention heads, d h is the hidden state dimension at a certain moment, W Q , W K , W V They are the corresponding Query (Q), Keys (K), and Values ​​(V) mapping matrices respectively.

[0100] Step 2.3: Concatenate all attention heads to obtain the complete output result, and use layer normalization (Layer=Normalization,=LN) to process the accumulated output result and query matrix elements, M LN＝ LN(H+M a (H)), and then the unimodal feature vector Z is obtained by integration through the fully connected layer:

[0101]

[0102] Where Z = [Z T ,Z V ,Z A ], They represent the final output text, visual and audio feature vectors with contextual temporal information respectively.

[0103] Furthermore, step 3 specifically includes:

[0104] Step 3.1: This embodiment improves the cross-modal attention network as follows Figure 2 As shown in Figure 2, due to the different sampling rates of sequences of different modalities, most of the time series are unaligned data. The cross-modal transformer (Cross = Modal = Transformer, = CM) considers the interaction between different time series and solves the problem of data misalignment in the multimodal fusion process. The target modality α and auxiliary modality β, each modality (unaligned) sequence feature is represented as and Where L and d represent the sequence length and feature dimension. The potential adaptation from β mode to α mode is expressed as follows:

[0105]

[0106] Among them, Y α Represents the cross-modal attention implementation operator, the enhanced query (Query) From target modality α, key and value From the modal β, They are the corresponding weight matrices respectively. The source modality β reconstructs the feature information of the target modality α with low-order signals, interacts through key / value pairs, and captures important related information across modalities.

[0107] Step 3.2: This embodiment improves the cross-modal attention network as follows Figure 3 As shown, in text mode (Z T ) and visual modality (Z V ) as an example, the unimodal feature sequence with contextual temporal information is input into the cross-modal attention interaction layer, and Z V→T ,Z T→V Respectively represent the reconstruction of text features by visual low-order signals and the reconstruction of visual features by text low-order signals. To avoid gradient explosion caused by too large values, add layer normalization LN and residual connection, and then use two CM modules to bidirectionally calculate the similarity between the auxiliary modality and the target modality, and continuously strengthen the target modality. Feedforward calculation is performed according to i=1,2,…,n layers, with Z V→T For example, the calculation process is as follows:

[0108]

[0109] Among them, P1 is used as an intermediate variable to obtain the feedforward value of cross-modal attention, f is a position feedforward sublayer with θ as a parameter, which gives sufficient attention to the i-1 time step of the visual modality through the i-th time step of the text modality, and the i-th time step of P1 is V β The weighted sum of is determined by the i-th row in Softmax.

[0110] Step 3.3 specifically includes: using low-order visual modal information to continuously strengthen the text modality, so that the text modality can learn information from the visual modality, and through n layers of cross-modal attention superposition, obtain unidirectional cross-modal text features At the same time, we also obtain the visual features after the text mode strengthens the visual mode. The enhanced two modal features are concatenated through the activation function tanh to capture the hidden correlation between the modalities and obtain the feature vector Use the =Softmax function to calculate the attention distribution matrix att1, and then multiply the obtained attention distribution matrix with the matrix of multimodal feature fusion to obtain the final weighted text and visual cross-modal interaction feature ICA VT , the specific calculation is as follows.

[0111]

[0112] in, Represents the concatenation operation of vectors, W VT and b VT They represent the mapping matrix and bias term after feature concatenation, respectively, and · represents matrix multiplication.

[0113] Furthermore, step 4 specifically includes: inputting the output features of the improved cross-modal interaction into the self-attention layer to obtain the modality internal related features Self VT , the specific calculation is as follows:

[0114]

[0115]

[0116] Self VT =att2·Z1

[0117] Among them, Z1 represents the feature vector obtained by concatenating text features, speech features and bimodal interaction features, Self VT Represents the attention feature vector after information filtering.

[0118] Furthermore, step 5 specifically includes: performing cross-modal interactions between text, speech, and visual modalities in pairs to obtain the text-visual modality attention feature Self VT, visual-speech modality attention feature Self AV , speech-text modality attention feature Self TA , concatenate the obtained attention feature vectors, and use the fully connected layer to integrate the obtained inter-modal interaction features and modal internal features, and input them into Sofmax for sentiment classification. The calculation process is as follows:

[0119]

[0120] F1=ReLU(W R Z2+b R )

[0121] ICA-MSA=Softmax(F1)

[0122] Among them, Z2 is the feature vector obtained by concatenating the attention feature vector and the context feature vector, F1 is the multimodal fusion feature obtained by integrating the concatenated feature vectors through the fully connected layer, and W R and b R are the initialization weights and bias terms of the activation function ReLU, and ICA-MSA is the final classification result.

[0123] Experimental verification:

[0124] (1) Comparative experiment:

[0125] Table 1

[0126]

[0127] The experimental results are shown in Table 1. In the task of sentiment binary classification, the ICA-MSA model proposed in this embodiment shows better performance in all indicators compared with other models. Under two different data sets, the ICA-MSA model has significantly improved all indicators compared with the simple feature-level fusion EF-LSTM and decision-level fusion LF-LSTM models, among which the binary classification accuracy has increased by about 5%. Compared with the models based on non-attention methods such as TFN, LMF, and MFM for multimodal fusion, the binary classification accuracy has increased by about 3.5%, which fully demonstrates that the multimodal fusion method based on the attention fusion strategy can better capture the interactivity between different modalities. Compared with the MTCN and MulT models, the improved cross-modal attention method used in this embodiment also shows certain advantages, which further illustrates that through the improved cross-modal fusion strategy, considering the different effects of the temporal nature of the modality itself and the low- and high-order modal information on the sentiment classification results, it pays more attention to the deeper correlation within the modality, so that the model can learn more complete emotional expression and make more accurate emotional judgments.

[0128] (2) Model ablation experiment comparison:

[0129] Table 2

[0130]

[0131] The experimental results are shown in Table 2. The sentiment analysis results of the fusion of three modal features, text, vision and audio, are the best, which fully proves the importance of multimodal fusion. It is shown from different data sets that larger-scale training data can improve the performance of sentiment analysis to a certain extent. In the experiments of single modality and dual modality under the same data set, the results of text single modality and dual modality with text modality are significantly better than the other two, indicating that in general, the sentiment polarity of text modality features is the most significant. Compared with single modality, the results of the two indicators of the model under dual modality are obviously better, but the accuracy of V+A modality is slightly lower than that of visual modality, which is due to the relatively weak sentiment polarity of visual and audio modalities and the interference of redundant information. Therefore, the effective fusion of text, vision and audio trimodal features helps to improve the performance of sentiment classification and make up for the problem of incomplete sentiment expression of a single modality.

[0132] (3) Hyperparameter influence:

[0133] The number of cross-modal attention stacking layers n in the cross-modal interaction layer is also a major hyperparameter that affects the final performance of the model. It specifies the number of cross-modal Transformers, and performs feed-forward calculations from i=1 to n layers. The auxiliary modality continuously updates its sequence information. In order to study the impact of the number of cross-modal attention layers n on the overall performance, n is set to 1 to 9. The experimental indicators include Acc-2 and F1-Score. Figure 5 (a) shows the experimental results of Acc-2 index, Figure 5 (b) shows the experimental results of the F1-Score indicator. According to the analysis of the experimental results, the two indicators have similar change trends. The accuracy and F1 value both increase first and then decrease. When the number of cross-modal module stacking layers is 4, the model performance is the best, indicating that the number of layers and performance are not positively correlated. When the number of layers is higher than 4, the model effect begins to decline significantly, and model training becomes more difficult and uncontrollable.

[0134] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multimodal sentiment analysis method based on cross-modal attention, characterized in that: include: Get the video segment to be analyzed; The video segment to be analyzed is input into a preset multimodal sentiment analysis model, and the sentiment analysis result is output, wherein the multimodal sentiment analysis model includes an input layer, a unimodal feature extraction layer, a cross-modal interaction layer, a multimodal fusion layer, and an output layer, wherein the input layer is used to extract the features of the video segment to be analyzed; the unimodal feature extraction layer is used to extract unimodal context information from the features to obtain unimodal context features; the cross-modal interaction layer is used to perform cross-modal feature interaction on the unimodal context features to obtain a number of bimodal joint features; The multimodal fusion layer is used to fuse the bimodal joint features to obtain cross-modal interaction features; the output layer is used to splice the unimodal context features and cross-modal interaction features to obtain multimodal fusion features, and output the sentiment analysis results based on the multimodal fusion features.

2. The multimodal sentiment analysis method based on cross-modal attention according to claim 1, characterized in that: The input layer extracts the features of the video segment to be analyzed including: Obtaining a text sequence, a voice sequence, and a visual sequence in the video segment to be analyzed; A text feature vector, a speech feature vector, and a visual feature vector are extracted based on the text sequence, the speech sequence, and the visual sequence, respectively.

3. The multimodal sentiment analysis method based on cross-modal attention according to claim 2 is characterized in that: The unimodal feature extraction layer extracts unimodal context information from the feature, and obtaining the unimodal context feature includes: Mapping the text feature vector, the speech feature vector, and the visual feature vector to the same dimension through a fixed-dimensional one-dimensional convolution to obtain local temporal features; Input the local time series features into BiGRU, obtain the bidirectional hidden state and context features at each moment, then use the multi-head attention mechanism to enhance the context features, calculate the query index similarity of the context features, and obtain the query matrix; The outputs of the multi-head attention mechanism are spliced ​​to obtain the output result, and the output result and the query matrix are processed by layer normalization, and then integrated through a fully connected layer to obtain a unimodal feature vector with contextual temporal information.

4. The multimodal sentiment analysis method based on cross-modal attention according to claim 1, characterized in that: The cross-modal interaction layer performs cross-modal feature interaction on the unimodal context features to obtain a plurality of bimodal joint features, including: Interacting the unimodal context features in pairs; The two interacting unimodal context features are reconstructed using low-order signals, and then passed through several cross-modal converters to obtain two unidirectional cross-modal features; The two unidirectional cross-modal features are concatenated through an activation function to obtain concatenated features; The attention distribution of the splicing feature is calculated, and the attention distribution is multiplied by the splicing feature to obtain the bimodal joint feature.

5. The multimodal sentiment analysis method based on cross-modal attention according to claim 4 is characterized in that: The working method of the cross-modal converter is: Among them, Y α Represents the cross-modal attention implementation operator, the enhanced query (Query) From target modality α, key and value From the auxiliary modal β, are the corresponding weight matrices, They represent the sequence features of the target modality α and the auxiliary modality β respectively, L and d represent the sequence length and feature dimension respectively.

6. The multimodal sentiment analysis method based on cross-modal attention according to claim 4 is characterized in that: The multimodal fusion layer performs fusion processing on the bimodal joint features to obtain cross-modal interaction features, including: The pairwise interaction features are concatenated with the corresponding bimodal joint features to obtain the joint features; Based on the joint features, a self-attention mechanism is used to obtain the modal internal correlation features of the pairwise interactive features; After filtering the relevant features within the modality, the modal attention features of the pairwise interactive features are obtained, that is, the cross-modal interaction features.

7. The multimodal sentiment analysis method based on cross-modal attention according to claim 1, characterized in that: The output layer splices the unimodal context features and the cross-modal interaction features to obtain multimodal fusion features, and the method for outputting the sentiment analysis result based on the multimodal fusion features is: F1=ReLU(W R Z2+b R ) ICA-MSA=Softmax(F1) in, Represents the concatenation operation of the vector, Z2 is the feature vector obtained by concatenating the unimodal context feature and the modal attention feature, Self VT is the text-visual modality attention feature, Self AV is the visual-speech modality attention feature, Self TA is the speech-text modality attention feature, F1 is the multimodal fusion feature obtained by integrating the concatenated feature vectors through the fully connected layer, and W R and b R are the initialization weights and bias terms of the activation function ReLU, and ICA-MSA is the final classification result.

Images