Multi-modal sentiment analysis method based on modal focusing

Abstract

The invention discloses a multi-modal sentiment analysis method based on modal focusing, which takes text, visual and audio modals as centers, focuses on each modal one by one, aims to explore how to perform interactive fusion on information of a main modal and other two secondary modals, and provides a multi-modal sentiment analysis method based on modal focusing in a subsequent fusion process. The three main modes are connected in an effective mode to serve as fusion representation of guidance, and more comprehensive and effective multi-mode information is provided for sentiment analysis tasks. Comprising the following steps: (1) shared mask learning: obtaining a shared mask between a primary mode and a secondary mode by training a cross-modal prediction model; (2) a cross-modal interaction stage: performing interactive fusion on the characteristics of the primary and secondary modals by adopting a cross-modal attention mechanism with a shared mask to obtain fusion representation dominated by each modality; and (3) a subsequent fusion stage: endowing the three fusion expressions with respective weights by using a weighted attention mechanism for connection to obtain a final multi-modal fusion vector. According to the method, the feature relation and cross-modal interaction between the primary and secondary modals are fully mined and utilized, so that the condition of multi-modal information imbalance is processed, and a more accurate sentiment analysis result is achieved.

Description

Technical Field

[0001] The invention relates to a multimodal sentiment analysis method based on modal focus, and belongs to the field of social media analysis. Background Art

[0002] With the continuous development of Internet infrastructure and the popularity of mobile applications, social networking platforms such as Douyin and Bilibili are carrying a large amount of high-quality video data generated by users, which contains rich emotional information. Analyzing multiple modal information such as text, vision and audio in these videos has become one of the important directions for studying people's emotional state in social networks. Multimodal sentiment analysis can not only help the government understand public opinion and emotions, but also help improve the effectiveness of personalized recommendation systems and user experience, bringing huge positive impact and value to society.

[0003] The continuous development of deep learning has driven the continuous updating and iteration of multimodal sentiment analysis technology. Studies have shown that multimodal fusion is crucial in multimodal learning. It can interactively integrate information from various modalities, provide more comprehensive data support for sentiment tasks, and achieve more accurate analysis results. Therefore, how to effectively handle the interaction between modalities has become a key technical node in multimodal fusion. In the study of multimodal sentiment analysis, according to different technologies, researchers can divide the way they deal with the interaction between modalities into two types: one is the non-modal center fusion multimodal sentiment analysis method. This method believes that each modality contributes equally to the sentiment analysis task, so a simple average strategy is used to integrate the information of each modality. However, the essence of this method is to treat the three modal information equally, and does not fully consider the impact of the difference in sentiment information between different modalities on the fusion results. The other is the modality-centered fusion multimodal sentiment analysis method. This method believes that the text modality contributes most significantly to sentiment analysis due to its structured and rich semantic content. Therefore, when performing multimodal information fusion, the text modality is usually treated as the core. However, it is worth noting that different modalities may be unbalanced in sentiment expression, and not all sentiment analysis samples should be dominated by the text modality. Sometimes the audio or visual modality plays a key role, which should not be ignored. Summary of the invention:

[0004] Purpose of the invention: In view of the deficiencies in the prior art, the present invention proposes a multimodal sentiment analysis method based on modality focus suitable for social networks. By focusing on the main contribution of each modality in the sentiment analysis task, this method can handle the situation of unbalanced multimodal information and improve the accuracy and precision of detection.

[0005] Technical solution: A multimodal sentiment analysis method based on modal focus. The method includes the following steps:

[0006] (1) Shared mask learning: obtaining the shared mask between primary and secondary modalities by training a cross-modal prediction model;

[0007] (2) In the cross-modal interaction stage, a cross-modal attention mechanism with a shared mask is used to interactively fuse the features of the primary and secondary modalities, thereby obtaining a fused representation dominated by each modality.

[0008] (3) In the subsequent fusion stage, a weighted attention mechanism is used to assign weights to the three fusion representations for connection, and the final multimodal fusion vector is obtained.

[0009] Preferably, in step (1), obtaining a shared mask between the primary and secondary modalities by training a cross-modal prediction model specifically comprises the following steps:

[0010] (11) Training cross-modal prediction model Seq2Seq (Attention): Given a sequence of text, visual and audio features T is the length of the sequence, and t is the time step. Select any one of the modes as the main mode, and the remaining modes are the secondary modes X i′ , i′∈{l,v,a}, i′≠i. The Seq2Seq (Attention) model is used to predict the feature sequence of the secondary modality from the feature sequence of the primary modality to obtain the attention weight distribution between the primary and secondary modalities:

[0011]

[0012]

[0013] Among them, h j Represents the hidden state after encoding the main modal sequence input at each time step, represents the hidden state related to the secondary modality currently output by the decoder, e tj Yes j and The association score, α tj is the attention weight distribution obtained by performing a softmax operation on the association score.

[0014] (12) Obtaining a shared mask: Obtaining the attention weight distribution α tj Each row of t is sorted by weight, only the larger attention weights are retained, and the rest of the attention weights are deleted. The modified attention weight distribution is mapped to a shared mask, where the attention weight exists is marked as 1, and where it does not exist is marked as 0, to obtain the final shared mask.

[0015] Preferably, in step (2), a cross-modal attention mechanism with a shared mask is used to interactively fuse the features of the primary and secondary modalities to obtain a fusion representation dominated by each modality, which specifically includes the following steps:

[0016] (21) Multimodal feature extraction: Use GloVe word embedding, Facet, and COVAREP to extract multimodal features X from utterances containing text, vision, and hearing. i , i∈{l,v,a}. Then three LSTM networks are used to input multimodal features X i Encode to get U i , the encoded multimodal features are expressed as follows:

[0017] U l =LSTM l (X l ) (3)

[0018] U v =LSTM v (X v ) (4)

[0019] U a =LSTM a (X a ) (5)

[0020] (22) Obtain the attention score between the primary and secondary modalities: Select any one of the three modal features as the primary modal feature u i , the remaining two modal features are used as secondary modal features First, the attention score of the input main modality feature on the secondary modality feature representation to be predicted is calculated. The attention score between the two main and secondary modalities is expressed as follows:

[0021]

[0022]

[0023] in represents the attention score of the primary modality on the secondary modality features, W1, W3, W2, W4, b1, b3 are the parameters of the score calculation function.

[0024] (23) Learning the attention weights between primary and secondary modalities: Obtaining two attention scores of the primary modality feature on the secondary modality feature to be predicted On this basis, the softmax function is further used to calculate the preliminary attention weights:

[0025]

[0026]

[0027] in, represents the result of the exponential attention score of the primary and secondary modalities at specific time steps t1 and t2, represents the sum of the exponential attention scores for each time step within the entire sequence length L, and Represents the attention weights of the main modality i and the secondary modalities i′1, i′2 at time steps t1 and t2. After calculating the weights at all time steps, the preliminary attention weights between the main and secondary modalities will be obtained. and

[0028] In order to achieve the secondary modality's attention to the relevant parts of the primary modality features and mask other features, a method of multiplying the shared mask with the preliminary attention weights is adopted to produce attention weights that only retain the higher correlation between the primary and secondary modalities:

[0029]

[0030]

[0031] in, Represents element-wise multiplication, which is used to multiply the attention weight by the shared mask to achieve the adjustment and restriction of the attention weight, considering the influence of the dominant modality on the secondary modality.

[0032] (24) Obtaining shared features of the secondary modality: The adjusted and restricted attention weights With secondary modal features Multiply them together to get the shared eigenvector of the secondary modes guided by the main mode. and

[0033]

[0034]

[0035] (25) Multimodal fusion vector learning: the shared features of the secondary modalities and the main modal feature u i Connect them and input them into the fusion network LSTM to generate a multimodal fusion vector f dominated by one of the three modes m Then a self-attention layer is further used to learn the final representation F of the multimodal fusion vector m :

[0036]

[0037] F m =SelfAttentionLayer(f m ) (15)

[0038] In step (2), text, vision, and audio are each considered as the primary modality. When any modality serves as the primary modality, the other modalities are considered as secondary modalities. Finally, the three primary and secondary modal combination data of text-vision / audio, vision-text / audio, and audio-text, vision, are obtained and brought into the above process for learning, and finally a multimodal fusion vector dominated by each modality is obtained.

[0039] Preferably, in step (3), in the subsequent fusion stage, a weighted attention mechanism is used to assign weights to the three fusion representations for connection, so as to obtain the final multimodal fusion vector. The specific steps are as follows:

[0040] (31) The three fused representation connections are input into the average pooling layer, and after linear transformation and ReLU activation function processing, the features are extracted:

[0041]

[0042] Then, using the features extracted in the previous step, after linear transformation and Sigmoid activation function, the contribution weight of each fusion vector in the sentiment analysis task is calculated:

[0043]

[0044] Among them, [.] represents concatenation, avepool represents average pooling, and W5, W6, and W7 are the parameter matrices of weighted attention.

[0045] (32) The fusion vector after contribution weight update is input into the regression layer to generate the final sentiment intensity prediction value

[0046] Compared with the existing invention, the present invention has the following advantages:

[0047] (1) The present invention proposes a sentiment analysis method based on modality focus. Compared with the method that under-optimizes non-textual representation and multimodal representation due to excessive dominance of textual modality, or the method that weakens important contributing modal information due to simple averaging strategy, this method focuses on each modality and utilizes cross-modal attention mechanism to achieve full interaction between modalities centered on each modality. At the same time, during the interaction process, shared features are captured with the help of shared masks between primary and secondary modalities to enhance the expression ability of multimodal fusion information.

[0048] (2) The present invention uses weighted attention to determine the contribution of each fusion vector to the sentiment analysis task in the subsequent fusion stage, reducing the interference of redundant information and increasing the influence of more important fusion vectors in a given task.

[0049] (3) Experimental results show that the present invention can effectively deal with information imbalance among multiple modalities. Compared with other methods, it can effectively improve the classification accuracy and achieve better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is an overall flow chart of an embodiment of the present invention;

[0051] Figure 2 This is a specific flow chart of training a cross-modal prediction model Seq2Seq (Attention) according to an embodiment of the present invention;

[0052] Figure 3 A flowchart of obtaining a shared mask according to an embodiment of the present invention;

[0053] Figure 4 A flowchart of fusing primary and secondary modal features using cross-modal attention with a shared mask according to an embodiment of the present invention;

[0054] Figure 5 A flowchart of using a weighted attention mechanism to assign weights to fused representations according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention provides a multimodal sentiment analysis method based on modal focus, which is applicable to video data of social networks. The overall flow chart of the method is as follows: Figure 1 The specific steps are as follows:

[0056] Step (1) obtains the shared mask between the primary and secondary modalities by training the cross-modal prediction model:

[0057] Step (11) Train the cross-modal prediction model Seq2Seq (Attention): The specific process is as follows Figure 2 As shown in Figure 1, the Seq2Seq model with attention is selected as the cross-modal prediction model. Given a sequence of text, visual, and audio features T is the length of the sequence, and t is the time step. Select any one of the modes as the main mode, and the remaining modes are the secondary modes X i′ , i′∈{l, v, a}, i′≠i. The encoder of the Seq2Seq (Attention) model converts the main modal features X i Takes as input and outputs the hidden state:

[0058]

[0059] in, is the main modal feature X i The input to the encoder at time step t, is the hidden state of the encoder at time step t-1, and the hidden state Represents the input at time step t The encoding result is finally obtained by the vector generated by the hidden state sequence

[0060] Next, the decoder uses the previous hidden state at the tth time step and the encoder’s hidden state h j As input, calculate the correlation score between the position of the hidden state encoded by the main modality sequence of the input at each time step and the hidden state of the current output of the decoder:

[0061]

[0062] The correlation score e tj Use the softmax function to calculate the attention weight distribution:

[0063]

[0064] The attention weight α tj With the hidden state h at each time step t j The weighted summation obtains the corresponding context vector:

[0065]

[0066] Finally, the output of the decoder’s next hidden state and the predicted secondary modality sequence at that position is calculated:

[0067]

[0068]

[0069] Step (12) Obtaining the shared mask: After the training process in step (11), the attention weight distribution between the primary and secondary modalities is obtained, and a three-step operation is performed on it. The specific steps are as follows: Figure 3 As shown, we get the shared mask of primary and secondary modes:

[0070] a) Attention weight distribution α obtained during the training of the cross-modal prediction model tj ,The depth of the edge in the attention graph indicates the size of the attention weight.

[0071] b) For each row t of the attention map, the attention weight α tj Sort them, and then only keep the edges with larger weights between each secondary modal feature node and the primary modal node, and delete other edges.

[0072] c) Map the processed attention map to a shared mask smask. If there is an edge between the main modality node t1 and the secondary modality node t2, it means that the predicted secondary modality node t1 is most relevant to the input main modality node t2. is marked as 1, otherwise it is 0.

[0073] Step (2) uses a cross-modal attention mechanism with a shared mask to interactively fuse the features of the primary and secondary modalities, and obtain a fusion representation dominated by each modality. The specific interaction process is as follows: Figure 4 As shown:

[0074] Step (21) Multimodal feature extraction: Given a discourse, on the CMU-MOSI and CMU-MOSEI datasets, GloVe word embedding is used to map the words in the text to a vector representation X in a high-dimensional space l =(l1,l2,...,l n ), use Facet to extract visual features X v =(v1, v2, ..., v n ), use COVAREP to extract acoustic features X a =(a1, a2, ..., a n ), including 12 Mel-frequency cepstral coefficients (MFCCs) and other low-order features. On the CH-SIMS dataset, the word vector X is extracted using the pre-trained Chinese BERT word embedding l =(l1,l2,...,l n ), using MTCNN and MultiComp OpenFace2.0 to extract visual features such as face and eye gaze v =(v1, v2, ..., v n ), using the LibROSA speech toolkit to extract acoustic features X a =(a1, a2, ..., a n ). Then three LSTM networks are used to transform the input multimodal features X l ,X v , X a Encode to get U l , U v ,U a , the encoded multimodal features are expressed as follows:

[0075] U l =LSTM l (X l ) (7)

[0076] U l =LSTM l (X l) (8)

[0077] U l =LSTM l (X l ) (9)

[0078] Step (22) Attention score between primary and secondary modalities: Select any one of the three modal features as the primary modal feature u i , the remaining two modal features are used as secondary modal features In order to calculate the attention score of the input primary modality feature on the predicted secondary modality feature representation, we first transform the primary modality feature u i and secondary modal features The attention scores between the two primary and secondary modalities are expressed as follows:

[0079]

[0080]

[0081] in represents the attention score of the primary modality on the secondary modality features, W1, W3, W2, W4, b1, b3 are the parameters of the score calculation function.

[0082] Step (23) Attention weights between primary and secondary modalities: Obtain two attention scores of the primary modality feature on the secondary modality feature to be predicted On this basis, the softmax function is further used to calculate the preliminary attention weights:

[0083]

[0084]

[0085] in, represents the result of the exponential attention score of the primary and secondary modalities at specific time steps t1 and t2, represents the sum of the exponential attention scores for each time step within the entire sequence length L, and Represents the attention weights of the main modality t and the secondary modalities i′1, i′2 at time steps t1 and t2. After calculating the weights at all time steps, the preliminary attention weights between the main and secondary modalities will be obtained. and

[0086] In order to achieve the secondary modality's attention to the relevant parts of the primary modality features and mask other features, a method of multiplying the shared mask with the preliminary attention weights is adopted to produce attention weights that only retain the higher correlation between the primary and secondary modalities:

[0087]

[0088]

[0089] in, Represents element-wise multiplication, which is used to multiply the attention weight by the shared mask to achieve the adjustment and restriction of the attention weight, considering the influence of the dominant modality on the secondary modality.

[0090] Step (24) Shared features of the secondary modality: The adjusted and constrained attention weights With secondary modal features Multiply them together to get the shared eigenvector of the secondary modes guided by the main mode. and

[0091]

[0092]

[0093] Step (25) Multimodal fusion vector learning: In order to fuse the shared features of the primary modality and the secondary modality, the shared features of the secondary modality are and the main modal feature u i Connect them and input them into the fusion network LSTM. After time series modeling and information integration, a multimodal fusion vector f is generated with one of the three modes as the core. m Then, a self-attention layer is used to learn the association and importance between different modalities, and finally a multimodal fusion vector representation F is obtained. m :

[0094]

[0095] F m =SelfAttentionLayer(f m ) (19)

[0096] In step (2), text, vision, and audio are each considered as the primary modality. When any modality serves as the primary modality, the other modalities are considered as secondary modalities. Finally, the three primary and secondary modal combination data of text-vision / audio, vision-text / audio, and audio-text / vision are obtained and brought into the above process for learning, and finally a multimodal fusion vector dominated by each modality is obtained.

[0097] In step (3), in the subsequent fusion stage, a weighted attention mechanism is used to assign weights to the three fusion representations for connection to obtain the final multimodal fusion vector. Figure 5 The specific steps are as follows:

[0098] Step (31) connects the three fused representations and inputs them into the average pooling layer, which integrates the dimensions to reduce information redundancy and retain important features. Then, after processing in the linear layer, the features of the vector are mapped to a higher-dimensional space to enhance the model's ability to express features. Subsequently, through the application of the ReLU activation function, the nonlinear features in the vector are activated, making it more representative. After the three-step processing, the extracted features are:

[0099]

[0100] In order to quantify the contribution of each fusion vector in the sentiment analysis task, the previously extracted features are subjected to another layer of linear transformation and Sigmoid activation function to obtain the weight corresponding to the fusion vector:

[0101]

[0102] Among them, [.] represents concatenation, avepool represents average pooling, W5, W6, W7 are the parameter matrices of weighted attention, is the weight of the text-dominated fusion vector, is the weight of the vision-driven fusion vector, is the weight of the audio-dominated fusion vector.

[0103] Step (32) inputs the fusion vector after contribution weight update into the regression layer of the two-layer neural network with ReLU activation function. The regression layer gradually maps the multimodal fusion vector to the hidden layer through multiple linear transformations and ReLU nonlinear processing, and generates the final sentiment intensity prediction value.

[0104]

[0105]

[0106] The experimental verification is as follows:

[0107] The datasets selected in this embodiment are CMU-MOSI, CMU-MOSEI and CH-SIMS datasets. CMU-MOSI contains 2198 speech clips with emotion annotations extracted from 93 Youtube movie review videos, covering forms such as monologues, speeches and movies. The emotional score range is [-3, 3], and positive and negative values ​​represent positive and negative emotions respectively. The larger the absolute value, the stronger the emotion. CMU-MOSEI is an improved version of CMU-MOSI, containing 23,453 video utterances from 1,000 speakers and 250 topics. The scoring is carried out from two aspects: emotional intensity and emotion, where the range of emotional intensity is [-3, 3], and the emotional types include happiness, sadness, anger, fear, disgust and surprise. CH-SIMS is a Chinese multimodal emotion dataset, including 60 videos, divided into 2281 clips. The emotional intensity score range of each clip is [-1, 1], and is divided into five types: negative, weak negative, neutral, weak positive and positive. Different from the previous two datasets, CH-SIMS provides three unimodal labels and one multimodal label, which is helpful for analyzing the imbalance of sentiment information between modalities. There are 228 samples with imbalanced sentiment information in the dataset, accounting for 10% of the total samples.

[0108] This example is implemented using the Pytorch framework, and the experimental graphics card supports NVIDIA RTX 2080 Ti graphics card. Adam is used as the optimizer, and dropout is applied before the last linear layer for regularization. The cross-modal prediction model Seq2Seq (Attention) was trained for 40 epochs on the three datasets, with the corresponding Batch Size set to 24, the initial learning rate set to 0.0001, and the dropout set to 0.5. The cross-modal interaction fusion for emotion prediction was trained for 30 epochs on the CMU-MOSI and CMU-MOSEI datasets, with the corresponding Batch Size set to 24, the initial learning rate set to 0.001, and the dropout set to 0.5. It was trained for 30 epochs on the CH-SIMS dataset, with the Batch Size set to 16, the initial learning rate set to 0.002, and the dropout set to 0.2. The following are the experimental results verified on the three datasets.

[0109] Table 1 Experimental results of our method on three datasets

[0110]

[0111] Table 2 Comparison results of methods based on CH-SIMS data set

[0112]

[0113] As can be seen from Table 1, the multimodal sentiment analysis method based on modality focus proposed in this paper has excellent experimental results on the three datasets. Among them, the performance on the CMU-MOSEI dataset is the most outstanding. This result can be attributed to the fact that the sample size of the CMU-MOSEI dataset is the largest, so the overall data fitting effect is better, and the improvement effect of the model in all aspects is better than that of the other two datasets. However, the performance on the CH-SIMS dataset is slightly inferior to that of other datasets. This is mainly due to the presence of 10% of sentiment information imbalance samples in the CH-SIMS dataset, which affects the accuracy of sentiment analysis. To further verify the effectiveness of the proposed method, we selected a representative from the non-modality-centered sentiment analysis method (MuIT) and the modality-centered sentiment analysis method (TETFN) and compared them with the proposed method on the CH-SIMS dataset. The results are shown in Table 2. The comparison results show that in the dataset with imbalanced sentiment information samples, the proposed method is superior to other design methods. This is because our method uses two attention mechanisms to focus on each modality through a cross-modal interaction fusion module centered on each modality, effectively solving the problem of multimodal information imbalance. Compared with the simple averaging strategy adopted by non-modal centric fusion methods, our method technically avoids the weakening of important modal information. At the same time, compared with modality-centric fusion methods, our method not only focuses on the important contribution of text modality, but also fully considers the main position of visual and audio modalities, thus avoiding the problem of insufficient optimization of non-text representation and multimodal representation caused by the dominance of text modality. These results verify the effectiveness of our method in multimodal sentiment analysis.

[0114] The beneficial effects of the present invention are as follows: the modality focus technology proposed in the present invention enhances the contribution of each modality by emphasizing its importance in the sentiment analysis task. The use of shared masks improves the effectiveness of the interaction between the primary and secondary modalities. In addition, by introducing a weighted attention mechanism, the interference of redundant information is reduced, thereby increasing the contribution of useful information.

[0115] The above content is a further detailed description of the present invention in combination with specific / preferred implementation methods. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0116] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A multimodal sentiment analysis method based on modal focus, comprising the following steps: (1) Shared mask learning: obtaining the shared mask between primary and secondary modalities by training a cross-modal prediction model; (2) In the cross-modal interaction stage, a cross-modal attention mechanism with a shared mask is used to interactively fuse the features of the primary and secondary modalities, thereby obtaining a fused representation dominated by each modality. (3) In the subsequent fusion stage, a weighted attention mechanism is used to assign weights to the three fusion representations for connection, and the final multimodal fusion vector is obtained.

2. A multimodal sentiment analysis method based on modal focus as claimed in claim 1, characterized in that In step (1), obtaining the shared mask between the primary and secondary modalities by training the cross-modal prediction model specifically includes the following steps: (11) Training cross-modal prediction model Seq2Seq (Attention): Given a sequence of text, visual and audio features T is the length of the sequence, and t is the time step. Select any one of the modes as the main mode, and the remaining modes are the secondary modes X i′ , i′∈{l, v, a}, i′≠i. The Seq2Seq (Attention) model is used to predict the feature sequence of the secondary modality from the feature sequence of the primary modality to obtain the attention weight distribution between the primary and secondary modalities: Among them, h j Represents the hidden state after encoding the main modal sequence input at each time step, represents the hidden state related to the secondary modality currently output by the decoder, e tj Yes j and The association score, α tj is the attention weight distribution obtained by performing a softmax operation on the association score. (12) Obtaining a shared mask: Obtaining the attention weight distribution α tj Each row of t is sorted by weight, only the larger attention weights are retained, and the rest of the attention weights are deleted. The modified attention weight distribution is mapped to a shared mask, where the attention weight exists is marked as 1, and where it does not exist is marked as 0, to obtain the final shared mask.

3. A multimodal sentiment analysis method based on modal focus as claimed in claim 1, characterized in that In step (2), a cross-modal attention mechanism with a shared mask is used to interactively fuse the features of the primary and secondary modalities, and obtain a fusion representation dominated by each modality. The specific steps are as follows: (21) Multimodal feature extraction: Given a speech containing three modalities: text, vision, and hearing, extract the multimodal feature X i , i∈{l, v, a}. Then three LSTM networks are used to transform the input multimodal features X i Encode to get U i , the encoded multimodal features are expressed as follows: U i =LSTM i (X i ) (3) (22) Learning the attention weights between primary and secondary modalities: Select any one of the three modal features as the primary modal feature u i , the remaining two modal features are used as secondary modal features First, the attention score of the input main modality feature on the secondary modality feature representation to be predicted is calculated Based on the attention score, the softmax function is further used to calculate the preliminary attention weight: In order to achieve the secondary modality's attention to the relevant parts of the primary modality features and mask other features, a method of multiplying the shared mask with the preliminary attention weights is adopted to produce attention weights that only retain the higher correlation between the primary and secondary modalities: in, Represents element-wise multiplication, which is used to multiply the attention weight by the shared mask to achieve the adjustment and restriction of the attention weight, considering the influence of the dominant modality on the secondary modality. (23) Obtaining shared features of the secondary modality: The adjusted and restricted attention weights With secondary modal features Multiply them together to get the shared eigenvector of the secondary modes guided by the main mode. and (24) Multimodal fusion vector learning: the shared features of the secondary modalities and the main modal feature u i Connect them and input them into the fusion network LSTM to generate a multimodal fusion vector f dominated by one of the three modes m Then a self-attention layer is further used to learn the final representation F of the multimodal fusion vector m : F m =SelfAttentionLayer(f m ) (11) In step (2), text, vision, and audio are each considered as the primary modality. When any modality serves as the primary modality, the other modalities are considered as secondary modalities. Finally, the three primary and secondary modal combination data of text-vision / audio, vision-text / audio, and audio-text / vision are obtained and brought into the above process for learning, and finally a multimodal fusion vector dominated by each modality is obtained.

4. The multimodal sentiment analysis method based on modal focus according to claim 1, characterized in that In step (3), a weighted attention mechanism is used to assign weights to the three fusion representations for connection, and the final multimodal fusion vector is obtained. The specific steps are as follows: (31) The three fused representation connections are input into the average pooling layer, and after linear transformation and ReLU activation function processing, the features are extracted: Then, using the features extracted in the previous step, after linear transformation and Sigmoid activation function, the contribution weight of each fusion vector in the sentiment analysis task is calculated: Among them, [.] represents concatenation, avepool represents average pooling, and W5, W6, and W7 are the parameter matrices of weighted attention. (32) The fusion vector after contribution weight update is input into the regression layer to generate the final sentiment intensity prediction value

Images