Multi-modal sentiment analysis algorithm based on multi-scale similarity feature reconstruction

Abstract

The invention discloses a multi-modal sentiment analysis algorithm based on multi-scale similarity feature reconstruction. The multi-modal sentiment analysis algorithm comprises a feature extraction network; a joint representation module; a multi-scale similarity feature reconstruction module; a multi-modal fusion framework based on a symmetric cross-modal attention mechanism; first, features are extracted from a multi-modal representation aspect using a feature extraction network. Then, mapping the single-mode features into a space of the same dimension to obtain a joint representation of the single-mode features, respectively encoding and inputting the two modes into representations, reconstructing and learning sentiments, related to the text mode, of the two modes by using the text mode, and minimizing representation divergence between the two modes; and finally, fusing the multi-modal features by using a symmetric multi-modal fusion framework of a cross-modal attention mechanism, so that the model pays more attention to main emotions. The multi-modal fusion and representation learning method has significant advantages in the field of multi-modal sentiment analysis.

Description

Technical Field

[0001] The present invention relates to a multimodal sentiment analysis algorithm based on multi-scale similarity feature reconstruction, belonging to the technical field of multimodal sentiment analysis. Background Art

[0002] The present invention relates to multimodal sentiment analysis (MSA) technology in the field of natural language processing. At present, MSA technology has been widely used in the fields of human-computer interaction, public opinion analysis, etc. However, information of different modalities is very different in data form and processing method. Although adding one more modal information to a unified model may bring potential improvement in modeling effect, it also increases the complexity and difficulty of modeling. In addition, most traditional methods use alignment-based representation learning methods, which establish semantic relationships between modalities by aligning features or data samples of different modalities. However, there may be multiple possible alignment methods between modalities, and not all elements in one modality correspond to those in another modality. It is difficult to design a similarity measure between modalities. The above problems limit the further development and application of MSA technology. Therefore, the problem faced by the present invention is how to digitize the semantic information contained in multiple modal data into real-valued vectors and realize multimodal alignment, thereby improving the accuracy of multimodal sentiment analysis.

[0003] Combining the deep learning models U-Net and Transformer and applying them to MSA technology can utilize the U-shaped structure and jump connection of the U-Net model to better obtain contextual information and location information, and use Transformer to process this information, thereby improving the prediction accuracy of multimodal sentiment analysis. Summary of the Invention

[0004] In view of the above prior art, the present invention provides a multimodal sentiment analysis algorithm based on multi-scale similarity feature reconstruction, which includes the following steps:

[0005] Step 1: Use feature extraction network to extract text, audio and video feature maps;

[0006] Step 2: Use the joint representation module to map the single modal features to the same dimensional space;

[0007] Step 3: Use the multi-scale similarity feature reconstruction module to minimize the representation differences between multimodal features;

[0008] Step 4: Utilize the multimodal fusion framework with symmetric cross-modal attention mechanism to integrate the multimodal fusion representation;

[0009] Step 5: Use the prediction layer to predict the multimodal fusion features and obtain the predicted emotional results.

[0010] The step 1 specifically includes the following sub-steps:

[0011] Step 1.1: Import necessary libraries and modules into the Python environment, including the deep learning framework PyTorch, natural language processing libraries, etc.

[0012] Step 1.2: Download or load the pre-trained BERT (Bidirectional Encoder Representations from Transformers) model. This paper uses the bert-base-uncased pre-trained BERT.

[0013] Step 1.3: Input the prepared text data into the BERT-Base-Uncased model and perform forward propagation to obtain the text feature map.

[0014] The step 2 specifically includes the following sub-steps:

[0015] Step 2.1: Use the joint representation module to map the unimodal features to the same dimensional space. First, the input feature map is feature mapped. The initial feature dimensions of text, audio, and video are different, and they are mapped to a hidden dimension of 128.

[0016] Step 2.2: Use the joint representation module to extract features from the unimodal features. The output multimodal feature map is fed into the gated recurrent neural network for further feature extraction, maintaining the original latent dimension. Finally, the multimodal features in the same space are obtained through the ReLU activation function.

[0017] The step three specifically includes the following sub-steps:

[0018] Step 3.1: Input the multimodal feature vector to the multi-scale similarity feature reconstruction module. The first part of this module is the auxiliary modality reconstruction module, which converts the audio and video features into vectors of 256, 128, and 64 hidden dimensions. The encoder calculation formula for dimensionality reduction is expressed as: FFOutput = FF(AttnOutput) AttnOutput=LayerNorm(Attn(GRUOutput M )+GRUOutput M )

[0019] Where i represents the UE block. Attn(·) and FF(·) represent the outputs of the multi-head attention layer and feedforward layer in the encoder, respectively. Since the features of the auxiliary modality are extracted, M here only represents the visual and auditory modalities.

[0020] Step 3.2: The decoder upsamples the encoded modal features back to the input modal feature size. It uses a linear transformation layer in the feedforward layer to increase the hidden dimension and restore it to a hidden dimension of 256. The decoder calculation formula is expressed as: AttnOutput=LayerNorm(Attn(UEOutput M )+UEOutput M ) FFOutput = FF(AttnOutput)

[0021] Where i represents the number of UD blocks. Attn(·) and FF(·) represent the outputs of the multi-head attention layer and feedforward layer in the decoder, respectively. The modality of the cross-modal attention in the decoder is set to the text modality. The Q in the multi-head attention layer in the decoder layer represents visual or audio features, while K and V represent text features. The text modality is converted to the same dimension as the corresponding decoder layer through the text modality auxiliary layer for use in subsequent steps.

[0022] Step 3.3: The second module in the multi-scale similarity feature reconstruction module is the text module auxiliary module. It starts by concatenating the text embedding features of the joint representation module and uses the output of the joint representation module as input. Subsequently, these text features are mapped into features of different scales by the fully connected layer to obtain The formula of the fully connected layer is:

[0023] Where d represents the hidden dimension (32, 64, 128). Their feature scales correspond to the scales of each level of the decoder. The text modality auxiliary module enhances the sentiment feature representations in the visual and audio modalities with features from the text modality, achieving semantic alignment and increasing the information content of the sentiment features in the auxiliary modality.

[0024] The step 4 specifically includes the following sub-steps:

[0025] Step 4.1: GRUOutput L and UDOutput A They are sent to the self-attention and cross-modal multi-head attention layers respectively to obtain the high-level representation of the text and the interactive representation between cross-modalities. The attention mechanism formula is: SelfAttn=Attn(GRUOutput L ) CrossAttn = MHAttn(GRUOutput L ,UDOutput A )

[0026] Where MHAtn represents the multi-head attention mechanism, SelfAttn and CrossAttn are the self-attention tensor and cross-modal attention tensor obtained by the multi-head attention mechanism respectively. In the cross-modal attention mechanism, GRUOutput L As a query, GRUOutput A as keys and values;

[0027] Step 4.2: Define two weights W SA and W CA , and a bias b la .W SA and W CA After random initialization, the attention tensors of different proportions are obtained by multiplying them with the SelfAttn and CrossAttn tensors and then adding them together. The formula is expressed as: G=Sigmoid(Z1·W SA +Z2·W CA +b)

[0028] Where G is activated by the activation function to obtain a value in the range [0,1]. The two attention matrices are proportionally added by the value G to obtain the fused attention matrix. The attention matrix is ​​sent to the feedforward layer. Using the initial GRUOutput L Combined with the fused attention matrix as the residual part.

[0029] Step 4.3: Use this fused feature to continue fusion with the visual modality feature. The fusion structure is the same as above. The residual elements are still using GRUOutput L Finally, we get the feature M which is the fusion of three modalities. tav .

[0030] Step 5: The final multimodal fusion feature representation obtained in step 4.3 is fed into the classification layer to perform multimodal sentiment analysis, as shown below:

[0031] in represents the predicted result, and classifier(·) represents the classifier.

[0032] The advantages and benefits of this method are as follows: This method uses the self-attention mechanism and the cross-modal attention mechanism to realize the feature extraction of the text modality and the information interaction between the text modality and the auxiliary modality, and minimizes the representation divergence between multimodal features through the multi-scale similarity feature reconstruction module. This method first uses the first part of the multi-scale similarity feature reconstruction module to locate the information of the sentiment part in the auxiliary modality, and obtains multi-scale information through dimensionality reduction so that it can fully focus on the local features of the auxiliary modality to improve model performance. Then, the second part of the multi-scale similarity feature reconstruction module is used to extract multi-scale text modality features for reconstructing the auxiliary modality. This makes the auxiliary modality features closer to the text modality features, so that the fused representation is mainly based on the sentiment of the text modality. Finally, a multimodal fusion framework with a symmetrical cross-modal attention mechanism is used to integrate the multimodal fusion representations, and the final feature representation is fed to the prediction layer for prediction. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 It is the overall flow chart; Figure 2 This is the flow chart of the multi-scale similarity feature reconstruction module; Figure 3 It is a multimodal fusion framework diagram; DETAILED DESCRIPTION

[0033] The following is based on the attached Figure 1-3 The present invention will be further described: The technical solutions of the present invention will be described clearly and completely below with reference to the accompanying drawings. The embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0034] This paper proposes a multimodal sentiment analysis algorithm based on multi-scale similarity feature reconstruction, the specific steps are as follows Figure 1 shown, including:

[0035] Step 1: Use feature extraction network to extract text, audio and video feature maps;

[0036] Step 2: Use the joint representation module to map the single modal features to the same dimensional space;

[0037] Step 3: Use the multi-scale similarity feature reconstruction module to minimize the representation differences between multimodal features;

[0038] Step 4: Utilize the multimodal fusion framework with symmetric cross-modal attention mechanism to integrate the multimodal fusion representation;

[0039] Step 5: Use the prediction layer to predict the multimodal fusion features and obtain the predicted emotional results.

[0040] The core of the present invention includes three parts: the first part is to extract and map multimodal features, the second part is to extract multi-scale features and reconstruct the auxiliary modality using multi-scale text features, and the third part is to fuse multimodal features.

[0041] In step 1, the feature extraction network uses BERT and a gated recurrent neural network to extract expression feature maps. This process includes the deep learning framework PyTorch, the natural language processing library, etc., downloading and loading the BERT pre-trained model, preparing text data, and feeding it into the BERT pre-trained model for preprocessing.

[0042] In step 2, the joint representation module maps unimodal features to the same dimensional space. The multimodal feature map is then feature mapped, with text, audio, and video features all mapped to 128 hidden dimensions. The output multimodal feature map is fed into a gated recurrent neural network for further feature extraction, while maintaining the original hidden dimensions. Finally, the ReLU activation function is used to obtain multimodal features in the same space.

[0043] Step three utilizes the multi-scale similarity feature reconstruction module to obtain audio and video modal features after text-assisted reconstruction. This module combines the Transformer model with the U-Net model, including an auxiliary reconstruction module and a text-assisted module. This module uses the Transformer as the backbone network, adopts the U-Net network concept, and incorporates a text-assisted modality layer to assist in guiding the video and audio modalities.

[0044] The auxiliary modality reconstruction module includes two parts: encoder and decoder. The encoder part is responsible for extracting features from the video modality and audio modality in the network. In U-Net, it is usually composed of several convolutional layers. The present invention replaces it with the Encoder in Transformer (called U-Encoder block, UE for short). Among them, the Encoder still retains the original structure, but the feedforward layer is modified using the U-Net idea in the feedforward layer. As the input visual and audio modalities pass through these U-Encoder blocks, their hidden dimensions will decrease, thereby reducing the amount of calculation. The present invention uses three U-Encoder blocks to construct the encoder part, and its specific design is as follows: Figure 1 As shown, where B represents the batch size, T represents the sequence length, and D represents the hidden size of the embedding.

[0045] UE includes multi-head attention layers and feedforward layers, each of which includes a residual calculation and layer normalization processing. The structure of UE can be expressed as: FFOutput = FF(AttnOutput) AttnOutput=LayerNorm(Attn(GRUOutput M )+GRUOutput M )

[0046] Here, i represents the number of U-Encoder blocks. Attn(·) and FF(·) represent the outputs of the multi-head attention layer and the feedforward layer, respectively. Since features of the auxiliary modality are extracted, M here only represents the visual and auditory modalities.

[0047] The decoder block (U-Decoder block, UD for short) focuses on upsampling the encoded modal features back to the input modal feature size and is symmetrical with the encoder architecture. It uses a linear transformation layer in the feedforward layer to increase the hidden dimension. Skip connection is a key part of the U-Net network, which helps to retrieve information lost on certain layers to provide better context for the network. In each dimension of the decoder, the output of the corresponding encoder dimension layer is connected to the upsampled modal features. After the skip connection, the connected features are fed into the next decoder layer. The structure of UD can be expressed as: AttnOutput=LayerNorm(Attn(UEOutput M )+UEOutput M ) FFOutput = FF(AttnOutput)

[0048] Due to its symmetrical Transformer structure, unlike the traditional Transformer decoder, which has a multi-head attention layer for masking, this invention removes this layer and retains the cross-modal attention layer in the decoder. However, in order to increase the correlation between modalities and enhance the feature representation of the auxiliary modality, this invention sets the modality of cross-modal attention to textual modality. The Q in the multi-head attention layer in the decoder layer is a visual or audio feature, while K and V are textual features.

[0049] The second part in the multi-scale similarity feature reconstruction module is the text modality auxiliary module, such as Figure 3 It starts by connecting the text embedding features of the joint representation module and uses the output of the joint representation module as input. Subsequently, these text features are mapped into features of different scales by the fully connected layer to obtain It can be expressed as:

[0050] Here, d represents the hidden dimension (32, 64, 128). Their feature scales correspond to the scales of each decoder layer. The text-based auxiliary module uses features from the text modality to enhance the sentiment representations in the visual and audio modalities, achieving semantic alignment and increasing the information content of the sentiment features in the auxiliary modality. The features output by the text-based auxiliary module are only used in the computation of the U-Transformer model and do not participate in subsequent calculations.

[0051] Step four is to fuse multimodal features using a multimodal fusion framework with a cross-modal attention mechanism. This paper proposes a multimodal fusion framework based on a cross-modal attention mechanism to further capture the intermodal interactions between text, vision, and audio. This method improves upon the Encoder layer in BERT and employs a cross-modal attention mechanism for intermodal interactions. Furthermore, to enhance the representation of the text itself and slightly suppress the cross-modal representation between text and other modalities, the paper also introduces a self-attention mechanism to enhance the lower-level text representation.

[0052] according to Figure 2 As shown, GRUOutput L and UDOutput A They are sent to the self-attention and cross-modal multi-head attention layers respectively to obtain the high-level representation of the text and the interactive representation between cross-modalities. The formula is expressed as: SelfAttn=Attn(GRUOutput L ) CrossAttn = MHAttn(GRUOutput L ,UDOutput A )

[0053] Where MHAtn represents the multi-head attention mechanism, SelfAttn and CrossAttn are the self-attention tensor and cross-modal attention tensor obtained by the multi-head attention mechanism respectively. In the cross-modal attention mechanism, GRUOutput L As a query, GRUOutput A as keys and values.

[0054] Define two weights W SA and W CA , and a bias b la .W SA and W CAAfter random initialization, the attention matrices are multiplied with the SelfAttn and CrossAttn tensors to obtain different proportions of attention tensors and then added together. The weights are weighted differently according to the proportion of model learning, thereby enhancing or suppressing the amount of information in the modality. Finally, the bias is added and activated by the activation function, and a value G with a range of [0,1] is compressed. The two attention matrices are proportionally added using the value G to obtain the fused attention matrix. The subsequent structure is the same as in the Encoder, and the attention matrix is ​​sent to the feedforward layer. The final residual part uses the initial GRUOutput L . Use this fusion feature to continue to fuse with the features of the visual modality. The fusion structure is the same as the above structure. The residual elements still use GRUOutput L .

[0055] Step five is to obtain the feature M formed by the fusion of three modalities tav The final feature representation is then fed into the classification layer for multimodal sentiment analysis as follows:

[0056] in represents the predicted result, and classifier(·) represents the classifier.

[0057] In summary, the multimodal sentiment analysis method provided by the present invention addresses the differences in representation methods, semantic representations, and cross-modal relevance between heterogeneous modalities, constructs an effective deep learning model, and improves the algorithm structure. First, a feature extraction network is used to extract multimodal representations. Then, a multi-scale similarity reconstruction module is used to reconstruct auxiliary modalities of different scales using the main modality before multimodal feature fusion, separates out noise in the auxiliary modality that is irrelevant to emotion, reduces the impact on the semantics of the main modality during auxiliary modal fusion, and further improves the correlation of the multimodal feature semantic space. Finally, a multimodal fusion representation with deeper semantics is obtained through a cross-modal attention fusion mechanism. This method can more accurately learn the emotional information in multimodal data, and can improve the model's ability to represent the fusion results, thereby improving the model's sentiment analysis accuracy.

Claims

1. A multimodal sentiment analysis algorithm based on multi-scale similarity feature reconstruction, characterized by: The following steps are involved: Step 1: Use feature extraction network to extract text, audio and video feature maps; Step 2: Use the joint representation module to map the single modal features to the same dimensional space; Step 3: Use the multi-scale similarity feature reconstruction module to minimize the representation differences between multimodal features; Step 4: Utilize the multimodal fusion framework with symmetric cross-modal attention mechanism to integrate the multimodal fusion representation; Step 5: Use the prediction layer to predict the multimodal fusion features and obtain the predicted emotional results.

2. The multimodal sentiment analysis algorithm based on multi-scale similarity feature reconstruction according to claim 1, characterized in that: In step one, The method for extracting text, audio, and video feature maps utilizes a pre-trained BERT model to extract text feature representations. A deep learning framework library is imported and loaded with a pre-trained BERT model. Text data is prepared and pre-processed. The audio and video data are processed using the open-source toolkits OpenSMILE and OpenFace, respectively. The three modal data are aligned to generate the final pre-processed data as a multimodal feature map.

3. The multimodal sentiment analysis algorithm based on multi-scale similarity feature reconstruction according to claim 1, characterized in that: In step 2, The joint representation module maps each single modality feature to the same dimensional space. The modality embedding layer based on the gated recurrent neural network is introduced to unify the representation of each modality at the same time, which can better capture the dependency relationship with large time step distance in the time series. As input, they are sent to the gated recurrent neural network to get the output GRUOutput M . Where M belongs to (L, A, V), GRU represents the gated recurrent neural network model. In order to obtain richer time series information, instead of using the output of the last time step, the output of all time steps is used to consider the information of the entire input sequence.

4. The multimodal sentiment analysis algorithm based on multi-scale similarity feature reconstruction according to claim 1, characterized in that: In step three, The multi-scale similarity feature reconstruction module minimizes the representation divergence between multimodal features. Combining the Transformer and U-Net models, the multi-scale similarity feature reconstruction module is designed, including an auxiliary reconstruction module and a text modality auxiliary module. Using the Transformer as the backbone network and adopting the U-Net network concept, a text auxiliary modality layer is designed to assist in guiding the visual and auditory modalities. The auxiliary modality reconstruction module consists of two parts: an encoder and a decoder. The encoder is responsible for extracting features from the visual and audio modalities in the network. In the U-Net, the convolutional layers are replaced with the encoder in the Transformer (called the U-Encoder block, or UE for short). The encoder retains the original structure, but the feedforward layer is modified using the U-Net concept. UE includes multi-head attention layers and feedforward layers, each of which includes a residual calculation and layer normalization processing. The structure of UE can be expressed as: FFOutput = FF(AttnOutput) AttnOutput=LayerNorm(Attn(GRUOutput M )+GRUOutput M ) Here, i represents the number of U-Encoder blocks. Attn(·) and FF(·) represent the outputs of the multi-head attention layer and the feedforward layer, respectively. Since features of the auxiliary modality are extracted, M here only represents the visual and auditory modalities. The decoder block (U-Decoder block, abbreviated as UD) upsamples the encoded modal features back to the input modal feature size, symmetric with the encoder architecture. It uses linear transformation layers in the feedforward layer to increase the hidden dimension. Skip connections are a key component of the U-Net network. In each dimension of the decoder, the output of the corresponding encoder dimension layer is connected to the upsampled modal features. After the skip connection, the connected features are fed into the next decoder layer. The structure of UD can be expressed as: AttnOutput=LayerNorm(Attn(UEOutput M )+UEOutput M ) FFOutput = FF(AttnOutput) In the multi-scale similarity feature reconstruction module, there is also a module called the text module auxiliary module. It starts by connecting the text embedding features of the joint representation module and uses the output of the joint representation module as input. Subsequently, these text features are mapped into features of different scales by the fully connected layer to obtain Here, d represents the hidden dimension (32, 64, 128). The feature scales correspond to the scales of each decoder layer. The text modality auxiliary module uses features from the text modality to enhance the sentiment feature representations in the visual and audio modalities, achieving semantic alignment and increasing the information content of the sentiment features in the auxiliary modality. However, please note that the features output by this module are only used in the computation of the U-Transformer model and do not participate in subsequent calculations.

5. The multimodal sentiment analysis algorithm based on multi-scale similarity feature reconstruction according to claim 1, characterized in that: In step four, The multimodal fusion framework of the symmetrical cross-modal attention mechanism is used to integrate the representation of multimodal fusion. L and UDOutput A They are sent to the self-attention and cross-modal multi-head attention layers respectively to obtain high-level representations of the text and interactive representations between cross-modalities. SelfAttn=Attn(GRUOutput L ) CrossAttn=MHAttn(GRUOutput L ,UDOutput A ) Where MHAtn represents the multi-head attention mechanism, SelfAttn and CrossAttn are the self-attention tensor and cross-modal attention tensor obtained by the multi-head attention mechanism respectively. In the cross-modal attention mechanism, GRUOutput L As a query, GRUOutput A as keys and values. After this, define two weights W SA and W CA , and a bias b la .W SA and W CA After random initialization, the attention matrices are multiplied with the SelfAttn and CrossAttn tensors to obtain different proportions of attention tensors and then added together. The weights are weighted differently according to the proportion of model learning, thereby enhancing or suppressing the amount of information in the modality. Finally, the bias is added and activated by the activation function, and a value G with a range of [0,1] is compressed. The two attention matrices are proportionally added using the value G to obtain the fused attention matrix. The subsequent structure is the same as in the Encoder, and the attention matrix is ​​sent to the feedforward layer. The final residual part uses the initial GRUOutput L , rather than a fused attention matrix.

6. The multimodal sentiment analysis algorithm based on multi-scale similarity feature reconstruction according to claim 1, characterized in that: In step five, Get the feature M formed by the fusion of three modalities tav The final feature representation is then fed into the classification layer for multimodal sentiment analysis as follows: in Denotes the predicted result, and classifier(·) denotes the classifier. Finally, by updating the parameters of the fully connected layer during the learning process, the model is able to predict the final emotion based on the feature map.

Images