An attention network-based cross-modal sentiment analysis method

Abstract

This invention belongs to the fields of natural language processing, computer vision, and sentiment analysis. It discloses a cross-modal sentiment analysis method based on attention networks, comprising: Step 1: Extracting image features, image-text features, and aspect features; Step 2: The extracted image-text features enter a modality update layer. Each modality update layer includes one modality alignment module and two modality update modules. Each modality is aligned within the modality alignment module and then enters the modality update modules. By utilizing the correlation between different modalities, the features are gradually supplemented to finally obtain the interacted image and text features; Step 3: The image and text features are fused using a self-attention mechanism for multimodal analysis; Step 4: The image and text features are concatted with the multimodal features to perform sentiment prediction. This invention fully utilizes cross-modal information interaction, which helps improve the accuracy of sentiment prediction.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, computer vision, and sentiment analysis, specifically to a cross-modal sentiment analysis method based on attention networks. Background Technology

[0002] With the development of various online social platforms and network technologies, users are expressing their opinions online in more diverse ways, with an increasing number choosing to use videos, pictures, or articles to express their emotions and viewpoints. Analyzing the emotional tendencies and public opinion guidance contained within this multimodal information has become a challenge in the field of sentiment analysis. However, due to the heterogeneity and asynchronicity of multimodal data, fusing multimodal information is not easy. Regarding heterogeneity, different modalities exist in different feature spaces. Regarding asynchronicity, inconsistent sampling rates of time-series data from different modalities make it impossible to obtain the optimal mapping between different modalities. There is already a great deal of research on multimodal analysis, and the specific methods can be summarized into two categories: one uses cross-modal attention to provide soft mappings between different modalities, thereby modeling the asynchronicity of multimodal data. However, this type of method does not consider the heterogeneity of multimodal data. The other category considers the heterogeneity of multimodal data. Methods in this category divide each modality into a shared part and a private part, represented by different neural networks. The limitation of these methods is that they do not consider the asynchronicity between different modalities. Summary of the Invention

[0003] To address the issues of multimodal heterogeneity and heterogeneity, this invention proposes a cross-modal sentiment analysis method based on attention networks. It employs a modality alignment module and a modality update module, and utilizes an attention mechanism to perform cross-modal interaction, thereby improving the accuracy of multimodal sentiment analysis.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0005] This invention is a cross-modal sentiment analysis method based on attention networks, specifically including the following steps:

[0006] Step 1: Extract the image features and image-text features corresponding to the input image text;

[0007] Step 2: The extracted image and text features enter the modality update layer. Each modality update layer includes a modality alignment module for aligning the representation space and two modality update modules. Each modality is aligned within the modality alignment module and then enters the modality update module. By utilizing the correlation between different modalities, the features are gradually supplemented to finally obtain the interactive image and text features.

[0008] Step 3: Use a self-attention mechanism to perform multimodal fusion of the interactive image features and text features obtained in Step 2 to obtain multimodal features;

[0009] Step 4: Perform a concat operation on the image features and image-text features from Step 1 and the fused multimodal features from Step 3 to predict sentiment.

[0010] Preferred: Step 2 specifically includes the following steps:

[0011] Step 2.1: The modal alignment module aligns the feature spaces of different modalities before modal interaction to obtain multimodal information;

[0012] Step 2.2: The aligned multimodal information enters the modality update module, progressively enhancing each modality. Each modality update layer contains two modality update modules. and Specifically, the text update module and the image update module employ an aspect-guided attention method in the first layer of the modality update layer to focus the text and visual features on the information portion of a given aspect and suppress less important parts. The specific process is as follows:

[0013]

[0014] in The hidden representation of the generated target mode, I A Representative feature vector, b (1) Represents learnable parameters. Indicates a variable parameter. Represents the mode vector;

[0015] Note the weights when calculating normalization:

[0016]

[0017] Using attention weights A new target mode vector is obtained by weighted averaging of the eigenvectors of the target mode.

[0018] Step 2.3: To capture bidirectional interactions between different modalities and enhance intermodal interactions, the modality update module introduces cross-modal attention and self-attention mechanisms to enhance the target modality. The specific process is as follows:

[0019]

[0020] Where * represents the target modality to be enhanced, and α represents the complementary modality. If the target modality is text, then the complementary modality is an image. The formula is as follows:

[0021]

[0022]

[0023] Among them, SA mul CMA mul The `Att` sub-table represents multi-head self-attention mechanism, multi-head cross-modal attention mechanism, normalization function, and additive attention mechanism. To better integrate image and text modalities, this invention uses additive attention mechanism, specifically represented as follows:

[0024]

[0025]

[0026]

[0027] Where G, W c b c The weights of each modality update module are dynamically calculated using an additive attention mechanism, representing learnable parameters, to achieve information exchange between the two modalities, ultimately resulting in a strengthened multimodal sequence. and

[0028] To learn deep abstract representations of multimodal features, a GRU is used to combine the results of the interactive attention mechanism with the input of the current layer. In the nth layer, a cross-modal attention mechanism and a self-attention mechanism are first used to obtain the enhanced multimodal sequence. Then, a GRU is used to obtain new text and image features. Here, n does not include the first layer. The first layer uses an aspect-guided attention mechanism. The specific process is as follows:

[0029]

[0030] Among them: SA mul Represents a multi-head self-attention mechanism. Let n be the target modal vector, and n represent the number of layers.

[0031] Preferably, in step 3, the image features and text features obtained in step 2 are fused using a self-attention mechanism for multimodal fusion, as specifically shown below:

[0032]

[0033] in: Both represent multimodal sequences, and FC is a fused multimodal function.

[0034] Preferably, step 4 specifically involves performing a concat operation on the text features, image features, and fused multimodal features from steps 1 and 3 to obtain a feature representation E containing the three features. mul As input data:

[0035] E mul =concat(X) mul ,X L ,X V )

[0036] A fully connected network is used to fuse features from the data, and a softmax classifier is used in the last layer for sentiment prediction. The sentiment prediction calculation formula is as follows:

[0037] P = softmax(W) m E+b m )

[0038] Among them W m b represents the weights of the fully connected layer. m P represents bias, and P represents sentiment prediction.

[0039] The preferred method for extracting image features using the VGG16 network is as follows:

[0040] Step 11: Input: Input a 224*224*3 image pixel matrix;

[0041] Step 12: Convolutional Pooling: The input image pixel matrix undergoes 5 rounds of convolutional pooling operations, with each round having a kernel size of 3*3*w, where w represents the matrix depth. After convolution, multiple feature maps are obtained through the ReLU activation function, and max pooling is used to filter local features. The convolution calculation formula is as follows:

[0042] f j =R(X) i *K j +b)

[0043] Where R represents the ReLU activation function, * represents the convolution operation, b represents the bias term, and K... j Convolution kernels representing different matrix depths;

[0044] Step 13: Fully connected: After three fully connected layers, a 1*1*1000 image feature representation vector is obtained;

[0045] Step 14: Finally, obtain the image feature vector using the pre-trained VGG16 network and use X. Vp ={X V1 ,X V2 …X Vn}express.

[0046] Preferred method: Step 1 uses a BERT pre-trained model to obtain image text features. The specific process is as follows:

[0047] Step 21: Text preprocessing: Preprocess meaningless words and symbols in online slang, and delete words that do not affect the judgment of the sentiment of the text as stop words;

[0048] Step 22: Extract the word vector sequence from the input text using a pre-trained BERT model. The input text, after segmentation and labeling, takes the word sequence as input and processes it through word embedding, segment embedding, and position embedding in a stack, ultimately generating word vectors representing text features. X Lp ={X L1 ,X L2 …X Ln}express.

[0049] Preferred method: The method for extracting aspect features from the given aspect phrase in step 1 is as follows:

[0050] Given an aspect phrase A = {A1, A2…A} n First, word embeddings are used to obtain the word embedding vector a. j Then, a bidirectional LSTM model is used to learn the hidden representation V of the word embedding vector for each aspect. j :

[0051]

[0052] Then take all hidden representations V j The average value is used as the final aspect feature vector V. A :

[0053]

[0054] The beneficial effects of this invention are:

[0055] (1) The sentiment analysis method of the present invention utilizes a modality alignment module and a modality update module, and adopts an attention mechanism to perform cross-modal interaction, thereby improving the accuracy of multimodal sentiment analysis.

[0056] (2) The modal update module of the present invention includes a modal alignment module and a modal update module. The modal alignment module is used to align the feature sequences of different modalities, which helps the interaction between modalities.

[0057] (3) The modal update module utilizes a multi-head self-attention mechanism and a cross-modal attention mechanism to enhance the interaction between modalities and fully integrate the shared and private characteristics of different modalities.

[0058] (4) In order to preserve the rich features between modalities, the present invention fuses the fused multimodal features with the initial modal features again, and then performs sentiment classification.

[0059] (5) This invention makes full use of cross-modal information interaction, which helps to improve the accuracy of sentiment prediction. Attached Figure Description

[0060] Figure 1 This is a flowchart of the sentiment analysis method of the present invention.

[0061] Figure 2 This is a diagram of the sentiment analysis method architecture of the present invention.

[0062] Figure 3 This is a diagram of the modal update module of the present invention. Detailed Implementation

[0063] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0064] like Figure 1-3 As shown, this invention is a cross-modal sentiment analysis method based on attention networks. It proposes a cross-modal sentiment analysis model based on visual attention networks, enhancing the information interaction between the text and image modalities through a modality update layer, thereby improving the robustness and accuracy of the model. Specifically, this cross-modal sentiment analysis method includes the following steps:

[0065] Step 1: Extract the image features, image text features, and aspect features of the given aspect phrases corresponding to the input image text.

[0066] The VGG16 network is used to extract image features. The VGG16 network consists of 13 convolutional layers, 5 pooling layers, and 3 fully connected layers. The convolutional layers obtain image feature maps through convolution. Dot multiplication is applied to the image representation matrix, with the convolutional kernel sliding with a certain stride and multiplying with each element at the corresponding position and the input matrix unit, ultimately obtaining the image feature map based on the current convolutional kernel. The pooling layers reduce the dimensionality of the convolutional feature map and use max pooling to filter local features. Finally, the fully connected layers synthesize the output features from the upper layers.

[0067] The specific process for extracting image features is as follows:

[0068] Step 11: Input: Input a 224*224*3 image pixel matrix;

[0069] Step 12: Convolutional Pooling: The input image pixel matrix undergoes 5 rounds of convolutional pooling operations, with each round having a kernel size of 3*3*w, where w represents the matrix depth. After convolution, multiple feature maps are obtained through the ReLU activation function, and max pooling is used to filter local features. The convolution calculation formula is as follows:

[0070] f j =R(X) i *K j +b)

[0071] Where R represents the ReLU activation function, * represents the convolution operation, b represents the bias term, and K... j Convolution kernels representing different matrix depths;

[0072] Step 13: Fully connected: After three fully connected layers, a 1*1*1000 image feature representation vector is obtained;

[0073] Step 14: Finally, obtain the image feature vector using the pre-trained VGG16 network.

[0074] X Vp ={X V1 ,X V2 …X Vn}express.

[0075] The BERT pre-trained model is used to obtain text features from images. The specific process is as follows:

[0076] Step 21: Text preprocessing: Preprocess meaningless words and symbols in online slang, and delete words that do not affect the judgment of the sentiment of the text as stop words;

[0077] Step 22: Extract the word vector sequence from the input text using a pre-trained BERT model. The input text, after segmentation and labeling, takes the word sequence as input and processes it through word embedding, segment embedding, and position embedding in a stack, ultimately generating word vectors representing text features. X Lp ={X L1 ,X L2 …X Ln}express.

[0078] The method for extracting aspect features from a given aspect phrase is as follows:

[0079] Given an aspect phrase A = {A1, A2…A} n First, word embeddings are used to obtain the word embedding vector a. j Then, a bidirectional LSTM model is used to learn the hidden representation V of the word embedding vector for each aspect. j :

[0080]

[0081] Then take all hidden representations V j The average value is used as the final aspect feature vector V. A :

[0082]

[0083] Step 2: The extracted image and text features enter the modality update layer. Each modality update layer includes a modality alignment module for aligning the representation space and two modality update modules. Each modality is aligned within the modality alignment module and then enters the modality update module. By utilizing the correlation between different modalities, the features are gradually supplemented to finally obtain the interactive image and text features.

[0084] Step 2.1: The modality alignment module aims to align the feature spaces of different modalities before modal interaction. First, it maps the single-modal representations of multiple modalities to the same storage space, as shown below:

[0085]

[0086] in Represents a text vector, Mem n Represents a storage space vector, and θ represents a parameter. Let f(·) represent the aligned modal vector, f(·) represent the exchange function between the modal vector and the storage space vector, and n represent the nth modal update layer. The specific calculation process of the modal alignment module is as follows:

[0087]

[0088] K = Mem n ·W K

[0089] Among them W q and W K The parameter Q represents the linear transformation. * The vectors represent the linear transformations of the two modes, and K represents the size of the storage space. The formula for calculating the similarity between the mode vectors and the storage space vectors is as follows:

[0090]

[0091] The weight of the j-th storage vector is represented as:

[0092]

[0093] The storage space vector, after a linear transformation, is represented as follows:

[0094] V = Mem n ·W v

[0095] W v The query vector represents the learnable parameters and is obtained by calculating the memory space vector and weights.

[0096]

[0097] Where: *∈{L,V} represents image features and text features, V *j This represents a memory space vector.

[0098] Step 2.2: The aligned multimodal information enters the modality update module, progressively enhancing each modality. Each modality update layer contains two modality update modules. and Specifically, the text update module and the image update module employ an aspect-guided attention method in the first layer of the modality update layer to focus the text and visual features on the information portion of a given aspect and suppress less important parts. The specific process is as follows:

[0099]

[0100] in The hidden representation of the generated target mode, I A Representative feature vector, b (1) Represents learnable parameters. Indicates a variable parameter. Represents the mode vector;

[0101] Note the weights when calculating normalization:

[0102]

[0103] Using attention weights A new target mode vector is obtained by weighted averaging of the eigenvectors of the target mode.

[0104] Step 2.3: To capture bidirectional interactions between different modalities and enhance intermodal interactions, the modality update module introduces cross-modal attention and self-attention mechanisms to enhance the target modality. The specific process is as follows:

[0105]

[0106] Where * represents the target modality to be enhanced, and α represents the complementary modality. If the target modality is text, then the complementary modality is an image. The formula is as follows:

[0107]

[0108]

[0109] Among them, SA mul CMA mul The `Att` sub-table represents multi-head self-attention mechanism, multi-head cross-modal attention mechanism, normalization function, and additive attention mechanism. To better integrate image and text modalities, this invention uses additive attention mechanism, specifically represented as follows:

[0110]

[0111]

[0112]

[0113] Where G, W c b c The weights of each modality update module are dynamically calculated using an additive attention mechanism, representing learnable parameters, to achieve information exchange between the two modalities, ultimately resulting in a strengthened multimodal sequence. and

[0114] Step 2.3 To learn a deep abstract representation of multimodal features, GRU is used to combine the results of the interactive attention mechanism with the input of the current layer. In the nth layer, cross-modal attention and self-attention mechanisms are first used to obtain the enhanced multimodal sequence, and then GRU is used to obtain new text and image features. Here, n does not include the first layer. The first layer uses an aspect-guided attention mechanism. The specific process is as follows:

[0115]

[0116] Among them: SA mul Represents a multi-head self-attention mechanism. Let n be the target modal vector, and n represent the number of layers.

[0117] Step 3: The image features and text features obtained in Step 2 are fused using a self-attention mechanism for multimodal fusion, as shown below:

[0118]

[0119] in: Both represent multimodal sequences, and FC is a fused multimodal function.

[0120] Step 4: Perform a concat operation on the image features and image-text features from Step 1 with the fused multimodal features from Step 3 to obtain E, which includes three feature representations. mul As input data:

[0121] Emul =concat(X) mul ,X L ,X V )

[0122] A fully connected network is used to fuse features from the data, and a softmax classifier is used in the last layer for sentiment prediction. The sentiment prediction calculation formula is as follows:

[0123] P = spftmax(W) m E+b m )

[0124] Among them W m b represents the weights of the fully connected layer. m P represents bias, and P represents sentiment prediction.

[0125] To preserve richer features between modalities, this invention uses L2 loss as the loss function, as detailed below:

[0126]

[0127] Where α represents the hyperparameter.

[0128] This invention makes full use of cross-modal information interaction, which helps to improve the accuracy of sentiment prediction.

Claims

1. A cross-modal sentiment analysis method based on attention networks, characterized in that: The cross-modal sentiment analysis method includes the following steps: Step 1: Extract the image features, image text features, and aspect features of the given aspect phrases corresponding to the input image text; Step 2: The extracted image and text features enter the modality update layer. Each modality update layer includes a modality alignment module for aligning the representation space and two modality update modules. Each modality is aligned within the modality alignment module and then enters the modality update module. By utilizing the correlation between different modalities, the features are gradually supplemented to finally obtain the interactive image and text features. Step 3: Use a self-attention mechanism to perform multimodal fusion of the interactive image features and text features obtained in Step 2 to obtain multimodal features; Step 4: Perform a concat operation on the image features and image-text features from Step 1 with the fused multimodal features from Step 3 to predict sentiment. Step 2 specifically includes the following steps: Step 2.1: The modal alignment module aligns the feature spaces of different modalities before modal interaction to obtain multimodal information; Step 2.2: The aligned multimodal information enters the modality update module, progressively enhancing each modality. Each modality update layer contains two modality update modules. and Specifically, the text update module and the image update module employ an aspect-guided attention method in the first layer of the modality update layer. The specific process is as follows: in The hidden representation representing the generated target modality. Representative feature vector, Represents learnable parameters. Indicates a variable parameter. Represents the mode vector; Note the weights when calculating normalization: Using attention weights A new target mode vector is obtained by weighted averaging of the eigenvectors of the target mode. ; Step 2.3: To capture bidirectional interactions between different modalities and enhance intermodal interactions, the modality update module introduces cross-modal attention and self-attention mechanisms to enhance the target modality. The specific process is as follows: in, This represents the target modality to be enhanced. This represents the complementary modality. If the target modality is text, then the complementary modality is an image, as shown in the formula below: in, , The `Att` sub-table represents multi-head self-attention mechanism, multi-head cross-modal attention mechanism, normalized function, and additive attention mechanism. The additive attention mechanism is specifically represented as follows: in , , The weights of each modality update module are dynamically calculated using an additive attention mechanism, representing learnable parameters, thereby achieving information exchange between the two modalities and ultimately obtaining the enhanced multimodal sequence. and ; In step 2.3, to learn a deep abstract representation of multimodal features, GRU is used to combine the result of the interactive attention mechanism with the input of the current layer. The layer first uses cross-modal attention and self-attention mechanisms to obtain the enhanced multimodal sequence, and then uses GRU to obtain new text and image features. The specific process is as follows: in: Represents a multi-head self-attention mechanism. For the target mode vector, Represents the number of floors.

2. The cross-modal sentiment analysis method based on attention networks according to claim 1, characterized in that: In step 3, the image features and text features obtained in step 2 are fused using a self-attention mechanism for multimodal fusion, as specifically shown below: in: Both represent multimodal sequences. It is a fusion of multimodal functions.

3. The cross-modal sentiment analysis method based on attention networks according to claim 1, characterized in that: Step 4 specifically involves: combining the text features, image features, and fused multimodal features from steps 1 and 3. The operation yields a representation containing three features. As input data: Use a fully connected network to perform feature fusion on the data, and use it in the last layer. The classifier performs sentiment prediction, and the sentiment prediction calculation formula is as follows: in Represents the weights of the fully connected layer. Represents bias. Represents sentiment prediction.

4. The cross-modal sentiment analysis method based on attention networks according to claim 1, characterized in that: The method for extracting aspect features from the given aspect phrase in step 1 is as follows: Given aspect phrases First, word embeddings are used to obtain word embedding vectors. Then, a bidirectional LSTM model is used to learn the hidden representation of the word embedding vector for each aspect. : Then retrieve all hidden representations. The average value is used as the final aspect feature vector. : 。 5. The cross-modal sentiment analysis method based on attention networks according to claim 1, characterized in that: Step 1 uses the VGG16 network to extract image features. The VGG16 network consists of 13 convolutional layers, 5 pooling layers, and 3 fully connected layers. The specific process of using the VGG16 network to extract image features is as follows: Step 11: Input: Input The image pixel matrix; Step 12: Convolutional Pooling: The input image pixel matrix undergoes 5 rounds of convolutional pooling operations, with each round having a kernel size of [missing value]. , Representing the matrix depth, after convolution, multiple feature maps are obtained through the ReLU activation function, and max pooling is used to filter local features. The convolution calculation formula is as follows: Where R represents the ReLU activation function. 'b' represents the convolution operation, and 'b' represents the bias term. Convolution kernels representing different matrix depths; Step 13: Fully connected: After three fully connected layers, we obtain... Image feature representation vector; Step 14: Finally, obtain the image feature vector using the pre-trained VGG16 network. express.

6. The cross-modal sentiment analysis method based on attention networks according to claim 1, characterized in that: Step 1 uses a BERT pre-trained model to obtain image text features. The specific process is as follows: Step 21: Text preprocessing: Preprocess meaningless words and symbols in online slang, and delete words that do not affect the judgment of the sentiment of the text as stop words; Step 22: Extract the word vector sequence from the input text using a pre-trained BERT model. The input text, after segmentation and labeling, takes the word sequence as input and processes it through word embedding, segment embedding, and position embedding in a stack, ultimately generating word vectors representing text features. express.

Images