A collaborative attention fusion-based aspect-level multimodal sentiment analysis method

Abstract

This paper proposes an aspect-level multimodal sentiment analysis method based on a collaborative attention-based global-local feature fusion network. The method includes the following steps: 1) obtaining a left text context representation, a right text context representation, an aspect aggregation representation, and a local image representation; 2) generating an aspect-guided global text feature representation and an aspect-guided global image feature representation; 3) constructing a cross-modal feature interaction mechanism to simultaneously generate a visually integrated local text feature representation and a text-integrated local visual feature representation; 4) building a gated multimodal fusion mechanism; and 5) obtaining sentiment polarity prediction results. This method improves performance in aspect-level multimodal sentiment analysis tasks, effectively capturing global semantic correlations within modalities and local semantic alignment between modalities.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal content understanding and data analysis, and in particular to an aspect-level multimodal sentiment analysis method based on collaborative attention fusion. Background Art

[0002] Aspect-level (i.e., entity-dependent) sentiment analysis of social media posts has recently gained increasing attention. Given a sentence, a corresponding image, and a target entity, the goal is to predict the sentiment of the target entity mentioned in the user's post. Aspect-level sentiment analysis can more accurately express fine-grained sentiment. Most existing tasks focus on analyzing sentiment for the sentence or image as a whole, while aspect-level sentiment analysis considers not only the implicit sentiment in the sentence or image but also the target to which the sentiment is attached, as different entities correspond to different entities. Therefore, how to construct an alignment between opinion targets and textual / visual content in a unified model, model intra-modal dynamics, and discover aligned fusion between modalities remains a challenge.

[0003] In existing aspect-level multimodal sentiment analysis, most of them realize multimodal feature fusion from the perspective of global features, without considering the potential correlation of local fine-grained features in different modalities. The key content of different modalities pointed to by aspects may be localized, so the fine-grained semantic association between each word in the text and each local area in the image needs to be modeled. Since the attention mechanism has applications in various fields, in the multimodal feature fusion of the present invention, a gated collaborative attention mechanism is used to learn the local semantic alignment of text-guided image context and the local semantic alignment of image-guided text context. In addition, multimodal information at different levels is hierarchically fused to achieve multi-layer deep fusion of global-local features for aspect-level multimodal sentiment analysis. Summary of the Invention

[0004] This paper addresses the challenges of aspect-level multimodal sentiment analysis and proposes a collaborative attention fusion-based aspect-level multimodal sentiment analysis method. This method can better capture the dynamics within and between modalities, achieving better aspect-level multimodal sentiment analysis results.

[0005] The technical solution for realizing the present invention is:

[0006] An aspect-level multimodal sentiment analysis method based on collaborative attention fusion includes the following steps:

[0007] 1) Obtaining left-text context representation, right-text context representation, aspect aggregation representation, and local image representation: Use the pre-trained BERT model or Glove word embedding to obtain a text embedding sequence. Then use a long short-term memory network to learn the contextual dependencies of the text embedding sequence to obtain the left-text context representation, right-text context representation, and aspect representation. The aspect representations are average-pooled to obtain the aspect aggregation representation. For images, a ResNet network is used to obtain the image feature representation and convert it into a local image representation.

[0008] 2) Generate aspect-guided global text feature representation and aspect-guided global image feature representation: Use the attention mechanism guided by the aspect aggregate representation to guide the left text context representation and the right text context representation obtained in step 1), respectively. Then use low-rank linear pooling to interact the aspect aggregate representation with the left and right text context representations guided by the aspect aggregate representation, and concatenate the results of the interaction to obtain the aspect-guided global text feature representation. Use the attention mechanism guided by the aspect aggregate representation to guide the local image representation obtained in step 1) to obtain the aspect-guided global image feature representation.

[0009] 3) Construct a cross-modal feature interaction mechanism to simultaneously generate local text feature representations integrated with vision and local visual feature representations integrated with text: Use multi-head self-attention to learn the text embedding sequence and image local representation obtained in step 1), then obtain text sequence context features and image local context features. Use a text-guided visual context attention mechanism to obtain text-guided visual context features, and then fuse them with the text sequence context features through a gating mechanism to generate local text feature representations integrated with vision. At the same time, use a visual-guided text context attention mechanism to obtain visual-guided text context features, and fuse them with the image local context features through a gating mechanism to generate local visual feature representations integrated with text.

[0010] 4) Constructing a gated multimodal fusion mechanism: Using a gating mechanism, the aspect-guided global text feature representation obtained in step 2) is filtered and fused with the aspect-guided global image feature representation obtained in step 2) to obtain a global multimodal fusion representation. Simultaneously, the vision-integrated local text feature representation and the text-integrated local visual feature representation obtained in step 3) are fused to obtain a local multimodal fusion representation.

[0011] 5) Obtaining the sentiment polarity prediction result: The global text feature representation and global image feature representation obtained in step 2), the global multimodal fusion representation obtained in step 4), and the local multimodal fusion representation are concatenated and input into the softmax function to obtain the polarity of the sentiment classification.

[0012] Obtaining the left text context representation, the right text context representation, the aspect aggregation representation, and the image local representation in step 1) includes the following steps:

[0013] 1.1) First, a pre-trained BERT model or Glove word embedding is used to obtain a text embedding sequence. Then, a long short-term memory network is used to learn the contextual dependencies of the text embedding sequence. This is used to obtain the left text context representation, the right text context representation, and the aspect representation. The aspect representations are then average-pooled to obtain the aspect aggregate representation.

[0014] 1.2) For images, the ResNet model is used to extract image features to obtain image feature representation, and the image is divided into different regions to obtain image local representation in order to facilitate fine-grained guidance of image local features.

[0015] Generating the aspect-guided global text feature representation and the aspect-guided global image feature representation in step 2) comprises the following steps:

[0016] 2.1) For generating aspect-guided global text feature representation:

[0017] ① Using the bilinear interaction between the left text context representation and the aspect aggregation representation obtained in step 1), calculate the attention weight of each hidden state in the left text context, and then perform the weighted summation to obtain the left text context representation guided by the aspect aggregation representation. The same method is used to obtain the right text context representation guided by the aspect aggregation representation.

[0018] ② Low-rank bilinear pooling is used to learn the interaction between aspect aggregation representation and left and right text context representation guided by aspect aggregation representation, and the obtained features are concatenated to obtain aspect-guided global text feature representation. Low-rank bilinear pooling can focus on the pairwise interaction of features and can achieve effective attention to aspects and left and right context;

[0019] 2.2) For the aspect-guided global image feature representation, the bilinear interaction between the local image representation obtained in step 1) and the aspect aggregation representation is used to calculate the attention weight of each region of the image, and then the weighted sum is performed to obtain the aspect-guided global image feature representation.

[0020] The step 3) of constructing a cross-modal feature interaction mechanism to simultaneously generate a local text feature representation integrated with vision and a local visual feature representation integrated with text comprises the following steps:

[0021] 3.1) Use multi-head self-attention to learn the text embedding sequence and image local representation obtained in step 1) respectively;

[0022] 3.2) Design text-guided visual context attention and visual-guided text-context attention to model cross-modal semantic correlation alignment respectively:

[0023] ① Integrate local text feature representation into vision;

[0024] First, the vector of the jth word in the text sequence context feature after multi-head self-attention learning and the i-th region vector in the image local context feature after multi-head self-attention learning are projected into the c-dimensional common space, and then the features of the two are fused. Subsequently, the fused vector is linearly transformed, and the attention scores of the jth word in the text sequence context feature and other image local context features are calculated in turn. The normalized attention scores are obtained by softmax operation, and then the attention scores are weighted to each image local feature area and summed to obtain the visual context feature guided by the jth word. Since different modalities contribute differently to emotions, a gated fusion method is used to aggregate the word and the visual context features guided by the word to obtain the visual context features guided by all words. These features are averagely pooled to obtain the local text feature representation integrated into the vision.

[0025] ② Local visual feature representation integrated into text:

[0026] First, the i-th region vector in the image local context feature after multi-head self-attention learning and the j-th word vector in the text sequence context feature after multi-head self-attention learning are projected into the c-dimensional common space, and then the features of the two are fused. Subsequently, the fused vector is linearly transformed, and the attention scores of the i-th visual block and other text sequences in the image local context feature are calculated in turn. The normalized attention scores are obtained by softmax operation, and then the attention scores are weighted to each text sequence and summed to obtain the text context feature guided by the i-th visual block. Since different modalities contribute differently to emotions, a gated fusion method is used to aggregate the visual block and the text context feature guided by the visual block to obtain the text context features guided by all visual blocks. These features are averaged and pooled to obtain the local visual feature representation integrated into the text.

[0027] The construction of the gated multimodal fusion mechanism in step 4) includes the following steps:

[0028] 4.1) Using a gating mechanism to represent the aspect-guided global text feature obtained in step 2) is used to filter and fuse the aspect-guided global image feature obtained in step 2);

[0029] 4.2) Bilinear pooling is used to capture the interaction between the global text feature representation and the filtered aspect-guided global image features to obtain a global multimodal fusion representation;

[0030] 4.3) Simultaneously, the local text feature representation integrated with vision obtained in step 3) and the local visual feature representation integrated with text are nonlinearly fused to obtain a local multimodal fusion representation;

[0031] The sentiment polarity prediction result of the aspect word is obtained in the step 5): the global text feature representation, the global image feature representation obtained in step 2), the global multimodal fusion representation obtained in step 4), and the local multimodal fusion representation are hierarchically spliced ​​and input into the softmax function to obtain the sentiment polarity prediction result corresponding to the aspect word.

[0032] Compared with the existing technology, the beneficial effects of this technical solution are:

[0033] 1. This technical solution designs two text encoding methods to obtain feature representations of text. It also divides the text into left and right texts using aspect words as cut points. These two texts interact with aspect words separately, enabling the model to fully capture aspect-aware text semantic information.

[0034] 2. This technical solution designs text-guided visual context attention and visual-guided text-context attention based on different modalities to model cross-modal semantic association alignment. This enhances the granular response of different modalities and learns fine-grained, consistent semantic information across multiple modalities.

[0035] 3. This technical solution not only explores the interaction between aspects and multimodality and the semantic dependencies within modalities from a global perspective, but also explores more fine-grained multimodal interaction alignment from a local perspective, enhancing feature expression and helping to improve the performance of the model in aspect-level multimodal sentiment analysis tasks.

[0036] This approach can better capture the dynamics within and between modalities to achieve better aspect-level multimodal sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the process of the embodiment. DETAILED DESCRIPTION

[0038] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the present invention is not limited thereto.

[0039] Example:

[0040] Reference Figure 1 ,The aspect-level multimodal sentiment analysis method based on collaborative attention fusion,includes the following steps:

[0041] 1) Obtain left text context representation, right text context representation, aspect aggregation representation, and image local representation:

[0042] Given a set of multimodal image-text datasets D, assume that each sample d∈D contains a text sentence T=(t1,t2,...,t n ) and an associated image O, there is also an aspect sequence Where A is a subsequence of T. Assume that all aspects A (i.e., words or phrases) in T are provided. The task input uses (T, O) pairs and their aspects A. The goal is to predict the sentiment polarity of the aspect word A. Then, use GLove+LSTM or BERT as the sentence encoder to extract the contextual hidden state representation. For the LSTM encoder, first, from the query matrix E∈R e×|v| Obtain the embedding sequence of sentence T in

[15] , where |v| is the size of the word embedding matrix and e represents the dimension of the word embedding. Then, input the embedding sequence in the sentence into the LSTM to generate a hidden state vector. For the BERT encoder, this example converts the given text and aspect words into "[CLS]+sentence+[SEP]" and "[CLS]+aspect word+[SEP]". Each word is represented by a vector with a dimension of 768. These vectors are then input into subsequent modules.

[0043] In order to better extract the high-level semantic features of image O, the convolutional layer of the pre-trained 152-layer ResNet is used as the image feature representation, whose size is represented as a tensor of s×m×m. Then, the image feature representation is divided into m×m visual area blocks of equal size, and the feature dimension size of each visual area block is s. Then the image local representation R = {r w |r w ∈R s ,w=1,2....,m×m}, where r w is the representation of each visual block;

[0044] In order to better interact aspect words with text, this example divides the input sentence T into three parts, namely and Represent the left text, right text and aspect body respectively, L, R, C are the input lengths of the left text, right text and aspect body respectively. Since the position of aspect words can intuitively reflect the importance of context words relative to the aspect, in order to identify the position of aspect words, two indicator tags are added before and after the aspect words (i.e. and ), for example, with "NBA" as the query, the text input is "Cavaliers reach NBAEastern Conference Finals with 100-99 victory, sweeping Hawks." This example uses a standard LSTM network to learn the hidden state vector of each aspect word

[0045]

[0046] in Θ represents all parameters in LSTM, and the hidden states of all aspects of words are obtained Afterwards, the average of these hidden states is used as the aspect aggregation representation

[0047] Two independent LSTM networks are used to obtain the left and right text context representations: and

[0048] 2) Generate aspect-guided global text feature representation and aspect-guided global image feature representation:

[0049] 2.1) Aspect-guided global text feature representation:

[0050] Based on the hidden state feature representation in step 1), the attention mechanism is further used to learn the semantics of the left and right context representations under the guidance of aspect aggregation representation. Generally, given different aspect words as queries, the importance of each context word should be different. Therefore, in the aspect aggregation representation H a As the input case, each left text context is represented by H a The bilinear interaction relationship calculates the attention weight of each hidden state in the left context:

[0051]

[0052]

[0053] in, and b l ∈R represents the trainable parameters, and the aspect aggregation representation guides the left text context representation H l as follows:

[0054]

[0055] Similarly, the right text context representation H guided by the aspect aggregation representation can be derived r ;

[0056] Although most methods use feature splicing to integrate aspect word information and text information, simple feature splicing will inevitably ignore the high-order interactions between them. Therefore, this example uses low-rank bilinear pooling to learn the interaction between aspect aggregation representation and the left and right text context guided by aspect aggregation representation. This method focuses on the pairwise interaction of features, which can achieve effective attention to aspects and left and right contexts. The aspect aggregation representation H a The left text context representation H guided by the aspect aggregation representation is respectively l , right text context representation H guided by aspect aggregation representation r The interaction is as follows:

[0057]

[0058]

[0059] Among them, W l1 , W l2 , W r1 , W r2 ∈R d×d , U l , U r ∈R d×d and b l , b r ∈R d represents the trainable parameters, σ is the nonlinear transformation function tanh, It is an element-by-element multiplication. In order to avoid the loss of text information, H lt 、H rt With H l 、H r Combined as aspect-guided global text feature representation:

[0060]

[0061] 2.2) Aspect-guided global image feature representation:

[0062] The text content in multimodal social media posts is short and sometimes even incomplete. Simply learning aspect-guided global text features may still not be enough to make correct sentiment predictions. Therefore, we further learn aspect-guided global image feature representation to assist aspect-guided global text feature representation and improve the robustness of the model. Usually, a given aspect is only related to certain areas of the image. Therefore, this example applies a visual attention mechanism to image features to enhance the visual feature response related to aspect words and suppress irrelevant interference;

[0063] In terms of aggregation representation H a As input, the attention weight of each visual region guided by the aspect aggregation representation is calculated as follows:

[0064]

[0065]

[0066] in, p∈R d and b v ∈R d are learnable parameters. Based on these visual attention weights, we can obtain aspect-guided global image feature representation:

[0067]

[0068] Among them, r gv The image feature representation is s-dimensional. In order to be consistent with the dimension of the text representation, the tanh function is used to transform r gv Transform into a d-dimensional vector:

[0069] G v =tanh(W v r gv +b v ),

[0070] Among them, W v ∈R d×s and b v ∈R d is a trainable parameter;

[0071] 3) Construct a cross-modal feature interaction mechanism to simultaneously generate local text feature representations integrated with vision and local visual feature representations integrated with text:

[0072] Reference example Figure 1 A cross-modal feature interaction mechanism is designed to simultaneously generate local text feature representations that are integrated with vision and local visual feature representations that are integrated with text. Different words in the text and different regions in the image have different degrees of contextual dependence in their respective modalities. To better model the contextual semantic dependencies of different modalities and emphasize the degree of contextual dependence of different local regions in each modality, this example uses a multi-head self-attention mechanism to learn the semantic dependencies within the text embedding sequence and the local image representation respectively. Traditional attention is defined as: Among them, Q represents query, K represents key, and V represents value. It is used to limit the value of the dot product. Multi-Head Attention (MHA) aims to perform multiple attention functions in parallel. It is an improvement on the traditional attention mechanism. The multi-head self-attention (MHSA) used can be regarded as a special MHA. In MHSA, the three inputs are the same, that is, Q = K = V. Therefore, MHSA = MultiHead(X, X, X), where X represents the input of the MHSA network. By using the MHSA mechanism, the text representation T can be converted into text sequence context features. Among them H t The calculation is as follows:

[0073] H t =MHSA(T)=MultiHead(T,T,T);

[0074] Similarly, the local image representation R is represented as the local context feature of the image after multi-head self-attention Among them H I The calculation is as follows:

[0075] H I =MHSA(R)=MultiHead(R,R,R);

[0076] 3.1) Local text feature representation integrated with vision:

[0077] In order to learn the most relevant visual region from each word, we first transform the local context feature H of the image after multi-head self-attention learning into I and text sequence context feature H t Interact and calculate the attention scores in turn. Inspired by the cross-modal association learning on VQA, two low-rank projection matrices are used to transform the two feature vectors (i.e., the vector of the j-th word in the text) and the vector of the i-th region in the image From H t and H I ) is projected into a c-dimensional common space, and then the features of the two are fused:

[0078]

[0079] Among them, W r ∈R d×c and W t ∈R e×c Represents the parameter matrix, and then the fused vector x ij Perform a linear transformation and obtain the normalized attention score through the softmax operation:

[0080]

[0081] Where W∈R c and b∈R 1 Represent weight parameters and bias parameters respectively, and the attention score can be used to focus on the word The attention strength of different visual areas, then, the word Guided Visual Context Features I j as follows:

[0082]

[0083] Compared to the visual area Text-guided visual context attention mechanism, Feature I j More reflective with words Since different modalities contribute to different degrees of emotion in related image areas, a gated fusion method is proposed to aggregate the representation of the word. and the visual context feature I guided by the word j :

[0084]

[0085]

[0086] Among them, W f and b f represents the weight matrix and bias parameters, z j Expressive words Visual context features guided by the word I j The degree of relevance is used to obtain the visual context features guided by all words. Finally, the mean of these features is used as the local text feature representation H integrated into the vision LT , as shown below:

[0087]

[0088] 3.2) Local visual feature representation integrated into text:

[0089] The attention mechanism that guides the visual context with different words in the text as the center is designed to learn the most relevant visual context information of each word in the text and perform gated association alignment. Similarly, a visually guided text context attention mechanism is designed with different visual blocks in the image as the center, and further learns the association alignment of different words centered on the image area. That is, the text context features guided by each visual block are learned, and the gating mechanism is used to fuse the visual block features with the text context features of different association levels guided by the visual block. The conversion process of the visually guided text context attention mechanism is similar to step 3.1), and the formula is as follows:

[0090]

[0091]

[0092]

[0093] Then, a gated fusion method is used to aggregate the representation of the visual block and the text context features guided by the visual block:

[0094]

[0095]

[0096] In this way, we can obtain the text context features guided by all visual blocks Finally, the mean of these features is used as the local visual feature representation H integrated into the text LV , as shown below:

[0097]

[0098] 4) Constructing a gated multimodal fusion mechanism:

[0099] Although the aspect-guided global text feature representation and the aspect-guided global image feature representation are learned in step 2), the global image feature may be less relevant or even irrelevant to the global text feature. In this case, directly fusing the aspect-guided image feature and the aspect-guided text feature often introduces some noise. Therefore, in order to better learn the semantic association between image and text and dynamically eliminate the noise brought by the fusion of image, this example proposes a gated multimodal fusion module, specifically based on the aspect-guided global text feature H GT The global image feature r is used as the main filter to fusion gv , as shown below:

[0100] c=σ(W H H GT +W R r gv +b z ),

[0101] Among them, W H ∈R d×4d , W R ∈R d×s , and b∈R d is a learnable parameter, σ is a sigmoid activation function, and according to the gated output c, the filtered aspect-guided global image feature H can be generated. visual , as shown below:

[0102]

[0103] The aspect-guided global text features and the filtered aspect-guided global image features are subjected to bilinear pooling operations to capture the interaction between multimodal global features and obtain the global multimodal fusion representation H GM , as shown below:

[0104]

[0105] Among them, W T ∈R 4d×d , W V ∈R d×d , P g ∈R d×d , b g ∈R d is a learnable parameter, σ is the nonlinear transformation function tanh;

[0106] In addition, since the output of step 3) is a fine-grained multimodal fusion representation, text and image are associated with the corresponding modality based on their own modality. In order to enrich the fine-grained semantic content, the local text feature representation H is further integrated into the visual LT and local visual feature representation H integrated into text LV Get the local multimodal fusion representation H LM , as shown below:

[0107] H LM =tanh(W c [H LT ;H LV ]+b c ),

[0108] Where W c ∈R 2d and b c ∈R d is a trainable parameter;

[0109] 5) Get the sentiment polarity prediction result:

[0110] In order to fuse multimodal features from a global-local multi-level perspective, we further fuse the aspect-guided global text feature representation H GT and aspect-guided global image feature representation H visual , global multimodal fusion representation H GM and local multimodal fusion representation H LM , generating a hierarchical fusion of multimodal sentiment semantic representation as follows:

[0111]

[0112] Subsequently, the multimodal representation H is input into the softmax function for the classification of sentiment polarity as shown below:

[0113] p(y|H)=softmax(W T H+b),

[0114] Where W∈R 7d×3 and b∈R 3 is a trainable parameter;

[0115] To optimize all parameters of the model, the objective function is to minimize the standard cross entropy function as follows:

[0116]

[0117] To better understand this example, two public multimodal datasets are selected in this embodiment. The sentiment polarity is divided into positive, negative, and neutral. The sentiment polarity distribution of these two datasets is shown in Table 1:

[0118] Table 1. Statistics of the multimodal Twitter dataset

[0119]

[0120] To verify the effectiveness of the method in this example, we selected the following baseline models for comparison. Among these models, only TomBERT uses the BERT pre-trained model to obtain text vector representations; the others all use the Glove pre-trained word embedding matrix. Accuracy (Acc) and Macro-F1 values ​​were used as evaluation metrics for comparative analysis. The results are shown in Table 2:

[0121] Table 2. Performance of GLFFCA and baseline models

[0122]

[0123]

[0124] The experimental results of this method and the comparative methods are shown in Table 2. As shown in Table 2, for models using the Glove pre-trained word embedding matrix to obtain vector representations, the proposed GLFFCA model achieves the best performance. It not only captures aspect-guided global multimodal information but also fully exploits local fine-grained multimodal alignment and interaction information. Furthermore, Table 2 shows that Res-Aspect's performance is quite limited, achieving an accuracy of approximately 60%. This indicates that textual content is crucial for aspect-level sentiment classification and should not be ignored. Furthermore, Table 2 shows that Res-RAM and Res-MGAN outperform RAM and MGAN, which use only text data, and also outperform the MemNet model using text data. This indicates that associated images can indeed provide complementary information to text. Furthermore, the difference between Res-RAM, Res-MGAN, Res-RAM-TFN, and Res-MGAN-TFN is the TFN fusion module. However, the performance of the TFN fusion method decreases, indicating that TFN-based fusion is not suitable for fine-grained aspect-level multimodal sentiment classification. MINI outperforms most baseline methods overall, demonstrating the effectiveness of integrating aspect-guided text and aspect-guided vision. However, because the MINI model is primarily based on a relatively weak MemNet model, its performance is still slightly worse than Res-ESTR. The ESAFN model outperforms other comparison methods but still lags behind the proposed GLFFCA model, further demonstrating that the proposed GLFFCA network is useful and reasonable for aspect-level multimodal sentiment classification.

[0125] When using the BERT pre-trained model to obtain vector representations, the GLFFCA+BERT model performs better than the TomBert model. The TomBert model uses a stacked BERT architecture to align aspects and images and capture intra-modal dynamics and inter-modal interactions. However, TomBert only considers multimodal interaction alignment locally, while GLFFCA+BERT not only considers multimodal content interactions at a local, fine-grained level but also considers multimodal interactions globally. This demonstrates the advantage of GLFFCA+BERT in learning multimodal interactions from both a global and local perspective. Furthermore, as shown in Table 2, GLFFCA+BERT pre-trained with BERT outperforms GLFFCA based on GLove, further demonstrating the advantages of obtaining word representations based on large-scale pre-trained models. Overall, the proposed method is useful and reasonable for aspect-level multimodal sentiment classification.

[0126] To evaluate the effectiveness of the different modules in the proposed method, this example conducts ablation studies on the model from the perspectives of multimodal global correlation, multimodal local correlation, and feature fusion. Ablation experiments were conducted on the Twitter-2015 and Twitter-2017 datasets, maintaining all training parameters identical. Accuracy and Macro-F1 were used as evaluation metrics. The ablation results are shown in Table 3.

[0127] The comparison method is as follows:

[0128] ①GFF: The cross-modal feature interaction mechanism in GLFFCA is deleted, and only the features extracted by aspect-guided global text features and aspect-guided global image features are retained for sentiment analysis.

[0129] ②GLFF-tgvca: In the cross-modal feature interaction mechanism in GLFFCA, the local text feature representation integrated with vision is retained, while the local visual feature representation integrated with text is discarded.

[0130] ③GLFF-vgtca: In the cross-modal feature interaction mechanism in GLFFCA, the local visual feature representation integrated with text is retained, while the local text feature representation integrated with vision is discarded.

[0131] ④GLFFCA-no-sg: In the cross-modal feature interaction mechanism in GLFFCA, the gate mechanism is not used to fuse fine-grained multimodal features, but they are directly fused on the feature sequence.

[0132] ⑤GLFFCA-no-fg: In the gated multimodal fusion mechanism of GLFFCA, gated fusion is not performed and the multimodal information at each level is directly spliced ​​for sentiment classification.

[0133] Table 3. Ablation experiments on Twitter-2015 and Twitter-2017

[0134]

[0135] Table 3 shows the results of ablation experiments on the proposed model. First, to demonstrate the effectiveness of the cross-modal feature interaction mechanism, this example removes the cross-modal feature interaction mechanism from GLFFCA. As shown in Table 3, GFF achieves accuracies of 72.03% and 66.29% on the two Twitter datasets, respectively, while the proposed GLFFCA achieves accuracies of 74.07% and 68.14%. This demonstrates that adding local semantic alignment features to global feature learning can assist in global feature learning, thereby improving the accuracy of multimodal sentiment analysis. Furthermore, to demonstrate the effectiveness of different attention mechanisms in the cross-modal feature interaction mechanism, this example analyzes the contribution of these attention mechanisms one by one. Specifically, when only text-guided visual context attention is used in the cross-modal feature interaction mechanism, the recognition accuracy of GLFF-tgvca on the two Twitter datasets decreases by 1.17% and 1.28%, respectively. When only visually guided text-context attention is used, the classification accuracy of GLFF-vgtca on the two Twitter datasets decreases by 0.3% and 1.28%, respectively. This demonstrates that both the text-guided visual context attention mechanism and the vision-guided text-context attention mechanism are reasonable and effective in cross-modal alignment, and that the multi-interaction alignment of text and image can mine richer cross-modal local details. More importantly, the text-guided visual context attention mechanism is more effective than the vision-guided text-context attention mechanism, indicating that in social media datasets, text contains richer semantic information than images, and that image data contains more noise than text data. Furthermore, within the cross-modal feature interaction mechanism, this example also incorporates a gating mechanism for local feature fusion after cross-modal alignment. Table 3 shows that the performance of GLFFCA-no-sg decreases by 2.04% and 1.86% in classification accuracy compared to GLFFCA on the two Twitter datasets, respectively. This demonstrates the effectiveness of the gating mechanism in cross-modal alignment fusion, as data from different modalities typically contribute differently to sentiment semantics. Finally, this example also designs a gated multimodal fusion mechanism. As shown in Table 3, the classification performance of GLFFCA-no-fg, which does not use a gate mechanism, is 1.53% lower than that of GLFFCA on the Twitter-2017 dataset, and 0.78% lower than that of GLFFCA on the Twitter-2015 dataset. This shows that in some cases, some features in the image may not be very relevant to the text context, and direct feature fusion may introduce noise.In summary, the proposed GLFFCA method for aspect-level multimodal sentiment analysis, GLFFCA, simultaneously mines global associations between text and corresponding images based on aspect entities. It also employs a cross-modal feature interaction mechanism to mine local semantic associations between text and corresponding images. This approach compensates for the lack of fine-grained information in global text and image features, and achieves finer-grained, multi-interaction alignment between text and image. Furthermore, based on the characteristics of multimodal features at different levels, a gated fusion method is designed to achieve deep fusion of these features. Finally, comparative experiments on the Twitter-2015 and Twitter-2017 datasets demonstrate the effectiveness of this method, improving the performance of aspect-level multimodal sentiment analysis.

Claims

1. A collaborative attention fusion-based aspect-level multimodal sentiment analysis method, characterized by: The steps include: 1) Obtaining the left text context representation, right text context representation, aspect aggregation representation, and local image representation: Use the pre-trained BERT model or Glove word embedding to obtain a text embedding sequence, then use a long short-term memory network to learn the contextual dependencies of the text embedding sequence to obtain the left text context representation, right text context representation, and aspect representation. The aspect representations are then average-pooled to obtain the aspect aggregation representation. For images, a ResNet network is used to obtain the image feature representation and convert it into a local image representation. Obtaining the left text context representation, right text context representation, aspect aggregation representation, and local image representation includes the following steps: 1.1) First, a pre-trained BERT model or Glove word embedding is used to obtain a text embedding sequence. Then, a long short-term memory network is used to learn the contextual dependencies of the text embedding sequence. This is used to obtain the left text context representation, the right text context representation, and the aspect representation. The aspect representations are then average-pooled to obtain the aspect aggregate representation. 1.2) For images, a ResNet model is used to extract image features to obtain image feature representations. The image is then divided into different regions to obtain local image representations, which facilitates fine-grained guidance of local image features. 2) Generate aspect-guided global text feature representation and aspect-guided global image feature representation: Use the attention mechanism guided by the aspect aggregate representation to guide the left and right text context representations obtained in step 1), respectively. Then, use low-rank linear pooling to interact the aspect aggregate representation with the left and right text context representations guided by the aspect aggregate representation, and concatenate the results of the interaction to obtain the aspect-guided global text feature representation. Use the attention mechanism guided by the aspect aggregate representation to guide the local image representation obtained in step 1) to obtain the aspect-guided global image feature representation. Generating aspect-guided global text feature representation and aspect-guided global image feature representation includes the following steps: 2.1) For generating aspect-guided global text feature representation: ① Using the bilinear interaction between the left text context representation and the aspect aggregation representation obtained in step 1), calculate the attention weight of each hidden state in the left text context, and then perform the weighted summation to obtain the left text context representation guided by the aspect aggregation representation. The same method is used to obtain the right text context representation guided by the aspect aggregation representation. ② Low-rank bilinear pooling is used to learn the interaction between aspect aggregation representation and left and right context representation guided by aspect aggregation representation, and the obtained features are concatenated to obtain aspect-guided global text feature representation. Low-rank bilinear pooling can focus on the pairwise interaction of features and can effectively focus on aspects and left and right context; 2.2) For the aspect-guided global image feature representation, the bilinear interaction between the local image representation obtained in step 1) and the aspect aggregation representation is used to calculate the attention weight of each image region, and then the weighted sum is used to obtain the aspect-guided global image feature representation; 3) Construct a cross-modal feature interaction mechanism to simultaneously generate local text feature representations that are integrated with vision and local visual feature representations that are integrated with text: The text embedding sequence and image local representations obtained in step 1) are learned separately through multi-head self-attention. Then, text sequence context features and image local context features are obtained. The text-guided visual context features are obtained using a text-guided visual context attention mechanism. These are then fused with the text sequence context features through a gating mechanism to generate local text feature representations that are integrated with vision. At the same time, visual-guided text context features are obtained using a visual-guided text context attention mechanism. These are then fused with the image local context features through a gating mechanism to generate local visual feature representations that are integrated with text. 4) Constructing a gated multimodal fusion mechanism: Using a gating mechanism, the aspect-guided global text feature representation obtained in step 2) is filtered and fused with the aspect-guided global image feature representation obtained in step 2) to obtain a global multimodal fusion representation. Simultaneously, the vision-integrated local text feature representation and the text-integrated local visual feature representation obtained in step 3) are fused to obtain a local multimodal fusion representation. 5) Obtain sentiment polarity prediction results: The global text feature representation and global image feature representation obtained in step 2), the global multimodal fusion representation obtained in step 4), and the local multimodal fusion representation are concatenated and input into the softmax function to obtain the polarity of sentiment classification.

2. The aspect-level multimodal sentiment analysis method based on collaborative attention fusion according to claim 1 is characterized in that The construction of a cross-modal feature interaction mechanism in step 3) to simultaneously generate a local text feature representation integrated with vision and a local visual feature representation integrated with text includes the following steps: 3.1) Use multi-head self-attention to learn the text embedding sequence and image local representation obtained in step 1) respectively; 3.2) Design text-guided visual context attention and visual-guided text-context attention to model cross-modal semantic correlation alignment respectively: ① Integrate local text feature representation into vision; First, the vector of the jth word in the text sequence context feature after multi-head self-attention learning and the i-th region vector in the image local context feature after multi-head self-attention learning are projected into the c-dimensional common space, and then the features of the two are fused. Subsequently, the fused vector is linearly transformed, and the attention scores of the jth word in the text sequence context feature and other image local context features are calculated in turn. The normalized attention scores are obtained by softmax operation, and then the attention scores are weighted to each image local feature area and summed to obtain the visual context feature guided by the jth word. Since different modalities contribute differently to emotions, a gated fusion method is used to aggregate the word and the visual context features guided by the word to obtain the visual context features guided by all words. These features are averagely pooled to obtain the local text feature representation integrated into the vision. ② Local visual feature representation integrated into text: First, the i-th region vector in the image local context feature after multi-head self-attention learning and the j-th word vector in the text sequence context feature after multi-head self-attention learning are projected into the c-dimensional common space, and then the features of the two are fused. Subsequently, the fused vector is linearly transformed, and the attention scores of the i-th visual block and other text sequences in the image local context feature are calculated in turn. The normalized attention scores are obtained by softmax operation, and then the attention scores are weighted to each text sequence and summed to obtain the text context feature guided by the i-th visual block. Since different modalities contribute differently to emotions, a gated fusion method is used to aggregate the visual block and the text context feature guided by the visual block to obtain the text context features guided by all visual blocks. These features are averaged and pooled to obtain the local visual feature representation integrated into the text.

3. The aspect-level multimodal sentiment analysis method based on collaborative attention fusion according to claim 1 is characterized in that The construction of the gated multimodal fusion mechanism in step 4) includes the following steps: 4.1) Using a gating mechanism, the aspect-guided global text feature representation obtained in step 2) is filtered and fused with the aspect-guided global image feature obtained in step 2); 4.2) Bilinear pooling is used to capture the interaction between the global text feature representation and the filtered aspect-guided global image features to obtain a global multimodal fusion representation; 4.3) At the same time, the local text feature representation integrated with vision and the local visual feature representation integrated with text obtained in step 3) are nonlinearly fused to obtain a local multimodal fusion representation.

4. The aspect-level multimodal sentiment analysis method based on collaborative attention fusion according to claim 1 is characterized in that The sentiment polarity prediction result of the aspect word is obtained in the step 5): the global text feature representation and the global image feature representation obtained in step 2) and the global multimodal fusion representation and the local multimodal fusion representation obtained in step 4) are hierarchically spliced ​​and input into the softmax function to obtain the sentiment polarity prediction result corresponding to the aspect word.

Images