Aspect-level multi-modal sentiment analysis method based on dual-channel and attention mechanism

Abstract

The application claims a kind of dual-channel and attention mechanism aspect-level multi-modal sentiment analysis method, which is based on neural network, the sentiment information contained in image feature is extracted by aspect word feature and sentence feature joint attention mechanism multi-scale, and GCN network is introduced into aspect-level multi-modal sentiment analysis task, greatly improve the feature extraction and interactive fusion ability of model.In the present application, the pre-training encoder is used to extract aspect word, sentence feature and image feature in the feature extraction layer.In the attention mechanism layer, after the bidirectional fusion of aspect word and sentence feature, the final aspect word feature and sentence feature representation are obtained.Image feature establishes image feature extraction network through channel attention mechanism and spatial attention mechanism.Finally, the interactive fusion features of each modality are dynamically extracted by GCN module.In the experiment, the performance indicators of aspect-level multi-modal sentiment analysis based on attention mechanism are improved on the data set.

Description

Technical Field

[0001] This invention belongs to the field of computer language processing and sentiment analysis, specifically involving an aspect-level multimodal sentiment analysis method based on a dual-channel and attention mechanism. Background Technology

[0002] In recent years, user-generated content on various online platforms has grown rapidly. Therefore, analyzing word content to identify public opinion on target aspects or entities has become increasingly important. How to utilize artificial intelligence and related technologies to uncover specific sentiment tendencies within these words has gradually become a research hotspot in recent years.

[0003] Emotions express people's attitudes towards objective things, and are usually conveyed through various means such as body language, facial expressions, and written language. Sentiment Analysis (SA), also known as Opinion Mining (OM), aims to extract opinions from a large number of unstructured sentences and classify them into positive, neutral, or negative sentiment polarities. In the Internet age, with the development of social platforms such as Weibo, Zhihu, and WeChat, text and images have gradually become the main carriers for users to convey opinions and emotions about target aspects or entities in the online world. Aspect-based sentiment analysis has received widespread attention from academia and industry in the past decade.

[0004] Early methods typically employed machine learning techniques such as sentiment dictionaries, dependency relations, and statistical methods to generate sentence features. However, traditional methods required significant human effort in feature selection and extraction, lacked connections between aspect words and sentence context, and exhibited poor transferability and robustness. With the success of deep learning methods in various natural language processing tasks, the application of neural networks in aspect-level sentiment analysis was also promoted. By leveraging various neural network models in deep learning to learn and extract feature associations between aspect words and sentence context, model performance has gradually improved. Numerous deep network models, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Graph Neural Networks (GNNs), and attention mechanisms, have been proposed, further advancing sentence-based aspect-level sentiment analysis.

[0005] As content on many online platforms becomes increasingly multimodal, the prediction of sentiment polarity from information in other modalities has gradually attracted researchers' attention. Furthermore, the research achievements of deep learning in image processing have provided a theoretical foundation for aspect-level multimodal sentiment analysis. Xu et al. first introduced image modal information into aspect-level sentiment analysis, using CNN to extract image features and Long Short-Term Memory (LSTM) networks to extract sentence features. Through interactive attention mechanisms, they experimentally verified the feasibility of their proposed method. Subsequently, Gu et al. used a Bidirectional Gate Recurrent Unit (BiGRU) network and a multi-head self-attention mechanism to encode sentence semantic information, and employed a ResNet-152 model and capsule network to extract image features. In multimodal interaction fusion, they used a multi-head attention network to maximize the contribution of each modality to sentiment transmission, thus improving network performance. Yu et al. proposed a hierarchical interaction module to model the pairwise interactions between given aspect words, sentence information, and image information. To bridge the semantic gap between sentence features and image features, an auxiliary reconstruction module based on the autoencoder concept is proposed, improving model performance. However, existing models still have some shortcomings: 1) During image feature extraction, they cannot fully extract channel and spatial information from the image, resulting in the ineffective combination of sentiment information and aspect word information. 2) They cannot effectively perform intermodal information fusion, leading to unsatisfactory model performance. Therefore, this paper studies aspect-level multimodal sentiment analysis tasks and proposes a more efficient model.

[0006] CN114936623A discloses an aspect-level sentiment analysis method that integrates multimodal data. First, data preprocessing is performed, adjusting text and image formats to suit the input requirements of neural networks. Second, feature extraction is conducted, using Bi-LSTM to extract text features after word embedding and a ResNet50 network to extract image features. Next, multimodal aspect extraction and alignment are performed, using sequence labeling to extract aspect terms from the text and a memory network with attention and point-wise convolution operations for implicit alignment of image regions and aspect terms. Then, based on positional attention, text features are used, Gaussian modeling is applied to explicit contextual positions, and a memory network extracts aspect-sensitive text representations. Following this, multimodal data fusion is performed, fusing discriminant matrices to integrate the multimodal data. Finally, sentiment classification is performed, utilizing the fused feature information for sentiment classification. This invention uses multimodal data for aspect-level sentiment analysis, extracting complementary multimodal information and improving the accuracy of sentiment analysis tasks.

[0007] The method described above, which uses average aspect word vectors, is prone to semantic confusion and hinders the interaction between aspect words and sentence and image features. Secondly, in extracting image features, this method ignores the auxiliary role of semantic information from the sentence context. Therefore, the ability of this method to fuse multimodal data is limited. Summary of the Invention

[0008] This invention aims to solve the problems of the prior art. It proposes an aspect-level multimodal sentiment analysis method based on a dual-channel and attention mechanism. The technical solution of this invention is as follows:

[0009] An aspect-level multimodal sentiment analysis method based on dual-channel and attention mechanisms includes the following steps:

[0010] Step 1: Extract hidden feature representations from sentence features and aspect word features in the dataset using a BERT pre-trained encoder; extract image features using a ResNet-152 pre-trained network; aspect words are subsequences belonging to sentences.

[0011] Step 2: Calculate the feature relevance of sentence features and aspect word features through a multi-head attention mechanism, so that highly similar features receive corresponding attention weights; finally, obtain aspect word features guided by sentences and sentence features guided by aspect words;

[0012] Step 3: Utilize sentence-guided aspect features to weight image features, and obtain image channel features through a channel attention mechanism;

[0013] Step 4: Using sentence features guided by aspect words and weighted image channel features, a spatial attention map is generated using the spatial relationships of features in the spatial attention mechanism to obtain the final feature representation of the image;

[0014] Step 5: Calculate the dynamic adjacency matrix using sentence features guided by aspect words and the final image features generated through channel attention and spatial attention; use the aggregation and message passing capabilities of graph neural networks to obtain the final fused feature representation;

[0015] Step 6: The final fused features, aspect word features obtained in Step 2 through the multi-head attention mechanism, and sentence features are pooled and classified by the classification module.

[0016] Furthermore, in step 1, the sentence features and aspect word features in the dataset are extracted using a BERT pre-trained encoder to extract hidden feature representations, and the image features are extracted using a ResNet-152 pre-trained network. Specifically:

[0017] Two BERT-based pre-trained sentence feature encoders output sentence and aspect word feature information; a pre-trained ResNet network is used to extract image features; pre-trained models provide better initialization parameters for the model, and fine-tuning on the target task improves the model's generalization performance and accelerates convergence; the BERT pre-trained model obtains sentence features. and aspect features ;in Represents sentence length. Representative aspect: word length Represents the output feature dimension; image features are represented as ,in Representing the ResNet-152 model, Channels representing image features , These represent the width and height of the image feature, respectively; where Representing original aspects: words, sentences, and images; This represents aspect words, sentence and image features extracted by a pre-trained network.

[0018] Furthermore, step 2 employs a multi-head attention mechanism to fuse relevant information between aspect word features and sentence features. The specific method is as follows:

[0019] To obtain the interaction features between sentence features and aspect word features, a multi-head attention mechanism is adopted to calculate the similarity between the two and effectively achieve feature fusion. The expression is as follows:

[0020] ;

[0021] ;

[0022] ;

[0023] Represents a multi-head attention mechanism. Represents input features, Scaling factor This represents the output of the i-th layer in the Transformer. Representative level normalization, For activation function, , These represent the trainable parameter matrices, respectively;

[0024] The aspect term features and sentence features are used as query matrices respectively. Calculate aspect word features guided by sentence features. and sentence features guided by aspect words .

[0025] Furthermore, step 3 utilizes sentence-guided aspect word features to weight image features, and obtains image channel features through a channel attention mechanism. The specific method is as follows:

[0026] Each channel of the image features is treated as a feature detector. In the channel attention mechanism, important features related to aspect words in each channel of the image are extracted. To introduce aspect word features into the image, aspect word features and image features are first fused through a multi-head self-attention mechanism. The specific formula is as follows:

[0027]

[0028]

[0029] In the channel attention mechanism, its input... It utilizes a multi-head attention mechanism, guided by aspect-word-guided image features. The representative multilayer perceptron, Represents average pooling. Represents max pooling. represent Activation function The output represents the channel attention.

[0030] Furthermore, in step 4: using the sentence features guided by aspect words to weight the image channel features, in the spatial attention mechanism, a spatial attention map is generated using the spatial relationships of the features to obtain the final feature representation of the image. The specific steps are as follows:

[0031] By employing a multi-head attention mechanism, the image features output by the channel attention mechanism are weighted using sentence features guided by aspect words. This process highlights important regions in the image features that are related to the sentiment of aspect words within the channel attention mechanism. The specific formula is as follows:

[0032]

[0033]

[0034] Formula (7) represents the implementation details of the channel attention mechanism, where Represents matrix concatenation. Represents convolution operation. Represents the ReLU activation function. This represents image features guided by sentence features through a multi-head attention mechanism. The output represents spatial attention.

[0035] Furthermore, step 5 calculates a dynamic adjacency matrix using sentence features guided by aspect words and the final image features generated through channel attention and spatial attention; the aggregation and message passing capabilities of graph neural networks are then used to obtain the final fused feature representation, specifically including:

[0036] Sentence features are concatenated with the final image features, and an attention matrix is ​​obtained through a self-attention mechanism. This matrix is ​​then used as the adjacency matrix of the GCN. In the GCN, for a given node in the graph... ,in It represents all the nodes in the graph, corresponding to the concatenated matrix of sentence features and final image features. As an adjacency matrix among all nodes The weights depend on the similarity between nodes;

[0037]

[0038]

[0039]

[0040] in, The output representing splicing spatial attention Sentence features introduced by phenomenological words , For nodes The Feature representation of the layer's output, For GCN The trainable weight matrix of the layer, The ReLU activation function is used; since GCN performs feature extraction and encoding between associated nodes, therefore... Output of all nodes in the layer Represented as:

[0041]

[0042] n represents the number of nodes.

[0043] Furthermore, in step 6: the final fused features, the aspect word features obtained in step 2 through the multi-head attention mechanism, and the sentence features are pooled and classified by the classification module. The specific steps are as follows:

[0044] For aspect word features and sentence features, since [CLS] was initially added as a label when extracting features using the pre-trained model, the final hidden state of this label is used as the set representation of aspect word and sentence features, denoted as . and For the first representation in the output of the fusion with GCN features, since it is a weighted sum of the features, this feature is used as the classification feature. Therefore, the total output features after pooling and concatenation are... It can be represented as:

[0045]

[0046] During the classification stage:

[0047]

[0048]

[0049] in These are trainable weights, and the loss value is calculated using the cross-entropy loss function. , , These represent the number of training samples and the true labels of the samples, respectively.

[0050] The advantages and beneficial effects of this invention are as follows:

[0051] The main advantage of this invention lies in the following: In step 3, the channels in the image can be considered as feature extractors, and channel attention is used to extract important features related to aspect words in the image channels. To integrate aspect word information into the image channel features, a multi-head attention mechanism is used to interact the features of both before channel attention, facilitating the guiding role of aspect words in channel attention. Then, in step 4, spatial attention mainly extracts regional features related to aspect words in the image. Since sentence features also have regional associations related to aspect words, sentence features are introduced into the spatial attention mechanism to guide the extraction of regional features related to aspect words in the image. These two steps progressively extract deep features from the image, enhancing the contribution of image features to sentiment classification in subsequent multimodal fusion. Attached Figure Description

[0052] Figure 1 This is a flowchart of an aspect-level multimodal sentiment analysis method based on a dual-channel and attention mechanism, according to a preferred embodiment of the present invention.

[0053] Figure 2 This is a framework diagram for a multimodal sentiment analysis model at the aspect level. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0055] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0056] Figure 1 This is a flowchart illustrating the overall process of an aspect-level multimodal sentiment analysis method based on a dual-channel and attention mechanism, as described in this invention. The following is a summary of the process. Figure 1 Further description. The main steps included in this invention are:

[0057] Step 1: Extract hidden feature representations from sentence features and aspect word features in the dataset using a BERT pre-trained encoder. Image features are extracted using a ResNet-152 pre-trained network.

[0058] Step 2: Calculate the feature relevance between sentence features and aspect word features using a multi-head attention mechanism, assigning appropriate attention weights to highly similar features. This ultimately yields aspect word features guided by the sentence and sentence features guided by the aspect words.

[0059] Step 3: Use sentence-guided aspect features to weight the original image features, and obtain the image channel features through a channel attention mechanism.

[0060] Step 4: Use the sentence features guided by aspect words to weight the image channel features, and use the spatial relationships between features to generate a spatial attention map to obtain the final feature representation of the image.

[0061] Step 5: Calculate the dynamic adjacency matrix using sentence features guided by aspect words and image features generated through channel attention and spatial attention. Utilize the aggregation and message-passing capabilities of graph neural networks to obtain the final fused feature representation.

[0062] Step 6: Use a pooling mechanism to classify the final fused feature representation through the classification module.

[0063] 2. Figure 2 This is a framework diagram of a multimodal sentiment analysis model at this level. The following section combines... Figure 2 The structural principle of the present invention will be further explained below. As shown in the figure, the method model of the present invention has four layers, and the specific contents of each layer are as follows:

[0064] (1) Modal feature extraction layer

[0065] Given a set of multimodal samples, which contains a sentence with n words. and related images and a Aspect of word sequence For aspect terms It is also associated with an emotional tag. Related. This invention will link sentences. and aspect words Features were extracted from the two BERT encoders respectively, and the images were processed. Feature extraction is performed using the ResNe-152 network. For the input of the BERT sentence encoder, the [CLS] tag is added to the beginning of the sentence, and the [SEP] tag is added to the end. Finally, the sentence features are obtained using a BERT pre-trained model. and aspect features ;in Represents sentence length. Representative aspect: word length Represents the feature dimension. For image features, it is represented as... ,in, The number of channels representing image features. and These represent the width and height of the image feature, respectively.

[0066] (2) Attention mechanism layer

[0067] To further extract the correlations between features and the modal interactions between sentence words and image features, this model employs a multi-head attention mechanism to extract potential correlation features between aspect word features and sentence features. In image feature extraction, a channel attention mechanism fusing aspect word features and a spatial attention mechanism fusing sentence features are used. This integrates image modal features with sentence modal features while extracting aspect word-related features at different scales within the image modality, enabling the GCN network to identify adjacency relationships between important nodes. Through the message passing and aggregation mechanisms of the GCN network, multimodal features are extracted at a deeper level.

[0068] 1) Interaction between sentence and aspect word features

[0069] To obtain the interaction features between sentences and aspect words, strengthen the correlation information between different features, and filter out redundant information, this invention employs a multi-head attention mechanism to calculate the similarity between the features of the two and effectively achieve feature fusion. The expression for the multi-head attention mechanism is:

[0070]

[0071] Where T represents the matrix transpose. This represents a multi-head attention mechanism, consisting of three parts: Query (Q), Key (K), and Value (V). The attention value generated through the interaction between Q and K is mapped to V via a dot product. (Scale factor) It is the feature dimension of each attention head.

[0072] Aspect features and sentence features are fused together using a multi-head attention mechanism to obtain the final output features. The aspect features, guided by the sentence features, are then processed through the multi-head attention mechanism, followed by linear transformation and residual connections to obtain the final output features. The specific formula is as follows:

[0073]

[0074]

[0075] LayerNorm represents layer normalization, which ensures the stability of data feature distribution and accelerates model training; Glue represents the activation function. , This represents the trainable weight parameters. In the multi-head attention mechanism, aspect word features and sentence features are respectively used as query matrices. It is possible to calculate aspect word features guided by sentence features. And sentence features guided by aspect words are called .

[0076] 2) Channel attention mechanism

[0077] Since each channel of an image feature can be considered a feature detector, in the channel attention mechanism, the features after average pooling and max pooling, through a feedforward neural network layer, can extract important features related to aspect words in each channel of the image. To identify the sentiment distribution related to aspect words in image channels, this invention introduces aspect word features into the channel attention mechanism. First, aspect word features and image features are fused using a multi-head attention mechanism, and the fused features are used as input to the channel attention mechanism. The specific formula is as follows:

[0078]

[0079]

[0080] In the channel attention mechanism, its input... It uses a multi-head attention mechanism to guide image features through aspect words. The complete output then undergoes linear transformation and residual connection, as shown in formulas (2) and (3). This part uses... This is for the purpose of simplifying the description. Equation (5) represents the implementation details of the channel attention mechanism, where The representative multilayer perceptron contains trainable weight parameters in a neural network. Represents average pooling. This represents max pooling. This represents the ReLU activation function.

[0081] 3) Spatial attention mechanism

[0082] In spatial attention mechanisms, the spatial relationships of features can be utilized to learn the distribution regions of important sentiment features related to aspect words. During the sentence and aspect word feature interaction stage, the sentence features guided by aspect words already contain the location regions of important sentiment features related to aspect words within the sentence. To enhance the extraction capability of image feature regions, this invention fuses the sentence features guided by aspect words and the image features output by the channel attention mechanism, and then learns the distribution regions of important sentiment features within the spatial attention mechanism. The specific formula is as follows:

[0083]

[0084]

[0085] Formula (7) represents the implementation details of the spatial attention mechanism, where Represents matrix concatenation. Represents convolution operation. This represents the ReLU activation function.

[0086] (3) GCN feature fusion layer

[0087] Sentence features are concatenated with image features, and an attention matrix is ​​obtained through a self-attention mechanism, which is then used as the adjacency matrix of the GCN. Firstly, this attention matrix can learn the correlation features between the final sentence and image features, making the adjacency matrix more flexible. Secondly, it can adaptively adjust the importance of similar features between sentence and image features. In GCN, for a given node's graph... ,in It represents all the nodes in the graph, corresponding to the concatenated matrix of sentence features and final image features. As an adjacency matrix among all nodes The weights depend on the similarity between nodes.

[0088]

[0089]

[0090]

[0091] in, For nodes The Feature representation of the layer's output, For GCN The trainable weight matrix of the layer, This is the ReLU activation function. Since GCN performs feature extraction and encoding between associated nodes, therefore... Output of all nodes in the layer Represented as:

[0092]

[0093] (4) Output layer

[0094] For aspect word features and sentence features, since [CLS] was added as a label when the pre-trained model was initially used to extract features, the final hidden state of this label is used as the set representation of aspect word and sentence features, denoted as . and The first representation in the output of the fusion with GCN features is used as the classification feature because it is a weighted sum of the features. Therefore, the output feature can be expressed as:

[0095]

[0096] The output features of GCN are passed through a feedforward neural network to complete the classification task. The specific formula is as follows:

[0097]

[0098]

[0099] in These are trainable weights, and the loss value is calculated using the cross-entropy loss function. .

[0100] Experimental simulation

[0101]

[0102] The model proposed in this chapter was tested on the TWITTER-2015 and TWITTER-2017 datasets. As shown in Table 1, compared to other models, the proposed model achieved the best experimental results in both classification accuracy and macro-average F1 score. Compared to the second-best TomBERT network model, it improved classification accuracy and macro-average F1 score by 1.35% and 1.25% respectively on TWITTER-2015, and by 1.38% and 1.53% respectively on TWITTER-2017. This is because this chapter extracts deep semantic associations related to aspect terms in images by fusing aspect term features and sentence features, and further improves the model's classification performance through the feature fusion capability of graph neural networks. The TomBERT model uses BERT to extract aspect term-sensitive visual representations, but it fails to employ an effective feature fusion method. While the MIMN model, which uses a multi-hop memory network to extract features, achieves multi-hop fusion of bimodal features, it does not deeply extract the interaction features between sentence content and visual information.

[0103] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.

[0104] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0105] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A method for aspect-level multimodal sentiment analysis based on a dual-channel and attention mechanism, characterized in that, Includes the following steps: Step 1: Extract hidden feature representations from the sentence features and aspect word features in the dataset using a BERT pre-trained encoder. Image features were extracted using a ResNet-152 pre-trained network; aspect words are subsequences belonging to sentences; Step 2: Calculate the feature relevance of sentence features and aspect word features through a multi-head attention mechanism, so that highly similar features receive corresponding attention weights; finally, obtain aspect word features guided by sentences and sentence features guided by aspect words; Step 3: Utilize sentence-guided aspect features to weight image features, and obtain image channel features through a channel attention mechanism; Step 4: Using sentence features guided by aspect words and weighted image channel features, a spatial attention map is generated using the spatial relationships of features in the spatial attention mechanism to obtain the final feature representation of the image; Step 5: Calculate the dynamic adjacency matrix using sentence features guided by aspect words and the final image features generated through channel attention and spatial attention; use the aggregation and message passing capabilities of graph neural networks to obtain the final fused feature representation; Step 6: The final fused features, aspect word features obtained in Step 2 through the multi-head attention mechanism, and sentence features are pooled and classified by the classification module.

2. The aspect-level multimodal sentiment analysis method based on dual-channel and attention mechanisms according to claim 1, characterized in that, Step 1 involves using a BERT pre-trained encoder to extract hidden feature representations from the sentence features and aspect word features in the dataset, and using a ResNet-152 pre-trained network to extract image features. Specifically: Two BERT-based pre-trained sentence feature encoders output sentence and aspect word feature information; a pre-trained ResNet network is used to extract image features; pre-trained models provide better initialization parameters for the model, and fine-tuning on the target task improves the model's generalization performance and accelerates convergence; the BERT pre-trained model obtains sentence features. and aspect features ;in Represents sentence length. Representative aspect: word length Represents the output feature dimension; Image features are represented as ,in Representing the ResNet-152 model, Channels representing image features , These represent the width and height of the image feature, respectively; where Representing original aspects: words, sentences, and images; This represents aspect words, sentence and image features extracted by a pre-trained network.

3. The aspect-level multimodal sentiment analysis method based on dual-channel and attention mechanisms according to claim 2, characterized in that, Step 2 employs a multi-head attention mechanism to fuse relevant information between aspect word features and sentence features. The specific method is as follows: To obtain the interaction features between sentence features and aspect word features, a multi-head attention mechanism is adopted to calculate the similarity between the two and effectively achieve feature fusion. The expression is as follows: ； ； ； Represents a multi-head attention mechanism. Represents input features, Scaling factor This represents the output of the i-th layer in the Transformer. Representative level normalization, For activation function, , These represent the trainable parameter matrices, respectively; The aspect term features and sentence features are used as query matrices respectively. Calculate aspect word features guided by sentence features. and sentence features guided by aspect words .

4. The aspect-level multimodal sentiment analysis method based on dual-channel and attention mechanisms according to claim 3, characterized in that, Step 3 utilizes sentence-guided aspect features to weight image features, and obtains image channel features through a channel attention mechanism. The specific method is as follows: Each channel of the image features is treated as a feature detector. In the channel attention mechanism, important features related to aspect words in each channel of the image are extracted. To introduce aspect word features into the image, aspect word features and image features are first fused through a multi-head self-attention mechanism. The specific formula is as follows: ； ； In the channel attention mechanism, its input... It utilizes a multi-head attention mechanism, guided by aspect-word-guided image features. The representative multilayer perceptron, Represents average pooling. Represents max pooling. represent Activation function The output represents the channel attention.

5. The aspect-level multimodal sentiment analysis method based on dual-channel and attention mechanisms according to claim 4, characterized in that, Step 4: Using sentence features guided by aspect words to weight image channel features, a spatial attention map is generated using the spatial relationships of features in the spatial attention mechanism to obtain the final image feature representation. The specific steps are as follows: By employing a multi-head attention mechanism, the image features output by the channel attention mechanism are weighted using sentence features guided by aspect words. This process highlights important regions in the image features that are related to the sentiment of aspect words within the channel attention mechanism. The specific formula is as follows: Formula (7) represents the implementation details of the channel attention mechanism, where Represents matrix concatenation. Represents convolution operation. Represents the ReLU activation function. This represents image features guided by sentence features through a multi-head attention mechanism. The output represents spatial attention.

6. The aspect-level multimodal sentiment analysis method based on dual-channel and attention mechanisms according to claim 5, characterized in that, Step 5 calculates a dynamic adjacency matrix using sentence features guided by aspect words and the final image features generated through channel attention and spatial attention; the aggregation and message passing capabilities of graph neural networks are then used to obtain the final fused feature representation, specifically including: Sentence features are concatenated with the final image features, and an attention matrix is ​​obtained through a self-attention mechanism. This matrix is ​​then used as the adjacency matrix of the GCN. In the GCN, for a given node in the graph... ,in It represents all the nodes in the graph, corresponding to the concatenated matrix of sentence features and final image features. As an adjacency matrix among all nodes The weights depend on the similarity between nodes; ； ； ； in, The output representing splicing spatial attention Sentence features introduced by phenomenological words , For nodes The Feature representation of the layer's output, For GCN The trainable weight matrix of the layer, The ReLU activation function is used; since GCN performs feature extraction and encoding between associated nodes, therefore... Output of all nodes in the layer Represented as: ； n represents the number of nodes.

7. The aspect-level multimodal sentiment analysis method based on dual-channel and attention mechanisms according to claim 6, characterized in that, Step 6: The final fused features, the aspect word features obtained in Step 2 through the multi-head attention mechanism, and the sentence features are pooled and classified by the classification module. The specific steps are as follows: For aspect word features and sentence features, since [CLS] was initially added as a label when extracting features using the pre-trained model, the final hidden state of this label is used as the set representation of aspect word and sentence features, denoted as . and For the first representation in the output of the fusion with GCN features, since it is a weighted sum of the features, this feature is used as the classification feature. Therefore, the total output features after pooling and concatenation are... It can be represented as: ； During the classification stage: ； ； in These are trainable weights, and the loss value is calculated using the cross-entropy loss function. , , These represent the number of training samples and the true labels of the samples, respectively.

Images