Multimodal aspect-level sentiment analysis method based on text and image gating fusion mechanism

Abstract

The application belongs to the technical field of multi-modal aspect-level sentiment analysis, and discloses a multi-modal aspect-level sentiment analysis method based on a text and image gating fusion mechanism. The method introduces the weighting of adjectives in adjective-noun pairs (ANPs) extracted from images of a data set as image auxiliary information while interacting with the text and the image, and realizes multi-modal feature fusion by constructing a gating mechanism that dynamically controls the input of the image and the image auxiliary information in the last feature fusion stage. Experimental results show that the method achieves competitive results on two Twitter data sets based on multi-modal aspect-level sentiment analysis, thereby verifying the effectiveness of the method.

Description

Technical Field

[0001] This invention relates to the field of multimodal aspect-level sentiment analysis technology, and in particular to a multimodal aspect-level sentiment analysis method based on a text and image gating fusion mechanism. Background Technology

[0002] With the rapid development of internet technology, people have gradually begun to participate in various online activities and share their views and opinions online. Currently, social media and numerous service platforms are widely used in people's daily lives. Simultaneously, users are increasingly inclined to express their emotions by posting multimodal information combining text and images for different topics and events. This trend has attracted widespread attention from the academic community to multimodal sentiment analysis research. In recent years, multimodal sentiment analysis has become a research hotspot in the field of affective computing.

[0003] Multimodal Aspect-Based Sentiment Analysis (MABSA) is an important fine-grained task in multimodal sentiment analysis research. It selects certain nouns in the text as aspect entities and combines the text and image content to further infer the sentiment polarity of the aspect entities. Figure 1 A representative example is presented, where the aspect entity is "Klay Thompson" in the text. We can predict the negative sentiment polarity of the aspect entity by using textual information supplemented by image information. Compared to multimodal global sentiment analysis, multimodal aspect-level sentiment analysis is a more refined and challenging sentiment analysis task, which can capture the sentiment polarity of internal entities in the text that global sentiment analysis cannot obtain.

[0004] Given the importance and broad application prospects of this field, researchers have proposed numerous multimodal aspect-level sentiment analysis models. For example, MIMN and ESAFN achieve intermodal interaction by using attention mechanisms to aspect entities, text, and images; TomBERT and Saliencybert achieve effective encoding and better representation learning by using pre-trained language and visual models to encode text word features and image visual features. These research results demonstrate that fusing images into traditional text sentiment analysis can enable models to achieve more accurate sentiment prediction capabilities. However, most of these models only use simple concatenation methods to construct the final feature representation and then achieve sentiment analysis in the multimodal feature fusion stage, without considering that information in the image that is irrelevant to the text semantics may introduce additional noise into the model. Summary of the Invention

[0005] This invention addresses the problem that most existing sentiment analysis models rely solely on simple concatenation methods to construct the final feature representation for sentiment analysis during the multimodal feature fusion stage, neglecting the potential introduction of irrelevant information from images into the model, which could introduce additional noise. To address this, we propose a Text-Image Gated Fusion Mechanism for Aspect-Based Multimodal Sentiment Analysis (TIGFM) model. For a given sample in the dataset, the model first uses pre-trained text and image encoders to obtain text and image feature representations, respectively. Then, a cross-modal attention mechanism is used to achieve interaction between text and image, resulting in an aspect entity-aware image representation. Furthermore, we introduce weighted adjectives from adjective-noun pairs (ANPs) extracted from the dataset images as auxiliary image information to enhance the image's semantic expressiveness. Finally, we design a multimodal feature fusion gating mechanism to dynamically control the contribution of image information to the final feature representation, thereby predicting the sentiment polarity of the sample.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A multimodal aspect-level sentiment analysis method based on text and image gating fusion mechanism includes:

[0008] Step 1: Extract features from unimodal text and image by using two types of pre-trained models, namely language and image, to obtain aspect entity representation and image feature representation;

[0009] Step 2: Based on the obtained aspect entity representation and image feature representation, obtain the aspect entity perception image representation by introducing a multimodal interaction layer;

[0010] Step 3: Introduce the adjective-noun pairs extracted from the image using DeepSentiBank to generate a semantic representation of the image from another level, and obtain image auxiliary information representation of aspect entities;

[0011] Step 4: The obtained aspect entity representation, the image representation of aspect entity perception, and the image auxiliary information representation of aspect entity are fused to obtain the final representation of aspect entity, thereby realizing sentiment prediction.

[0012] Furthermore, in step 1, for text encoding, a pre-trained language model RoBERTa is used as the text encoder to obtain context-aware aspect entity representations.

[0013] Furthermore, in step 1, for image encoding, a residual network is used as an image encoder to obtain the visual representation of the image, and the visual representation of the image is converted into the same dimension as the text representation through linear transformation to obtain the feature representation of the image.

[0014] Further, step 2 includes:

[0015] First, cross-modal feature interaction is performed on aspect entities and images, using aspect entity representations as query vectors and image representations as key and value vectors;

[0016] The outputs of the multi-head cross-modal attention mechanism from m heads are concatenated and linearly transformed to obtain the final output of the multi-head cross-modal attention mechanism.

[0017] Finally, the output of the multi-head cross-modal attention mechanism is normalized through two layers and a feedforward network to obtain an image representation of aspect entity perception.

[0018] Further, step 3 includes:

[0019] Select the k adjective-noun pairs with the highest confidence scores from the adjective-noun pairs extracted from the images in DeepSentiBank;

[0020] First, the adjectives and nouns in the selected k adjective-noun pairs are concatenated and then fed into a text encoder to obtain the adjective representation H. A And the noun H N Then, cosine similarity is used to calculate the aspect entity representation H. T and nouns representing H N The degree of semantic similarity between them is used to align aspect entities with nouns in adjective-noun pairs:

[0021]

[0022] Where α is H T and H N Similarity score;

[0023] For H A Each adjective in the text is assigned a similarity weight to its corresponding noun and aspect entity representation as image auxiliary information, and then the image auxiliary information representation of the aspect entity is obtained through a text encoder:

[0024] H T→A =α·H A .

[0025] Further, step 4 includes:

[0026] First, the aspect entity is represented as H. TImage representation of entity perception H T→v The image data is stitched together, and a gating mechanism that dynamically controls the input of image information is constructed through linear transformation and nonlinear activation functions.

[0027] g T→v =σ(W T→v [H T H T→v ])

[0028] Among them W T→v ∈R d*2d σ is a learnable linear transformation parameter, and σ is a nonlinear activation function using dot product, which is used to control the gated output within the range of [0, 1].

[0029] The above construction method is used to obtain a gating mechanism for dynamically controlling the input of image auxiliary information:

[0030] g T→A =σ(W r→A [H T H T→A ])

[0031] Among them W T→A ∈R d*2d H is a learnable linear transformation parameter. T→A It is an image-assisted information representation of an entity;

[0032] Next, the gated values ​​of the image and image auxiliary information are multiplied element-wise with the image representation of aspect entity perception and the image auxiliary information representation of aspect entity, respectively, to dynamically control the input of the image and image auxiliary information at the word level in the text. Then, the above representations are concatenated and fed into a Transformer-based multimodal self-attention layer to achieve feature fusion between modalities.

[0033] H = Transformer(H) T g T→v H T→v g T→A H T→A )

[0034] Finally, the first token of H is represented by the fused multimodal features. 0 The data is fed into a softmax layer to obtain the final sentiment label.

[0035] P(y|H)=softmax(WH 0 )

[0036] Where W is a learnable linear transformation parameter.

[0037] Furthermore, the cross-entropy loss constructed from the predicted values ​​and true values ​​of the sentiment labels of aspect entities is used as the training loss function for the model sentiment analysis task.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] (1) A multimodal aspect-level sentiment analysis method based on text and image gating fusion mechanism is proposed. This method introduces the weighted adjectives in the adjective-noun pairs extracted from the image as image auxiliary information while interacting with text and image, thereby obtaining better semantic representation and sentiment expression of the image;

[0040] (2) After obtaining the text, image and image auxiliary information representations, the present invention dynamically controls the input of the image and image auxiliary information representations by constructing a multimodal feature fusion gating mechanism in the final feature fusion stage, so as to prevent information that is not related to the text semantics from introducing additional noise into the model;

[0041] (3) Compared with most existing multimodal aspect-level sentiment analysis models, the method proposed in this invention achieves the best performance on two Twitter-based datasets. Attached Figure Description

[0042] Figure 1 This is a representative example of multimodal aspect-level sentiment analysis;

[0043] Figure 2 This is an embodiment of the overall architecture of a multimodal aspect-level sentiment analysis method based on a text and image gating fusion mechanism according to an embodiment of the present invention;

[0044] Figure 3 This is the internal structure of the multimodal interaction layer in an embodiment of the present invention;

[0045] Figure 4 This is the internal structure of the feature auxiliary module in an embodiment of the present invention;

[0046] Figure 5 This invention illustrates the impact of different values ​​of k on model performance in an embodiment of the invention; where (a) corresponds to the value of k on the Twitter-2015 dataset, and (b) corresponds to the value of k on the Twitter-2017 dataset.

[0047] Figure 6 This is an example analysis of the CapRoBERTa and TIGFM models in the embodiments of the present invention. Detailed Implementation

[0048] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0049] A multimodal aspect-level sentiment analysis method based on text and image gating fusion mechanism includes:

[0050] Step 1: Extract features from unimodal text and image by using two types of pre-trained models, namely language and image, to obtain aspect entity representation and image feature representation;

[0051] Step 2: Based on the obtained aspect entity representation and image feature representation, obtain the aspect entity perception image representation by introducing a multimodal interaction layer;

[0052] Step 3: Introduce the adjective-noun pairs extracted from the image using DeepSentiBank to generate a semantic representation of the image from another level, and obtain image auxiliary information representation of aspect entities;

[0053] Step 4: The obtained aspect entity representation, the image representation of aspect entity perception, and the image auxiliary information representation of aspect entity are fused to obtain the final representation of aspect entity, thereby realizing sentiment prediction.

[0054] Furthermore, the overall architecture of the TIGFM model is as follows: Figure 2 As shown, the model consists of four modules: (1) feature extraction module; (2) feature interaction module; (3) feature assistance module; and (4) feature fusion module. In this section, we first define the research task of multimodal aspect-level sentiment analysis, and then provide a more detailed introduction to each module in the TIGFM model.

[0055] Task Definition: Given a set of multimodal samples D = (x1, x2, ..., x...), ... d Each sample x is used as input to the model. i ∈D contains a text S = (w1, w2, ..., w...) consisting of m words. m ), a corresponding image I, and an aspect entity T = (w1, w2, ..., wn) with n words. n ), where T is a subsequence of words in S. The research task of this invention is to predict the sentiment label y∈Y for each aspect entity in its samples, where Y includes three categories: positive, negative, and neutral.

[0056] 1 Feature Extraction Module

[0057] This module uses two types of pre-trained models—language and image—to extract features from unimodal text and image respectively.

[0058] 1.1 Text Feature Extraction

[0059] Given an input text, we divide it into two parts: an aspect entity T and its corresponding context C. The context is obtained by padding the positions of the aspect entities in the text with the special character "$T$". For text encoding, this invention uses the pre-trained language model RoBERTa as the text encoder, which, as an extension of the BERT model, has shown good performance in various natural language processing tasks. Specifically, we follow the implementation mechanism of RoBERTa encoding. First, we use the concatenation of C and T to form the context-aware aspect entity input T′, and then add two special markers (i.e., add at the beginning). <s>, at the end of the addition< / s> After being fed into RoBERTa encoding, the context-aware aspect entity representation is obtained:

[0060] H T =RoBERTa(T′)

[0061] Where H T ∈R d*n d is the hidden dimension of each word vector, and n is the number of words contained in T′.

[0062] 1.2 Image Feature Extraction

[0063] For image encoding, this invention employs a Residual Network (ResNet) as the image encoder. Compared to the previous VGG network, ResNet addresses the vanishing gradient problem that arises with increasing layer count by using residual connections, enabling the extraction of deeper semantic information in image recognition tasks. Specifically, we first resize the given input image I to 224*224 pixels and denote it as I′. Then, we use the output of the last convolutional layer in the pre-trained 152-layer ResNet as the visual representation of the image.

[0064] H I =ResNet(I′)

[0065] Where H I ∈R 2048*49 49 represents the number of 7×7 visual blocks of equal size into which I′ is divided, and 2048 represents the vector dimension of each visual block. To ensure that the image representation and text representation reside in the same semantic space for subsequent cross-modal interaction, we transform the image representation to the same dimension as the text representation using a linear transformation:

[0066]

[0067] in These are learnable linear transformation parameters.

[0068] 2 Feature Interaction Module

[0069] After obtaining the feature representations of the aspect entities and images, in order to further learn the features of aspect entities in the image, this module introduces a multimodal interaction layer to obtain the image representation of aspect entity perception.

[0070] The internal structure of the multimodal interaction layer is as follows: Figure 3 As shown, its core technology is the Multi-Head Cross-Modal Attention Mechanism (MC-ATT). Specifically, we first perform cross-modal feature interaction between aspect entities and images, representing aspect entities as H... T As a query vector, the image representation H v The key and value vectors are used as the core and value vectors. The detailed calculation process of the i-th head in the cross-modal attention mechanism is as follows:

[0071]

[0072] Where m is the number of heads in MC-ATT, {W Qi W Ki W vi}∈R d / m×d These are the learnable linear transformation parameters for the Query, Key, and Value of the i-th header, respectively. Then, the MC-ATT outputs of the m headers are concatenated and subjected to a linear transformation to obtain the final MC-ATT output:

[0073] MC-ATT(H T H v ) = W m [MC-ATT1(H T H v ), ..., MC-ATT m (H T H v )] T

[0074] Among them W m These are the learnable linear transformation parameters of MC-ATT. Finally, the MC-ATT output is normalized through two layers (LN) and a feedforward network (FFN) to obtain an aspect-aware entity-perceptive image representation:

[0075] Z T→V =LN(H T +MC-ATT(H T H v ))

[0076] H T→V =LN(z T→v +FFN(Z T→V))

[0077] Where H T→V ∈Rd *t It is the final generated image representation of the aspect entity perception.

[0078] 3 Feature Auxiliary Modules

[0079] To enable a more intuitive semantic expression of emotional information in images, this module introduces DeepSentiBank (https: / / github.com / stephen-pilli / DeepSentiBank) to generate semantic representations of images from another level using the adjective-noun pairs (ANPs) extracted from the images.

[0080] Unlike the image representations described above, ANPs extract information such as nouns representing people or objects in an image, along with adjectives that modify them, enabling the model to understand image content at the textual semantic level. Specifically, DeepSentiBank generates a set of 2089 ANPs for each image in the samples, and we select the top k ANPs with the highest confidence scores for our experiments.

[0081] However, ANPs are essentially content extracted from images in a coarse-grained manner, which may contain image regions unrelated to aspect entities or semantic information that leads to image misidentification. Directly using these ANPs can introduce additional noise into the model due to their inaccuracy. First, the adjectives and nouns from the selected k ANPs are concatenated and fed into a text encoder to obtain the adjective representation H. A And the noun H N Then, cosine similarity is used to calculate the aspect entity representation H. T and nouns representing H N The semantic similarity between them is used to align aspect entities with nouns in ANPs:

[0082]

[0083] Where α is H T and H N The similarity score. Since adjectives in ANPs can help reflect the emotion conveyed by the image content to some extent, they are used as auxiliary information to express the emotion of aspect entities in the image from a textual semantic level. Figure 4 As shown, we have H A Each adjective in the text is assigned a similarity weight to its corresponding noun and aspect entity representation as image auxiliary information, and then the image auxiliary information representation of the aspect entity is obtained through a text encoder:

[0084] H T→A =α·H A

[0085] 4 Feature Fusion Module

[0086] This module represents the aspect entity H obtained above. T Image representation of entity perception H T→V And the image auxiliary information representation H of aspect entities T→A Sentiment prediction is achieved by fusing the aspects of entities in a specified manner to obtain the final representation.

[0087] This invention uses images as an alternative modal of information to aid in the semantic representation of text. While images can provide the model with some content not contained in the text, they may also contain regions irrelevant to the text's semantics, introducing additional noise. Furthermore, image-based auxiliary information may contain errors such as adjective recognition. Therefore, in the multimodal feature fusion stage, we construct a gating mechanism for information beyond the text to dynamically control the input of these feature representations. Specifically, we first... T and H T→V The image data is stitched together, and a gating mechanism that dynamically controls the input of image information is constructed through linear transformation and nonlinear activation functions.

[0088] g T→V =σ(W T→V [H T H T→V ])

[0089] Among them W T→V ∈R d*2d σ is a learnable linear transformation parameter, and σ is a nonlinear activation function using dot product, which is used to control the gated output within the range [0, 1]. Similarly, the above construction method can also be used to obtain a gating mechanism for dynamically controlling the input of image auxiliary information:

[0090] g T→A =σ(W T→A [H T H T→A ])

[0091] Among them W T→A ∈R d*2d These are learnable linear transformation parameters. Next, the gated values ​​of the image and image auxiliary information are multiplied element-wise with the image representation of the aspect entity perception and the image auxiliary information representation of the aspect entity, respectively, to dynamically control the input of the image and image auxiliary information at the word-level intensity in the text. Then, the above representations are concatenated and fed into a Transformer-based multimodal self-attention layer to achieve intermodal feature fusion.

[0092] H = Transformer(H) T g T→V H T→V g T→A H T→A )

[0093] Finally, the first token of H is represented by the fused multimodal features. 0 The data is fed into a softmax layer to obtain the final sentiment label.

[0094] P(y|H)=softmax(WH 0 )

[0095] Where W is a learnable linear transformation parameter.

[0096] To optimize the parameters in the model, this invention uses the cross-entropy loss constructed from the predicted and true values ​​of the sentiment labels of aspect entities as the training loss function for the model's sentiment analysis task:

[0097]

[0098] 5 Experiments

[0099] To demonstrate the effectiveness of the TIGFM model, we conducted a series of experiments on two Twitter datasets based on multimodal aspect-level sentiment analysis and selected some representative methods to compare their performance with the TIGFM model.

[0100] 5.1 Experimental Setup

[0101] This invention uses two datasets, Twitter-2015 and Twitter-2017, proposed by Yu et al., based on multimodal aspect-level sentiment analysis. These two datasets are sampled from tweets containing text and images published on the Twitter platform in 2014-2015 and 2016-2017. The relevant statistical information is shown in Table 1.

[0102] Table 1. Relevant statistics of the two Twitter datasets

[0103]

[0104] In the experiments, a pre-trained RoBERTa-base model was used as the text encoder, and ResNet-152 as the image encoder. Furthermore, AdamW was used as the learner to optimize the parameters during the alternating optimization process. For the hyperparameters of the TIGFM model, the batch size was set to 16, the training epochs to 9, the k-value to 5, and the learning rate to 1e-5. The average result of three independent training iterations of the TIGFM model was used as the final experimental result. All experiments were implemented using PyTorch and an NVIDIA Tesla V100 GPU.

[0105] 5.2 Comparison with baseline

[0106] This invention selects the following unimodal and multimodal models as baselines to compare their performance with the TIGFM model: (1) Res-Target: a method to directly obtain image visual features through the ResNet model; (2) AE-LSTM: an LSTM model that obtains aspect entity-related context through attention; (3) MGAN: a multi-granularity attention network that fuses aspect entities and context at different granularities; (4) BERT: a pre-trained language model based on Transformer for interaction between aspect entities and text; (5) RoBERTa: a model that uses a better training strategy and a larger corpus to train BE. RT further improves the pre-trained language model; (6) MIMN: a multi-hop memory network for interaction between aspect entities, text and image information; (7) ESAFN: an entity-aware attention fusion network for capturing the relationship between aspect entities, text and image information; (8) TomBERT: a multimodal aspect-level sentiment analysis method based on BERT; (9) CapBERT: a multimodal aspect-level sentiment analysis method that converts images into text captions and encodes them in combination with the input text; (10) CapRoBERTa: an extended baseline that uses RoBERTa to replace BERT in CapBERT.

[0107] Table 2. Experimental results of the TIGFM model and the baseline on the Twitter-2015 and Twitter-2017 datasets.

[0108]

[0109] 5.3 Experimental Results and Analysis

[0110] Table 2 shows the experimental results of the TIGFM model and the baseline on the Twitter-2015 and Twitter-2017 datasets. We use accuracy (Acc) and macro-average (Macro-F1) as evaluation metrics for model performance.

[0111] Based on the experimental results in Table 2, we can draw the following conclusions: (1) The performance of the Res-Target model is lower than that of all other models, indicating that images mostly play an auxiliary role to the text and cannot be used as an independent modality to dominate the sentiment prediction of the model; (2) The multimodal model that integrates image information performs better overall than the single-modal model, indicating that image information can supplement the text and enhance the model's sentiment prediction ability; (3) The TomBERT and CapBERT models perform much better than other multimodal models, indicating that using BERT as a text encoder can obtain more robust feature representations; (4) The CapBERT model achieves the best performance among all original baselines. (5) The CapRoBERTa model outperformed the CapBERT model in the above evaluation metrics, which also proves that RoBERTa is more powerful than BERT. (6) The TIGFM model achieved the best results on both datasets. Compared with the best performing model in the baseline, it was about 0.7% and 0.5% higher in Acc and Macro-F1 on the Twitter-2015 dataset, respectively, and about 1.1% and 2.0% higher in Acc and Macro-F1 on the Twitter-2017 dataset, respectively.

[0112] 5.4 Ablation Experiment

[0113] To further investigate the impact of each unit in the TIGFM model on its overall performance, this section conducts ablation analysis on three important units in the model on the Twitter-2015 and Twitter-2017 datasets: (1) image auxiliary information; (2) image gating mechanism; and (3) image auxiliary information gating mechanism. The experimental results are shown in Table 3. We first removed the above three units respectively, and then removed all three units to more comprehensively verify the contribution of our designed units to the improvement of model performance.

[0114] Table 3 Ablation Experiment Results of TIGFM Model

[0115]

[0116] First, removing image auxiliary information reduces Acc by approximately 1.3% and 0.5% on the two datasets, respectively, indicating that using the weighted adjectives in ANPs as image auxiliary information can more intuitively express the emotional information in the image semantically. Then, removing the image and the image auxiliary information gating mechanism reduces Acc by approximately 0.4% and 0.2%, and 0.5% and 0.4% on the two datasets, respectively. This shows that while images and adjectives in ANPs can provide content beyond the text to some extent, directly using them can introduce additional noise to the model due to the presence of information irrelevant to the text semantics. Finally, removing all the above units reduces Acc by approximately 1.5% and 0.6% on the two datasets, respectively, which verifies from another perspective that the units we designed all contribute to the improvement of model performance to a certain extent.

[0117] 5.5 Parameter Settings

[0118] All the experiments described above were based on the optimized model hyperparameters. This section provides a detailed analysis of the process of evaluating the optimal hyperparameters.

[0119] 5.5.1 Batch size values

[0120] Table 4 shows the impact of different batch size values ​​on model performance on the two datasets. We used 8, 16 and 32 as batch size values ​​for analysis.

[0121] Table 4. Impact of batch size on model performance

[0122]

[0123] The results in Table 4 show that the model achieved optimal performance on both datasets when the batch size was 16. We speculate the reasons are as follows: a batch size of 8 is too small for the number of samples in both datasets, leading to longer training times and difficulty converging, resulting in underfitting. Increasing the batch size within a certain range is beneficial for model convergence stability; however, a batch size of 32 may cause the model to get stuck in local minima, leading to decreased generalization performance. Therefore, the final batch size was set to 16.

[0124] 5.5.2k values

[0125] To explore the optimal value of the number of ANPs, k, we used four values, 1, 3, 5, and 7, respectively, for analysis. Figure 5(a) and (b) respectively show the impact of the value of k on the model performance on the two datasets.

[0126] against Figure 5 The results show that model performance increases with the number of ANPs. When k is 5, the model achieves the best performance on both datasets. However, when k is greater than 5, the model performance no longer improves but decreases. We speculate the reason is as follows: each text in the dataset may involve no more than 5 aspect entities. When k is greater than the number of aspect entities in the text, image auxiliary information may introduce additional noise to the model. Therefore, we ultimately set the value of k to 5.

[0127] 5.6 Sample Analysis

[0128] To better demonstrate the performance advantages of our proposed model on the two datasets, this section uses two representative examples to compare and analyze the TIGFM and CapRoBERTa models. Example information and prediction results are as follows: Figure 6 As shown.

[0129] Figure 6 (a) In the example, the TIGFM model combines the smiling facial expression of the aspect entity “Madonna” in the image with adjectives with positive connotations such as “bright” in ANPs to predict the positive emotional polarity of the aspect entity. Figure 6 (b) In the example, since the aspect entity "One Love Manchester" is represented by the entire scene in the image, and considering the vibrant stage and positive adjectives like "energetic" in the ANPs, the TIGFM model can also predict the positive emotional polarity of the aspect entity. However, the CapRoBERTa model made incorrect predictions in both examples, as determined through research. Figure 6 The captions in examples (a) and (b) “A group of people in traditional dressclothes.” and “A woman in a blue dress is talking on a cell phone.” do not contain words with emotional connotations. Furthermore, the CapRoBERTa model relies solely on these captions to obtain the semantics of the images, discarding the original images themselves. Therefore, it cannot accurately reflect the emotions expressed in the images to a certain extent.

[0130] In summary, this invention proposes a multimodal aspect-level sentiment analysis method based on a text and image gating fusion mechanism. This method introduces weighted adjectives from adjective-noun pairs (ANPs) extracted from images in the dataset as image auxiliary information, enabling better sentiment semantic expression of the image content in the samples. Furthermore, in the final feature fusion stage, a multimodal feature fusion gating mechanism that dynamically controls the input of image information is constructed to prevent information irrelevant to the text semantics from introducing additional noise into the model. Experimental results show that our proposed method outperforms all baseline models on the Twitter-2015 and Twitter-2017 datasets, thus validating the superiority of the TIGFM model and the effectiveness of our proposed method.

[0131] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal aspect-level sentiment analysis method based on text and image gating fusion mechanism, characterized in that, include: Step 1: Extract features from unimodal text and image by using two types of pre-trained models, namely language and image, to obtain aspect entity representation and image feature representation; Step 2: Based on the obtained aspect entity representation and image feature representation, obtain the aspect entity perception image representation by introducing a multimodal interaction layer; Step 3: Introduce the adjective-noun pairs extracted from the image using DeepSentiBank to generate a semantic representation of the image from another level, and obtain image auxiliary information representation of aspect entities; Step 4: The obtained aspect entity representation, the image representation of aspect entity perception, and the image auxiliary information representation of aspect entity are fused to obtain the final representation of aspect entity, thereby realizing sentiment prediction; Step 4 includes: First, the aspect entity is represented as H. T Image representation of entity perception H T→V The image data is stitched together, and a gating mechanism that dynamically controls the input of image information is constructed through linear transformation and nonlinear activation functions. g T→V =σ(W T→V [H T ;H T→V ]) Among them W T→V ∈R d*2d σ is a learnable linear transformation parameter, and σ is a nonlinear activation function using dot product, which is used to control the gated output within the range of [0,1]. The above construction method is used to obtain a gating mechanism for dynamically controlling the input of image auxiliary information: g T→A =σ(W T→A [H T ;H T→A ]) Among them W T→A ∈R d*2d H is a learnable linear transformation parameter. T→A It is an image-assisted information representation of an entity; Next, the gated values ​​of the image and image auxiliary information are multiplied element-wise with the image representation of aspect entity perception and the image auxiliary information representation of aspect entity, respectively, to dynamically control the input of the image and image auxiliary information at the word level in the text. Then, the above representations are concatenated and fed into a Transformer-based multimodal self-attention layer to achieve feature fusion between modalities. H=Transformer(H T ;g T→V H T→V ;g T→A H T→A ) Finally, the first token of H is represented by the fused multimodal features. 0 The data is fed into a softmax layer to obtain the final sentiment label. P(y|H)=softmax(WH 0 ) Where W is a learnable linear transformation parameter.

2. The multimodal aspect-level sentiment analysis method based on text and image gating fusion mechanism according to claim 1, characterized in that, In step 1, for text encoding, a pre-trained language model RoBERTa is used as the text encoder to obtain context-aware aspect entity representations.

3. The multimodal aspect-level sentiment analysis method based on text and image gating fusion mechanism according to claim 1, characterized in that, In step 1, for image encoding, a residual network is used as an image encoder to obtain the visual representation of the image, and the visual representation of the image is converted into the same dimension as the text representation through linear transformation to obtain the feature representation of the image.

4. The multimodal aspect-level sentiment analysis method based on text and image gating fusion mechanism according to claim 1, characterized in that, Step 2 includes: First, cross-modal feature interaction is performed on aspect entities and images, using aspect entity representations as query vectors and image representations as key and value vectors; The outputs of the multi-head cross-modal attention mechanism from m heads are concatenated and linearly transformed to obtain the final output of the multi-head cross-modal attention mechanism. Finally, the output of the multi-head cross-modal attention mechanism is normalized through two layers and a feedforward network to obtain an image representation of aspect entity perception.

5. The multimodal aspect-level sentiment analysis method based on text and image gating fusion mechanism according to claim 1, characterized in that, Step 3 includes: Select the k adjective-noun pairs with the highest confidence scores from the adjective-noun pairs extracted from the images in DeepSentiBank; First, the adjectives and nouns in the selected k adjective-noun pairs are concatenated and then fed into a text encoder to obtain the adjective representation H. A And the noun H N Then, cosine similarity is used to calculate the aspect entity representation H. T and nouns representing H N The degree of semantic similarity between them is used to align aspect entities with nouns in adjective-noun pairs: Where α is H T and H N Similarity score; For H A Each adjective in the text is assigned a similarity weight to its corresponding noun and aspect entity representation as image auxiliary information, and then the image auxiliary information representation of the aspect entity is obtained through a text encoder: H T→A =α·H A 。 6. The multimodal aspect-level sentiment analysis method based on text and image gating fusion mechanism according to claim 1, characterized in that, The cross-entropy loss, constructed by comparing the predicted and true values ​​of the sentiment labels of aspect entities, is used as the training loss function for the model's sentiment analysis task.

Images