A joint multimodal aspect-level sentiment analysis method based on adaptive attention fusion

Abstract

The present invention discloses a joint multimodal aspect-level sentiment analysis method based on adaptive attention fusion. Due to the inherent semantic gap between text and visual modalities, transferring text-based generative pre-training models to image-text multimodal sentiment analysis tasks faces challenges. To this end, the present invention embeds the text after word segmentation in image-text pair data to obtain word embedding vectors; uses a target detection model to extract visual features from each input image of the image-text pair data; uses an adaptive image-to-text fusion module to fuse visual features into text representation; uses a selective fusion module to filter relevant regional features based on text representation; uses a cross-modal Mixup module to achieve interaction between text and image; uses a visually enhanced BART module to enhance the encoding of multimodal information in a sequence-to-sequence model; and adaptively bridges the semantic gap between text and image representation based on a selective fusion mechanism of image and text.

Description

Technical Field

[0001] The present invention discloses a joint multimodal aspect-level sentiment analysis method based on adaptive attention fusion, which relates to the technical field of natural language processing. Background Art

[0002] Sentiment analysis is a crucial and classic problem in natural language processing. With the continuous development of the internet, platforms like Weibo and Twitter have accumulated a vast amount of multimodal data with images. Traditional text-only sentiment analysis is no longer sufficient, prompting the emergence of multimodal sentiment analysis. Joint multimodal aspect-level sentiment analysis aims to simultaneously extract the implicit aspect terms and the corresponding sentiment polarity from a given tweet with an image.

[0003] Multimodal aspect word extraction and multimodal aspect-level sentiment analysis are two important subtasks in multimodal sentiment analysis. Pre-trained generative models (such as BART and T5) have attracted increasing attention in the field of aspect-based sentiment analysis (ABSA). However, the semantic gap between text and image modalities makes it difficult to transfer text-based generative pre-trained models to image-text multimodal sentiment analysis tasks.

[0004] Recent research has demonstrated the effectiveness of pre-trained generative models such as BART and T5 for aspect-based sentiment analysis (ABSA). However, transferring text-based generative pre-trained models to multimodal sentiment analysis tasks involving text and images is challenging due to the inherent semantic gap between text and visual modalities. To address this issue, we propose a joint multimodal aspect-level sentiment analysis method based on adaptive attention fusion for joint multimodal aspect-based sentiment analysis (JMABSA). Summary of the Invention

[0005] The technical problem solved by the present invention is: the present invention provides a joint multimodal aspect-level sentiment analysis method based on adaptive attention fusion, which is based on the selective fusion mechanism of image and text, and aims to adaptively bridge the semantic gap between text and image representation, and adaptively transfer the text-based pre-trained model to the joint multimodal aspect-level sentiment analysis task.

[0006] The technical solution of the present invention is: a joint multimodal aspect-level sentiment analysis method based on adaptive attention fusion, comprising the following steps:

[0007] S1. Use the embedding layer of the pre-trained language model to embed the text after word segmentation in the image text pair data to obtain the word embedding vector;

[0008] S2. Use the object detection model to extract visual features from each input image of the image-text pair data;

[0009] S3, an adaptive image-to-text fusion module, fuses visual features into text representation and improves feature fusion effects through multimodal interaction;

[0010] S4, using the selective fusion module to filter relevant regional features based on text representation to improve sentiment analysis performance;

[0011] S5. Use the cross-modal Mixup module to achieve interaction between text and images and enhance the semantic connection between features;

[0012] S6. Utilize the visually enhanced BART module to enhance the encoding of multimodal information in sequence-to-sequence models.

[0013] Furthermore, the specific implementation of S1 includes:

[0014] Use the sequence-to-sequence model BART to obtain word embeddings, using <s>and< / s> To indicate the beginning and end of a sentence, the original text feature of the sample is represented as E, E∈R T×d , where T represents the text length and d represents the dimension of BART.

[0015] Furthermore, the specific implementation of S2 includes:

[0016] All object proposals are extracted from a given image V by the target detection model Faster R-CNN; then the 36 object proposals with the highest confidence are retained, denoted as R1 = Faster R1-CNN(V); where R1∈R 36×2048 , then align the dimension of R1 with the text embedding dimension of BART, and the final visual feature representation is R∈R 36×d , d represents the dimension of BART.

[0017] Furthermore, the specific implementation of S3 includes:

[0018] By using a multi-head self-attention layer, the self-attention operation is performed on the text to capture the intra-modal interaction of the text. This operation is expressed as the following formula by collecting information about nearby words: E′=Norm(E+ATT self (E));

[0019] Among them, ATT self Represents multi-head self-attention, the text feature is set as query / key / value matrix, and Norm represents normalization;

[0020] At the same time, a cross-modal Transformer layer is used to obtain cross-modal interaction between text and visual modalities, where the original text feature E is used as the query matrix and the visual feature R is used as the key / value matrix, which is expressed as the following formula: E X→V =ATT cross (E,R);

[0021] Among them, ATT cross represents the cross-modal attention layer; then, E X→V It is fed to the feedforward network FFN, followed by a normalization layer to further enhance the text representation, and the additional residual connection added from E′ is used to obtain the final text representation that integrates the visual information, that is, the enhanced text representation is: E″=Norm(E′+FFN(E X→V )).

[0022] Furthermore, the specific implementation of S4 includes:

[0023] The enhanced text representation obtained through interactive cross-modal operations further aims to filter irrelevant regional features for the text; essentially, selective fusion receives two inputs, one is the enhanced text representation E", and the other is the pure visual feature R;

[0024] First, R and E' are concatenated into a bimodal factor [R; E'], and the formula for using them to generate the similarity weight g between regional features and text features is as follows:

[0025] g=sigmoid(Linear([R;E″]))

[0026] Among them, sigmoid represents the Sigmoid nonlinear activation function; then, similarity weights are used to obtain regional features related to the text through selective filters. The calculation method is:

[0027] Furthermore, the specific implementation of S5 includes:

[0028] By linearly interpolating a pair of training samples, the training data is made to behave linearly, thereby creating new samples; the synthetic samples are generated as follows:

[0029] Where λ is a scalar used to balance text features and visual features, sampled from the Beta(α,β) distribution: λ~Be(α,β);

[0030] Where Be represents the Beta distribution, α and β are hyperparameters for controlling the λ distribution, and R′ is output as a visual representation containing multimodal information.

[0031] Furthermore, the specific implementation of S6 includes:

[0032] In the multimodal BART encoder, a special token is inserted before the visual features output by the cross-modal Mixup module. and are used to identify the starting and ending positions of the visual features, and the original text features E and the visual representation R′ containing multimodal information are used as the input of the multimodal output D′:

[0033] in represents the concatenation operation; D′ is then fed into the position embedding layer to obtain the final multimodal representation D″, which is expressed as: D″=Dropout(Norm(PE(D′)+D′);

[0034] where D″∈R (T+36)×d , PE represents the position embedding layer; finally, D″ is fed into the BART encoder as the final multimodal input;

[0035] The BART encoder output is represented as H m , the predicted distribution is calculated by Softmax and MLP: P(θ)=Softmax(MLP(H m )), where MLP is a multi-layer perceptron.

[0036] Beneficial effects of the present invention:

[0037] 1. By using images as pivotal information, this paper successfully eliminates the error propagation problem that occurs in joint multimodal aspect-level sentiment analysis, reduces the task difference between multimodal aspect word extraction and multimodal aspect-level sentiment analysis, and achieves dynamic enhancement of the performance of multimodal aspect word extraction and multimodal aspect-level sentiment analysis. By dynamically extracting visual details, their performance is improved.

[0038] 2. This paper can adaptively bridge the semantic gap between text and image representations and adaptively transfer text-based pre-trained models to JMABSA tasks;

[0039] 3. This invention can adaptively integrate task-specific visual information into the pre-trained BART encoder, prompting the network to learn multimodal representations, thereby enhancing the performance of the model;

[0040] 4. Experimental results on the Twitter15 and Twitter17 datasets show that our proposed method significantly improves the performance of multimodal aspect word extraction and multimodal aspect-level sentiment analysis, and improves the F1 score on the two test sets, significantly outperforming multiple baseline models in performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is an overall framework diagram of a multi-label text classification data enhancement method based on active feature selection provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The embodiments of the present invention are described below in conjunction with the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] like Figure 1 FIG. 1 is a diagram showing the overall framework of a joint multimodal aspect-level sentiment analysis method based on adaptive attention fusion provided by an embodiment of the present invention. The joint multimodal aspect-level sentiment analysis method based on adaptive attention fusion includes the following steps:

[0044] The steps include:

[0045] S1. Use the embedding layer of the pre-trained language model to embed the text after word segmentation in the image text pair data to obtain a word embedding vector; in some feasible implementations, the specific implementation of S1 includes:

[0046] Use the sequence-to-sequence model BART to obtain word embeddings, using <s> and< / s> To indicate the beginning and end of a sentence, the original text feature of the sample is represented as E, E∈R T×d , where T represents the text length and d represents the dimension of BART.

[0047] S2. Use the object detection model to extract visual features from each input image of the image-text pair data;

[0048] In some feasible implementations, the specific implementation of S2 includes:

[0049] All object proposals are extracted from a given image V by the target detection model Faster R-CNN; then the 36 object proposals with the highest confidence are retained, denoted as R1 = FasterR1-CNN(V); where R1∈R 36×2048 , then align the dimension of R1 with the text embedding dimension of BART, and the final visual feature representation is R∈R 36×d , d represents the dimension of BART.

[0050] S3, an adaptive image-to-text fusion module, fuses visual features into text representation and improves feature fusion effects through multimodal interaction;

[0051] In some feasible implementations, the specific implementation of S3 includes:

[0052] By using a multi-head self-attention layer, the self-attention operation is performed on the text to capture the intra-modal interaction of the text. This operation is expressed as the following formula by collecting information about nearby words: E′=Norm(E+ATT self (E));

[0053] Among them, ATT self Represents multi-head self-attention, the text feature is set as query / key / value matrix, and Norm represents normalization;

[0054] At the same time, a cross-modal Transformer layer is used to obtain cross-modal interaction between text and visual modalities, where the original text feature E is used as the query matrix and the visual feature R is used as the key / value matrix, which is expressed as the following formula: E X→V =ATT cross (E,R);

[0055] Among them, ATT cross represents the cross-modal attention layer; then, E X→V It is fed to the feedforward network FFN, followed by a normalization layer to further enhance the text representation, and the additional residual connection added from E′ is used to obtain the final text representation that integrates the visual information, that is, the enhanced text representation is: E″=Norm(E′+FFN(E X→V )).

[0056] S4. Using a selective fusion module to filter relevant regional features based on text representation to improve sentiment analysis performance; in some feasible implementations, the specific implementation of S4 includes:

[0057] The enhanced text representation obtained through interactive cross-modal operations further aims to filter irrelevant regional features for the text; essentially, selective fusion receives two inputs, one is the enhanced text representation E", and the other is the pure visual feature R;

[0058] First, R and E' are concatenated into a bimodal factor [R; E'], and the formula for using them to generate the similarity weight g between regional features and text features is as follows:

[0059] g=sigmoid(Linear([R;E″]))

[0060] Among them, sigmoid represents the Sigmoid nonlinear activation function; then, similarity weights are used to obtain regional features related to the text through selective filters. The calculation method is:

[0061] S5. Using a cross-modal Mixup module to achieve interaction between text and images and enhance semantic connections between features. In some feasible implementations, the specific implementation of S5 includes:

[0062] By linearly interpolating a pair of training samples, the training data is made to behave linearly, thereby creating new samples; the synthetic samples are generated as follows:

[0063] Where λ is a scalar used to balance text features and visual features, sampled from the Beta(α,β) distribution: λ~Be(α,β);

[0064] Where Be represents the Beta distribution, α and β are hyperparameters for controlling the λ distribution, and R′ is output as a visual representation containing multimodal information.

[0065] S6. Enhance the encoding of multimodal information in a sequence-to-sequence model using a visually enhanced BART module. In some feasible implementations, the specific implementation of S6 includes:

[0066] In the multimodal BART encoder, a special token is inserted before the visual features output by the cross-modal Mixup module. and are used to identify the starting and ending positions of the visual features, and the original text features E and the visual representation R′ containing multimodal information are used as the input of the multimodal output D′:

[0067] in Indicates the splicing operation; then D ′ Feed it to the position embedding layer to obtain the final multimodal representation D″, expressed as: D″=Dropout(Norm(PE(D′)+D′);

[0068] where D″∈R (T+36)×d , PE represents the position embedding layer; finally, D″ is fed into the BART encoder as the final multimodal input;

[0069] The BART encoder output is represented as H m , the predicted distribution is calculated by Softmax and MLP: P(θ)=Softmax(MLP(H m )), where MLP is a multi-layer perceptron.

[0070] To illustrate the effectiveness of the present invention, we compared it with existing methods and used two multimodal datasets, Twitter-15 and Twitter-17, for validation. The evaluation metrics were consistent with those of the comparison methods, using F1 as the metric for evaluating classification results. The experimental results for the datasets used are shown in Table 1:

[0071] Table 1 Experimental results

[0072]

[0073] As shown in Table 1, the experimental group's evaluation metrics significantly improve compared to all baseline models. Previous pipeline approaches, due to the separate execution of the two subtasks, suffer from error propagation issues. Our proposed model achieves top performance on the TWITTER-17 dataset, as evaluated by the F1 score. On the TWITTER-15 dataset, its F1 score is only 0.2% lower than that of the highly pre-trained VLP-MABSA. This demonstrates that our proposed model is competitive with all state-of-the-art methods, demonstrating its effectiveness.

[0074] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0075] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A joint multimodal aspect-level sentiment analysis method based on adaptive attention fusion, characterized by: The steps include: S1. Use the embedding layer of the pre-trained language model to embed the text after word segmentation in the image text pair data to obtain the word embedding vector; S2. Use the object detection model to extract visual features from each input image of the image-text pair data; S3, an adaptive image-to-text fusion module, fuses visual features into text representation and improves feature fusion effects through multimodal interaction; S4, using the selective fusion module to filter relevant regional features based on text representation to improve sentiment analysis performance; S5. Use the cross-modal Mixup module to achieve interaction between text and images and enhance the semantic connection between features; S6. Enhance the encoding of multimodal information in sequence-to-sequence models using the visually enhanced BART module. The specific implementation of S3 includes: By using a multi-head self-attention layer, the self-attention operation is performed on the text to capture the intra-modal interaction of the text. This operation is expressed as the following formula by collecting information about nearby words: ′ =Norm(E+ATT self (E)); Among them, ATT self Represents multi-head self-attention, the text feature is set as query / key / value matrix, and Norm represents normalization; At the same time, a cross-modal Transformer layer is used to obtain cross-modal interaction between text and visual modalities, where the original text feature E is used as the query matrix and the visual feature R is used as the key / value matrix, which is expressed as the following formula: E X→V =ATT cross ((E,R); Among them, ATT cross represents the cross-modal attention layer; then, E X→V It is fed to the feedforward network FFN, followed by a normalization layer to further enhance the text representation. The additional residual connection added from E′ is used to obtain the final text representation that integrates the visual information, that is, the enhanced text representation is: E″=Norm((E′+FFN((E X→V )); The specific implementation of S4 includes: The enhanced text representation obtained through interactive cross-modal operations further aims to filter irrelevant regional features for the text; essentially, selective fusion receives two inputs, one is the enhanced text representation E'' and the other is the pure visual feature R; First, R and E' are concatenated into a bimodal factor [R; E'], and the formula for using them to generate the similarity weight g between regional features and text features is as follows: g=sigmoid(Linear(([R;E″])) Among them, sigmoid represents the Sigmoid nonlinear activation function; then, similarity weights are used to obtain regional features related to the text through selective filters. The calculation method is: The specific implementation of S5 includes: By linearly interpolating a pair of training samples, the training data is made to behave linearly, thereby creating new samples; the synthetic samples are generated as follows: Where λ is a scalar used to balance text features and visual features, sampled from the Beta(α,β) distribution: λ~Be(α,β); Among them, Be represents Beta distribution, α and β are hyperparameters used to control λ distribution, and R ′ The output is a visual representation containing multimodal information; The specific implementation of S6 includes: In the multimodal BART encoder, a special token is inserted before the visual features output by the cross-modal Mixup module. and are used to identify the starting and ending positions of the visual features and to combine the original text features E with the visual representation R containing multimodal information ′ As input to the multimodal output D': in Indicates the splicing operation; then D ′ Feed it to the position embedding layer to obtain the final multimodal representation D″, expressed as: D″=Dropout((Norm(PE(D′)+D′)); where D″∈R (T+36)×d , PE represents the position embedding layer; finally, D″ is fed into the BART encoder as the final multimodal input; The BART encoder output is represented as H m , the predicted distribution is calculated by Softmax and MLP: P((θ)=Softmax((MLP(H m )), where MLP is a multi-layer perceptron.

2. The joint multimodal aspect-level sentiment analysis method based on adaptive attention fusion according to claim 1 is characterized in that The specific implementation of S1 includes: Use the sequence-to-sequence model BART to obtain word embeddings, using <s> and< / s> To indicate the beginning and end of a sentence, the original text feature of the sample is represented as E, E∈R T×d , where T represents the text length and d represents the dimension of BART.

3. The joint multimodal aspect-level sentiment analysis method based on adaptive attention fusion according to claim 1 is characterized in that The specific implementation of S2 includes: All object proposals are extracted from a given image V by the target detection model Faster R-CNN; then the 36 object proposals with the highest confidence are retained, denoted as R1 = Fasterr1-CNN(V); where r1∈R 36×2048 , then align the dimension of R1 with the text embedding dimension of BART, and the final visual feature representation is R∈R 36×d , d represents the dimension of BART.

Images