A multimodal aspect-level sentiment analysis method and system

Abstract

The present invention provides a multimodal aspect-level sentiment analysis method and system, the method comprising: obtaining a multimodal data set, using a text pre-training model to obtain a feature representation of the text, using a target detection model to obtain a feature representation of a regional image, and obtaining regional visual features of the image through a Transformer model; modeling the interaction between text features and regional visual features through a cross-modal Transformer layer, and generating a visual filtering matrix to remove visual noise. Through a cross-modal attention mechanism, the regional visual features are aligned with the most relevant text information, and an adaptive gating mechanism is used to fuse effective regional word pairs; based on the image-text relationship, the features are screened and filtered, and sentiment classification is achieved through the output layer. The present invention fully considers the impact of context on sentiment analysis, effectively removes redundancy through deep filtering and fusion of features, and accurately identifies sentiment tendencies in different contexts.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and computer vision, and in particular to a multimodal aspect-level sentiment analysis method and system. Background Art

[0002] With the rapid development of social media, multimedia data containing multiple modalities such as text, pictures, videos, and audio are widely disseminated on social platforms such as Facebook and Twitter. Multimodal Sentiment Analysis (MSA) has emerged in this context. It automatically identifies and captures users' emotional tendencies or attitudes by analyzing data in different modalities. This technology has wide application value in helping companies judge user satisfaction with their products and assisting governments in understanding public opinions on hot issues. However, in practical applications, people usually express multiple different opinions in a tweet or comment. For example, in a comment such as "Although the service is poor, the food is delicious", the sentiment polarity of the two aspects of "food" and "service" is positive and negative respectively. Therefore, identifying and analyzing the sentiment polarity of different aspects of entities in multimodal data has become a key challenge in the field of multimodal sentiment analysis.

[0003] To address this challenge, aspect-based multimodal sentiment analysis (ABMSA) has emerged as an emerging method. ABMSA aims to accurately identify the sentiment polarity of each aspect entity from multimodal data (usually text-image pairs). With the popularity of mobile devices and social networks, users are increasingly inclined to combine images with text to express emotions or opinions, which makes ABMSA show broad application prospects in personalized advertising, emotional cross-modal retrieval, opinion mining, decision support and other fields.

[0004] In view of the wide application prospects of ABMSA, current researchers have proposed many multimodal aspect-level sentiment analysis models. However, previous methods mainly rely on coarse-grained fusion mechanisms at the data level or decision level, which have the following four shortcomings: (1) The noise that may exist in visual information is ignored. (2) The features of each modality have not yet been accurately aligned and are affected by the semantic differences between different modalities. (3) The complementary role of contextual information in sentiment analysis is ignored, that is, the same object or the same word may trigger different emotions in different contexts. (4) Before the output layer, the extracted high-level features are not further filtered and fused, resulting in feature redundancy. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a multimodal aspect-level sentiment analysis method and system in view of the deficiencies in the prior art.

[0006] The method comprises the following steps:

[0007] Step 1: obtain a multimodal dataset, where each sample in the multimodal dataset contains a set of text, a related image, and an aspect word sequence, where the aspect word sequence is a subsequence of the text; use a special marker Replace the aspect word sequence in the text, connect the replaced text with the aspect word sequence, and use the optimized version of BERT, Robustly optimized BERT approach (RoBERTa), to encode and obtain text feature representation. In terms of image feature encoding, use Faster Region-based Convolutional Neural Network (Faster R-CNN) to obtain the regional feature representation of the image, and then use the Transformer model to enhance the modeling of regional visual objects, so as to obtain the regional level visual features of the image.

[0008] Step 2: Model the interaction between the text feature representation obtained in step 1 and the region-level visual features through the cross-modal Transformer layer, combine the text features and the region-level visual features to obtain the image-text relationship, and then generate a visual filter matrix to denoise the region-level visual features;

[0009] Step 3: Based on the cross-modal attention mechanism, the word-level attention operation is performed on the denoised region-level visual features in step 2 to identify the text information that best matches each visual region and align the image region with the text words in the feature embedding space.

[0010] Step 4: Use region and word adaptive gates to control the feature fusion of region word pairs to eliminate the influence of mismatched region word pairs, thereby obtaining more effective features.

[0011] In step 5, the features obtained in step 4 are supplemented with global information, the supplemented features are selected according to the image-text relationship, and the multimodal features are filtered and fused using the gating mechanism, and then the sentiment classification results are obtained through the output layer; the model loss is calculated using the cross entropy loss function, the training network parameters are optimized, and the trained multimodal sentiment analysis model is obtained.

[0012] Step 1 includes:

[0013] Step 1.1, encode the text modality: For each text S in a sample, use a special marker Replace the aspect word sequence in the text and convert the replaced text Aspect word sequence Connection, as text input , using the text pre-training model RoBERTa to obtain text feature representation :

[0014]

[0015] in , is the hidden dimension of the text feature, is the input length, represents the real number space;

[0016] Step 1.2, encode the image: for the input image , use the fast regional convolutional neural network FasterR-CNN to detect object proposals, sort the proposals according to the object category detection probability, and retain the first X1 (usually 100) object proposals as the regional representation of the image Regional feature representation :

[0017]

[0018] in , is the hidden dimension of the regional feature representation V, is the number of regions; the regional feature representation V is converted to the same hidden dimension d as the text feature through a linear layer to achieve dimensional alignment of text and visual features, and then the object-level regional visual features are obtained through the Transformer model. :

[0019]

[0020] in , and is a learnable parameter.

[0021] Step 2 includes:

[0022] Step 2.1: Use the cross-modal Transformer layer to represent text features as the query and the region-level visual features As keys and values, like this:

[0023]

[0024] in It is the region-level visual feature associated with each word in the text;

[0025] The cross-modal Transformer layer is specifically implemented as follows:

[0026]

[0027]

[0028]

[0029]

[0030] in , , They are respectively the query vector, key vector, and value vector after linear transformation; , , is a learnable parameter; is the scaling factor, ; represents cross-modal operation; T represents matrix transpose;

[0031] Region-level visual features associated with each word in the text Using the maximum pooling operation , to obtain the most important features for relationship classification; based on , use the Sigmoid function to get the predicted probability of the image-text relationship. The specific formula is:

[0032]

[0033]

[0034] in represents the most significant features related to the image-text relationship task, , is the maximum pooling operation; The category representing the image-text relationship, The category of the image-text relationship of the model prediction sample is The probability of is a scalar in the range [0, 1], and is a learnable parameter;

[0035] Use cross entropy loss to optimize the image-text relationship task:

[0036]

[0037] in is the loss function of the image-text relationship task, is the number of samples in the multimodal dataset, represents the actual category of the image-text relationship classification of the u-th sample, The image-text relationship category predicted by the model for the uth sample is The probability of

[0038] Predicting probability scores using image-text relationships Construct a visual mask matrix , all elements in the visual mask matrix R are equal to the predicted probability scores of the image-text relationship , the denoised regional-level visual features are obtained through the visual mask matrix R :

[0039]

[0040] in ; is element-wise multiplication.

[0041] Step 3 includes:

[0042] Step 3.1: Given a set of denoised region-level visual features and text feature representation set ,in Represents the denoised Region-level visual features, Indicates the first features of each word; compute the region-level visual and word-level textual attention matrix , the attention matrix Scaling and normalization to region-word attention matrix , using the region word attention matrix Aggregate all word features matched for each visual area, the formula is:

[0043]

[0044]

[0045]

[0046] in and is a learnable parameter, represents the relationship between the i-th visual area and the j-th word, ; is the scaling factor, ;matrix No. Row indicates the The text features matched by the visual regions;

[0047] Step 3.2, extract the matrix Perform the following processing:

[0048]

[0049]

[0050]

[0051] in and is a learnable parameter; Represents the relationship score matrix between region word pairs, ; Representation Matrix The matrix after scaling and normalization, is the scaling factor, ; Represents the regional word pair features after further extraction.

[0052] Step 4 includes:

[0053] Step 4.1, given the denoised region-level visual features and regional word pair features , calculate the adaptive gate to evaluate the degree of alignment:

[0054]

[0055]

[0056]

[0057]

[0058] in represents the word pair feature of the i-th region, represents the i-th region-level visual feature, Represents the measure of regional word pair features Adaptive gating value for alignment degree; It is the fusion feature; is a learnable parameter, ReLU is a linear rectification function; finally, the fused features are obtained , Represents the first regional word pair features, , which implies the alignment information between regions and word pairs.

[0059] Step 5 includes:

[0060] Step 5.1: De-noised region-level visual features , text feature representation The features obtained in step 4 Connect them separately, and pass the connected features through the fully connected layer to obtain the linearly transformed features. The formula is:

[0061]

[0062]

[0063] in Representing region-level visual features and The combined fusion features, Representing text features and The combined fusion features, , , , is a learnable parameter, Indicates a connection operation;

[0064] In the first case, if the predicted probability of the image-text relationship indicates that the probability of the relationship between the image and the text being "no relationship" is greater than the probability of "relationship", then first calculate and The relationship matrix of Plus multiplication of the relationship matrix , as the final feature set:

[0065]

[0066]

[0067] in Indicates that the image and text are unrelated and The relationship matrix of It represents the final feature set after fusion when the image and text have no relationship. , is a learnable parameter;

[0068] In the second case, if the predicted probability of the image-text relationship indicates that the probability of the relationship between the image and the text being "related" is greater than the probability of "no relationship", then first calculate and The relationship matrix is ​​then supplemented with region-level visual features. Plus multiplication of the relationship matrix , as the final feature set:

[0069]

[0070]

[0071] in Indicates that the image and text are related and The relationship matrix, It represents the final feature set after fusion when the image and text are related. , is a learnable parameter;

[0072] Step 5.2: The final feature set extracted in step 5.1 or Treat as a feature set , the feature set Input to the softmax layer to predict the final emotion:

[0073]

[0074] in The predicted emotion category is The probability of , are learnable parameters;

[0075] Use cross entropy loss to optimize the multimodal sentiment analysis task based on aspect words, which is achieved through sentiment supervision representation:

[0076]

[0077] in is the loss function for the multimodal aspect-level sentiment classification task, is the number of samples in the multimodal dataset, represents the actual category of the qth sample sentiment classification, It means that the model predicts the sentiment classification category of the qth sample as The probability of

[0078] Then we get the final loss function .

[0079] In step 5.2, the final loss function It is expressed as:

[0080]

[0081] in is a hyperparameter.

[0082] The present invention also provides a multimodal aspect-level sentiment analysis system implemented based on the method, comprising:

[0083] The image-text relationship module obtains a multimodal dataset, in which each sample contains a set of text, an associated image, and an aspect word sequence, wherein the aspect word sequence is a subsequence of the text. The special symbol $T$ is used to replace the aspect word sequence in the text, and the replaced text is connected to the aspect word sequence, and encoded using RoBERTa, an optimized version of BERT, to obtain text feature representation; in terms of image feature encoding, the fast regional convolutional neural network Faster R-CNN is used to obtain the regional feature representation of the image, and then the regional feature representation of the image is passed through the Transformer model to enhance the modeling of regional visual objects, thereby obtaining the regional-level visual features of the image; the interaction between the text feature representation and the regional-level visual features is modeled through the cross-modal Transformer layer, and the text features and the regional-level visual features are combined to obtain the image-text relationship, and then a visual filter matrix is ​​generated to denoise the regional-level visual features;

[0084] The cross-modal alignment correction module, based on the cross-modal attention mechanism, performs word-level attention operations on the denoised regional-level visual features to identify the text information that best matches each visual region, and aligns the image region with the text words in the feature embedding space. Since the learned regional word pairs are not all perfectly matched, mismatched regional word pair features will reduce performance. Therefore, the present invention uses regional and word adaptive gates to control the feature fusion of regional word pairs to eliminate the influence of mismatched regional word pairs, thereby obtaining more effective features;

[0085] The feature fusion module combines global information to supplement the features obtained by the cross-modal alignment correction module, selects the supplemented features according to the image-text relationship, and uses the gating mechanism to filter and fuse the multimodal features, and then obtains the sentiment classification results through the output layer.

[0086] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0087] The present invention also provides a storage medium storing a computer program or instruction. When the computer program or instruction is run on a computer, the steps of the method described are executed.

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] (1) The present invention utilizes the image-text relationship module to generate a visual mask matrix, which effectively removes visual noise, avoids interference, and improves the accuracy and robustness of the model.

[0090] (2) By introducing a cross-modal alignment correction module, the correspondence between region-level visual features and text words is captured, and the effective region-word pairs are fused through an adaptive gating mechanism to eliminate the influence of mismatched region-word pairs, enhance multimodal feature fusion, and improve the interaction effect.

[0091] (3) The present invention fully considers the impact of context on sentiment analysis, effectively removes redundancy through deep filtering and fusion of features, and accurately identifies sentiment tendencies in different contexts.

[0092] (4) The method proposed in this invention achieved competitive performance on Twitter 2015 and Twitter 2017. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 Flow chart of the method of the present invention.

[0094] Figure 2 It is a model framework diagram of the system of the present invention. DETAILED DESCRIPTION

[0095] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0096] This embodiment provides a multimodal aspect-level sentiment analysis method and system, the method flow chart is as follows Figure 1 As shown, it includes: using the optimized version of the text pre-training model BERT to obtain the feature representation of the text, using the target detection model Faster R-CNN to extract the feature representation of the regional image, and then obtaining the regional visual features of the image through Transformer; denoising the regional visual features through the image-text relationship module, and then accurately aligning the denoised regional visual features and text features through the cross-modal alignment correction module; finally, filtering and fusion of the aforementioned extracted features based on the image-text relationship score and combined with global information, and obtaining the classification result through the output layer.

[0097] The overall architecture of the system model provided in this embodiment is as follows Figure 2 As shown in the figure, the model consists of three modules: (1) image-text relationship module; (2) cross-modal alignment correction module; and (3) feature fusion module.

[0098] This embodiment method first defines the task:

[0099] For each sample, it contains Words of text , picture I, and aspect word sequence , where A is a subsequence of S, Indicates words, Indicates the first The first task formulated in this embodiment is the image-text relationship task. , assuming that the relationship between text S and image I is determined by the correlation label Indicates that can be related or unrelated. The goal of this task is to learn a Map to The second task formulated in this embodiment is a multimodal aspect-level sentiment classification task. For each sample , set aspect words With emotional labels The main goal of this task is to learn a Map to Sentiment classifier.

[0100] Then follow these steps:

[0101] Step 1, obtaining a multimodal dataset, wherein each sample in the multimodal dataset includes a set of texts, an associated image, and an aspect word sequence, wherein the aspect word sequence is a subsequence of the text; using a special symbol $T$ to replace the aspect word sequence in the text, and connecting the replaced text with the aspect word sequence, encoding using the optimized version of BERT, Robustly optimized BERT approach (RoBERTa), to obtain text feature representation; in terms of image feature encoding, using a fast region convolutional neural network Faster Region-based Convolutional Neural Network (Faster R-CNN) to obtain the regional feature representation of the image, and then the regional feature representation of the image is passed through a Transformer model to enhance the modeling between regional visual objects, thereby obtaining the regional level visual features of the image;

[0102] Step 2: Model the interaction between the text feature representation obtained in step 1 and the region-level visual features through the cross-modal Transformer layer, combine the text features and the region-level visual features to obtain the image-text relationship, and then generate a visual filter matrix to denoise the region-level visual features;

[0103] Step 3: Based on the cross-modal attention mechanism, the word-level attention operation is performed on the denoised region-level visual features in step 2 to identify the most relevant text information for each visual region and align the image region with the text words in the feature embedding space.

[0104] Step 4: Use region and word adaptive gates to control the feature fusion of region word pairs to eliminate the influence of negative region word pairs, thereby obtaining more effective features.

[0105] In step 5, the features obtained in step 4 are supplemented with global information, the supplemented features are selected according to the image-text relationship, and the multimodal features are filtered and fused using the gating mechanism, and then the sentiment classification results are obtained through the output layer; the model loss is calculated using the cross entropy loss function, the training network parameters are optimized, and the trained multimodal sentiment analysis model is obtained.

[0106] Step 1 includes:

[0107] Step 1.1, encode the text modality: for each sample text , using special flags Replace the aspect word sequence in the text and convert the replaced text Aspect word sequence Connection, as text input , using the text pre-training model RoBERTa to obtain text feature representation :

[0108]

[0109] in , represents the real number space;

[0110] Step 1.2, encode the image: for the input image , Faster R-CNN is used to detect object proposals, sort the proposals according to the object category detection probability, and retain the region representation of the top 100 object proposals as the image Regional feature representation :

[0111]

[0112] in ; Represent regional features Transformed to the same hidden dimension as the text features through a linear layer , to achieve the alignment of text and visual features in dimension, and then obtain the object-level region-level visual features through the Transformer model :

[0113]

[0114] in , and is a learnable parameter.

[0115] Step 2 includes:

[0116] Step 2.1: Use the cross-modal Transformer layer to represent text features as the query and the region-level visual features As keys and values, like this:

[0117]

[0118] in It is the region-level visual feature associated with each word in the text;

[0119] The cross-modal Transformer layer is specifically implemented as follows:

[0120]

[0121]

[0122]

[0123]

[0124] in , , They are respectively the query vector, key vector, and value vector after linear transformation; , , is a learnable parameter; is the scaling factor, .

[0125] Region-level visual features associated with each word in the text Using the maximum pooling operation , to obtain the most important features for relationship classification. , use the Sigmoid function to get the predicted probability of the image-text relationship. The specific formula is:

[0126]

[0127]

[0128] in represents the most significant features related to the image-text relationship task, , is the maximum pooling operation; The category representing the image-text relationship, The category of the image-text relationship of the model prediction sample is The probability of is a scalar in the range [0, 1], and is a learnable parameter;

[0129] Use cross entropy loss to optimize the image-text relationship task:

[0130]

[0131] in is the loss function of the image-text relationship task, is the number of samples in the multimodal dataset, represents the actual category of the image-text relationship classification of the u-th sample, The image-text relationship category predicted by the model for the uth sample is The probability of

[0132] Predicting probability scores using image-text relationships Construct a visual mask matrix , all elements in the visual mask matrix R are equal to the predicted probability scores of the image-text relationship , the denoised regional-level visual features are obtained through the visual mask matrix R :

[0133]

[0134] in ; is element-wise multiplication.

[0135] Step 3 includes:

[0136] Step 3.1: Given a set of denoised region-level visual features and text feature representation set ,in Represents the denoised Region-level visual features, Indicates the first Compute the regional visual and word-level textual attention matrix , the attention matrix Scaling and normalization to region-word attention matrix , using the region word attention matrix Aggregate all word features matched for each visual area, the formula is:

[0137]

[0138]

[0139]

[0140] in , is a learnable parameter, represents the relationship between the i-th visual area and the j-th word, ; is the scaling factor, ;matrix No. Row indicates the The text features matched by the visual regions;

[0141] Step 3.2, extract the matrix Perform the following processing:

[0142]

[0143]

[0144]

[0145] in and is a learnable parameter; Represents the relationship score matrix between region word pairs, ; Representation Matrix The matrix after scaling and normalization, is the scaling factor, ; Represents the regional word pair features after further extraction.

[0146] Step 4 includes:

[0147] Step 4.1, given the denoised region-level visual features and regional word pair features , calculate the adaptive gate to evaluate the degree of alignment:

[0148]

[0149]

[0150]

[0151]

[0152] in represents the word pair feature of the i-th region, represents the i-th region-level visual feature, Represents the measure of regional word pair features Adaptive gating value for alignment degree; is the fused feature, is a learnable parameter, ReLU is a linear rectification function; finally, the fused features are obtained , Represents the first regional word pair features, , which implies the alignment information between regions and word pairs.

[0153] Step 5 includes:

[0154] Step 5.1: De-noised region-level visual features , text feature representation The features obtained in step 4 Connect them separately, and pass the connected features through the fully connected layer to obtain the linearly transformed features. The formula is:

[0155]

[0156]

[0157] in Representing region-level visual features and The combined fusion features, Representing text features and The combined fusion features, , , , is a learnable parameter, Represents a join operation.

[0158] In the first case, if the predicted probability of the image-text relationship indicates that the probability of the relationship between the image and the text being "no relationship" is greater than the probability of "relationship", then first calculate and The relationship matrix of Plus multiplication of the relationship matrix , as the final feature set:

[0159]

[0160]

[0161] in Indicates that the image and text are unrelated and The relationship matrix of It represents the final feature set after fusion when the image and text have no relationship. is element-wise multiplication, , is a learnable parameter;

[0162] In the second case, if the predicted probability of the image-text relationship indicates that the probability of the relationship between the image and the text being "related" is greater than the probability of "no relationship", then first calculate and The relationship matrix is ​​then supplemented with region-level visual features. Plus multiplication of the relationship matrix , as the final feature set:

[0163]

[0164]

[0165] in Indicates that the image and text are related and The relationship matrix, It represents the final feature set after fusion when the image and text are related. is element-wise multiplication, , is a learnable parameter;

[0166] Step 5.2: extract the features from step 5.1 or Treat as a feature set , the feature set Input to the softmax layer to predict the final emotion:

[0167]

[0168] in The predicted sentiment category is The probability of , are learnable parameters.

[0169] Use cross entropy loss to optimize the multimodal sentiment analysis task based on aspect words, which is achieved through sentiment supervision representation:

[0170]

[0171] in is the loss function for the multimodal aspect-level sentiment classification task, is the number of samples in the multimodal dataset, represents the actual category of the qth sample sentiment classification, It means that the model predicts the sentiment classification category of the qth sample as probability.

[0172] The final loss function It is expressed as:

[0173]

[0174] in is a hyperparameter.

[0175] In this embodiment, the following settings are made:

[0176] For the image-text relationship task, this paper selected the dataset annotated in the multimodal aspect-level sentiment analysis paper "Targeted Multimodal Sentiment Classification based on Coarse-to-Fine Grained Image-Target Matching" published at the International Joint Conference on Artificial Intelligence in June 2022. This dataset is constructed based on the Twitter2017 dataset. The authors randomly selected 1176, 588, and 588 samples from the training set, development set, and test set of the Twitter2017 dataset for annotation, respectively. The statistical data of the dataset are as follows:

[0177] Table 1

[0178]

[0179] For the final multimodal aspect-level sentiment classification task, this embodiment uses the public datasets Twitter2015 and Twitter2017. These two Twitter datasets cover tweets published in 2014-2015 and 2016-2017. Multimodal tweets include text, pictures in tweets, and aspect terms mentioned therein. Each aspect term is assigned a sentiment label, which comes from the set of {positive, negative, neutral}. Table 2 shows the detailed statistics of the dataset.

[0180] Table 2

[0181]

[0182] In the experiment of this example, the optimized version of pre-trained BERT, RoBERTa-base, is used as the text encoder, and the fast regional convolutional neural network Faster R-CNN is used as the object detector. The batch size is set to 32, the learning rate of the image-text relationship task is 1e-6, the learning rate of the multimodal aspect-level sentiment classification task is 1e-5, and the parameters are Set to 1. After 12 batches of training, the model with the best performance on the validation set is selected as the final model, and its performance is evaluated on the test set. The present invention is based on the PyTorch framework and trained on the RTX3090 GPU. The present invention uses accuracy, recall and F1 score to evaluate the performance of the model.

[0183] Comparison Baseline: This example uses the following models as baselines to compare model performance. (1) ResNet-Aspect: This baseline method uses ResNet and BERT to extract visual and text features respectively, and performs sentiment classification through linear layers. (2) AE-LSTM: A model that focuses on different parts of a sentence when processing different aspect words through an attention mechanism. (3) BERT: Uses the Transformer model to capture the interaction between aspects and text content. (4) DualGCN: A dual graph convolutional network that integrates syntactic knowledge and semantic information is proposed. (5) ViLBERT: It proposes cross-modal feature interaction through a co-attention Transformer layer, extending BERT to joint reasoning about text and images. (6) TomBERT: A multimodal BERT architecture that obtains target-sensitive visual features and then combines text features. (7) CapTrBERT: A new method that uses an image translation task to map visual features to text features and then uses the BERT encoder structure for multimodal sentiment classification (8) HIMT: A general hierarchical interactive multimodal Transformer model. (9) KEF-TomBERT: A knowledge-enhanced framework that uses adjective-noun pairs extracted from images to enhance visual attention and emotion prediction. (10) ITM: It uses two auxiliary tasks to achieve coarse-grained to fine-grained matching of image targets and reduce the interference of noise in visual features.

[0184] Table 3

[0185]

[0186] As can be seen from Table 3, the method proposed in the present invention outperforms the performance of all compared baseline models on two public datasets, thereby verifying the effectiveness of the present invention.

[0187] The present invention provides a multimodal aspect-level sentiment analysis method and system. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A multimodal aspect-level sentiment analysis method, characterized in that: The following steps are involved: Step 1, obtaining a multimodal dataset, wherein each sample in the multimodal dataset includes a set of texts, an associated image, and an aspect word sequence, wherein the aspect word sequence is a subsequence of the text; using a flag to replace the aspect word sequence in the text, and connecting the replaced text with the aspect word sequence, encoding using RoBERTa, an optimized version of BERT, to obtain text feature representation; in terms of image feature encoding, using Faster R-CNN to obtain regional feature representation of the image, and then using the Transformer model to enhance the modeling of regional visual objects through the regional feature representation of the image, thereby obtaining regional-level visual features of the image; Step 2: Model the interaction between the text feature representation obtained in step 1 and the region-level visual features through the cross-modal Transformer layer, combine the text features and the region-level visual features to obtain the image-text relationship, and then generate a visual filter matrix to denoise the region-level visual features; Step 3: Based on the cross-modal attention mechanism, the word-level attention operation is performed on the denoised region-level visual features in step 2 to identify the text information that best matches each visual region and align the image region with the text words in the feature embedding space. Step 4: Use region and word adaptive gates to control the feature fusion of region word pairs, eliminate the influence of mismatched region word pairs, and thus obtain more effective features; Step 5: Supplement the features obtained in step 4 with global information, select the supplemented features according to the image-text relationship, and use the gating mechanism to filter and fuse the multimodal features, and then obtain the sentiment classification result through the output layer; The cross entropy loss function is used to calculate the model loss, optimize the training network parameters, and obtain the trained multimodal sentiment analysis model; Step 1 includes: Step 1.1, encode the text modality: for each sample text S, use the flag to replace the aspect word sequence in the text, and connect the replaced text S with the aspect word sequence A as the text input T′, and use the text pre-training model RoBERTa to obtain the text feature representation F T : F T =RoBERTa(T′) in d is the hidden dimension of the text feature, n is the input length, represents the real number space; Step 1.2, encode the image: For the input image I, use the fast regional convolutional neural network Faster R-CNN to detect object proposals, sort the proposals according to the object category detection probability, and retain the regional representations of the first X1 object proposals as the regional feature representation V of image I: V=Faster R-CNN(I) in l is the hidden dimension of the regional feature representation V, and m is the number of regions. The regional feature representation V is converted to the same hidden dimension d as the text feature through a linear layer to achieve the alignment of text and visual features in dimension, and then the object-level regional visual feature F is obtained through the Transformer model. V : F V =Transformer(W r V+b r ) in W r and b r is a learnable parameter; Step 2 includes: Step 2.1: Use the cross-modal Transformer layer to represent the text features F T as the query and the region-level visual feature F V As keys and values, like this: F′ V =CrossModalTransformer(F T ,F V ,F V ) in It is the region-level visual feature associated with each word in the text; The cross-modal Transformer layer is specifically implemented as follows: Q=W q F T K=W k F V V′=W V F V Where Q, K, and V′ are respectively the query vector, key vector, and value vector after linear transformation; W q , W k , W V is a learnable parameter; k1 is a scaling factor, Cross Modal Transformer represents cross-modal operation; T represents matrix transpose; For each word in the text, the region-level visual feature F′ is V Use the maximum pooling operation to get f′ V , to obtain the most important features for relationship classification; based on f′ V , use the Sigmoid function to get the predicted probability of the image-text relationship. The specific formula is: f′ V =max-pooling(F′ V ) P(c)=Sigmoid(W c f′ V +b c ) where f′ V represents the most significant features related to the image-text relationship task, max-pooling is the maximum pooling operation; c represents the category of the image-text relationship, P(c) is the probability that the model predicts that the image-text relationship of the sample is of category c, P(c) is a scalar in the range [0,1], W c and b c is a learnable parameter; Use cross entropy loss to optimize the image-text relationship task: in is the loss function of the image-text relationship task, M is the number of samples in the multimodal dataset, and c u represents the actual category of the image-text relationship classification of the u-th sample, P(c u ) indicates that the model predicts that the image-text relationship category of the u-th sample is c u probability; Construct a visual mask matrix using the image-text relationship prediction probability score P(c) All elements in the visual mask matrix R are equal to the predicted probability score P(c) of the image-text relationship. The denoised regional-level visual features F″ are obtained through the visual mask matrix R. V : F″ V =R⊙F′ V in ⊙ is element-by-element multiplication.

2. The method according to claim 1, characterized in that Step 3 includes: Step 3.1: Given the denoised region-level visual feature set F″ V = {v1…v n } and text feature representation set F T ={t1…t n }, where v n represents the nth region-level visual feature after denoising, t n Represents the features of the nth word after the text is segmented; calculates the regional visual and word-level text attention matrix U, and scales and normalizes the attention matrix U to the regional word attention matrix Using region-word attention matrix Aggregate all word features matched for each visual area, the formula is: U=(W v F″ V )(W t F T ) T Where W v and W t is a learnable parameter, U ij represents the relationship between the i-th visual area and the j-th word, k2 is the scaling factor, matrix The i-th row of represents the text features matched by the i-th visual area; Step 3.2, process the extracted matrix H as follows: O=(W h1 H)(W h2 h) Where W h1 and W h2 is a learnable parameter; O represents the relationship score matrix between region word pairs, represents the matrix after the matrix O is scaled and normalized, k3 is the scaling factor, Represents the regional word pair features after further extraction.

3. The method according to claim 2, characterized in that Step 4 includes: Step 4.1, given the denoised region-level visual feature F″ V = {v1…v n } and regional word pair features G = {g1…g n }, calculate the adaptive gate to evaluate the degree of alignment: n i =sigmoid(v i ⊙g i ) yes i =b i ⊙[v i ,g i ] d i JReLU(W d e i ) f i =d i +v i where g i represents the word pair feature of the i-th region, v i represents the i-th region-level visual feature, b i Represents the measure of regional word pair features g i Adaptive gating value of alignment degree; e i is the fused feature; W d is a learnable parameter, ReLU is a linear rectification function; finally, the fused feature F = {f1,…,f n }, f i represents the word pair feature of the ith region after correction and fusion, The alignment information between regions and word pairs is implied.

4. The method according to claim 3, characterized in that Step 5 includes: Step 5.1: The denoised region-level visual feature F″ V , text feature representation F T Connect them with the feature F obtained in step 4 respectively, and pass the connected features through the fully connected layer to obtain the linearly transformed features. The formula is: Z1=ReLU(W1[F⊕F″ V ]+b1) <h2 style=";text-align:left;direction:ltr">Z2 = ReLU(W2[F⊕F<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> ]+b2) Where Z1 represents the region-level visual feature F″ V The fusion feature after combining with F, Z2 represents the text feature F T The fusion features after combining with F, W1, W2, b1, b2 are learnable parameters, and ⊕ represents the connection operation; In the first case, if the predicted probability of the image-text relationship indicates that the probability of no relationship between the image and the text is greater than the probability of a relationship, the relationship matrix of Z1 and Z2 is calculated first, and then Z2 supplemented with text features plus Z1 multiplied by the relationship matrix is ​​used as the final feature set: α=sigmoid(W3Z1+W4Z2) F1=Z2+α⊙Z1 Where α represents the relationship matrix between Z1 and Z2 when the image and text have no relationship; F1 represents the final feature set after fusion when the image and text have no relationship, and W3 and W4 are learnable parameters; In the second case, if the predicted probability of the image-text relationship indicates that the probability of the image and text being related is greater than the probability of being unrelated, the relationship matrix of Z1 and Z2 is calculated first, and then Z1 supplemented by the region-level visual features plus Z2 multiplied by the relationship matrix are used as the final feature set: β=sigmoid(W5Z1+W6Z2) F2=Z1+β⊙Z2 Where β represents the relationship matrix of Z1 and Z2 when the image and text are related, F2 represents the final feature set after fusion when the image and text are related, and W5 and W6 are learnable parameters; Step 5.2: The final feature set F1 or F2 extracted in step 5.1 is regarded as the feature set Z, and the feature set Z is input into the softmax layer to predict the final emotion: P(y)=Softmax(W y Z+b) Where P(y) is the probability of predicting the emotion category as y, W y , b is a learnable parameter; Use cross entropy loss to optimize the multimodal sentiment analysis task based on aspect words, which is achieved through sentiment supervision representation: in is the loss function of the multimodal aspect-level sentiment classification task, N is the number of samples in the multimodal dataset, and y q represents the actual category of the qth sample sentiment classification, P(y q ) indicates that the model predicts that the sentiment classification category of the qth sample is y q probability; Then we get the final loss function 5. The method according to claim 4, characterized in that In step 5.2, the final loss function It is expressed as: Where λ1 is a hyperparameter.

6. A multimodal aspect-level sentiment analysis system implemented based on the method according to any one of claims 1 to 5, characterized in that: include: The image-text relationship module obtains a multimodal data set, where each sample in the multimodal data set includes a set of text, a related image, and an aspect word sequence, wherein the aspect word sequence is a subsequence of the text; the aspect word sequence in the text is replaced by a flag, and the replaced text is connected with the aspect word sequence, and encoded using RoBERTa, an optimized version of BERT, to obtain text feature representation; in terms of image feature encoding, the fast regional convolutional neural network Faster R-CNN is used to obtain the regional feature representation of the image, and then the regional feature representation of the image is passed through the Transformer model to enhance the modeling between regional visual objects, thereby obtaining the regional level visual features of the image; The interaction between text feature representation and region-level visual features is modeled through a cross-modal Transformer layer. The text features and region-level visual features are combined to obtain the image-text relationship. Then, a visual filter matrix is ​​generated to denoise the region-level visual features. The cross-modal alignment correction module, based on the cross-modal attention mechanism, performs word-level attention operations on the denoised region-level visual features to identify the text information that best matches each visual region and aligns the image region with the text words in the feature embedding space; The feature fusion module combines global information to supplement the features obtained by the cross-modal alignment correction module, selects the supplemented features according to the image-text relationship, and uses the gating mechanism to filter and fuse the multimodal features, and then obtains the sentiment classification results through the output layer.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 5 are executed.

Images