Multi-modal aspect-level sentiment analysis method, system and equipment combined with dual-granularity image information and medium

Abstract

The invention relates to the technical field of multi-modal aspect-level sentiment analysis, and discloses a multi-modal aspect-level sentiment analysis method, system and equipment combined with dual-granularity image information and a medium. By introducing dual-granularity image features and combining a cross-modal multi-head attention mechanism, deep interaction between an image and a text is remarkably enhanced, and the multi-modal aspect-level sentiment analysis efficiency is improved. Especially, before modal fusion, the precision of feature alignment is effectively improved by adopting a pairwise modal semantic enhancement strategy, and compared with an existing method, the method not only has breakthrough progress in utilization of fine-grained features of the image, but also realizes improvement in deep fusion of multi-modal information, and the method is suitable for popularization and application. And a new solution thought and method are provided for multi-modal sentiment analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal aspect-level sentiment analysis, and in particular to a multimodal aspect-level sentiment analysis method, system, device and medium combining dual-granularity image information. Background Art

[0002] Multimodal Aspect-Based Sentiment Analysis (MABSA) is an important fine-grained sentiment analysis task of multimodal sentiment analysis. Multimodal data is one of the key elements of MABSA tasks, but there are often some complex image-text pairs in the dataset, which brings challenges to fine-grained sentiment analysis tasks. In addition, the existing methods for obtaining fine-grained image information still have many shortcomings, such as the image information obtained is not fine enough, etc. In addition, most of the existing methods for implementing MABSA tasks rely on pre-trained single-modal models to extract modal features. For example, models such as RoBERTa and BERT are used to obtain text features, and models such as ResNet and R-CNN are used to obtain image features. Due to the lack of interaction between modalities, there is a problem of misaligned modal data, which makes the model fail to fully utilize the association information between text and image to assist in improving the performance of MABSA tasks.

[0003] Therefore, there is an urgent need for a new multimodal aspect-level sentiment analysis method, a multimodal aspect-level sentiment analysis method that combines dual-granularity image information, so as to make full use of the coarse-grained and fine-grained information of the image, thereby improving the precision and accuracy of multimodal aspect-level sentiment analysis. Summary of the invention

[0004] The present invention provides a multimodal aspect-level sentiment analysis method, system, device and medium combined with dual-granularity image information, so as to solve the defects of poor precision and low accuracy of the existing multimodal aspect-level sentiment analysis methods.

[0005] The present invention provides a method for constructing a multimodal aspect-level sentiment analysis model, comprising:

[0006] Acquire image sample data and text sample data, wherein each text in the text sample data is annotated with aspect words representing sentiment polarity to form annotated aspect word data;

[0007] According to the image sample data, text sample data, and annotated aspect word data, the image sample features, the spliced ​​text features, and the annotated aspect word features are obtained through the MIEF network;

[0008] According to the image sample features and the concatenated text features, the predicted aspect word data is obtained through the AE network;

[0009] According to the predicted aspect word data, image sample features, spliced ​​text features, and annotated aspect word features, the first multimodal fusion data and the second multimodal fusion data are obtained through the MDFF network by using the first cross-modal multi-head attention mechanism and the second cross-modal multi-head attention mechanism;

[0010] According to the first multimodal fusion data and the second multimodal fusion data, a multimodal aspect-level sentiment analysis model is trained by utilizing the first attention mechanism and the second attention mechanism.

[0011] According to a method for constructing a multimodal aspect-level sentiment analysis model provided by the present invention, the image sample features, the spliced ​​text features, and the annotated aspect word features are obtained through the MIEF network based on the image sample data, the text sample data, and the annotated aspect word data, including:

[0012] According to the image sample data, several image region features whose confidences meet the preset requirements are obtained through the Faster R-CNN network as image sample features;

[0013] Based on the image sample data, the coarse-grained features in the image are used through the ClipCap network to generate text data;

[0014] The generated text data and the text sample data are concatenated and passed through the Text Embedding network to obtain the concatenated text features;

[0015] According to the labeled aspect word data, the labeled aspect word features are obtained through the Aspect Embedding network.

[0016] According to a method for constructing a multimodal aspect-level sentiment analysis model provided by the present invention, the method obtains predicted aspect word data through an AE network based on image sample features and concatenated text features, including:

[0017] According to the image sample features and the concatenated text features, the encoding features are obtained through the BART encoder of the AE network combined with the text mask strategy;

[0018] According to the encoding features, the BART decoder of the AE network is used to set the aspect word extraction task as a special word segmentation index generation task to obtain the predicted aspect word data;

[0019] Among them, the BART encoder is a multi-layer bidirectional Transformer network, and the BART decoder is a multi-layer unidirectional Transformer network.

[0020] According to a method for constructing a multimodal aspect-level sentiment analysis model provided by the present invention, the text masking strategy is: randomly masking the word segmentation of the input text with a probability of 15%, so that the image and the masked text can regenerate the original text, and the loss function of the text masking strategy is:

[0021]

[0022] In the loss function of the text mask strategy, e i and denote the i-th word segmentation and mask multimodal input of the concatenated text features, respectively, and e ＜i represents the context information before the current time step i, P(·) represents the predicted probability distribution, and E M～D Represents the expectation of the multimodal data M on the entire dataset D, which is used to calculate the average loss, and K represents the length of the concatenated text feature.

[0023] According to a method for constructing a multimodal aspect-level sentiment analysis model provided by the present invention, the expression of the BART decoder is:

[0024]

[0025] In the expression of BART decoder, H e represents the feature output of the BART encoder, i.e., the encoding feature, l m = l img +l txt +l cap +l token , l img Represents the length of the image feature sequence, l token represents the length of a special word, d represents the hidden dimension, l txt Represents the length of the text feature sequence of the text sample data, l cap Indicates the length of the text feature sequence that generates text data, Represents a masked multimodal input.

[0026] According to a method for constructing a multimodal aspect-level sentiment analysis model provided by the present invention, the expression of the BART decoder is:

[0027]

[0028]

[0029]

[0030] In the expression of BART decoder, Indicates H e The text part, H erepresents the characteristic output of the BART encoder, E t Represents the embedding vector of a special word, P(a t ) represents the predicted word segmentation probability distribution, represents the feature output of the BART decoder before time step t, C d Special word indicating the end of decoding <eot>The embedding vector of .

[0031] According to a method for constructing a multimodal aspect-level sentiment analysis model provided by the present invention, the loss function of the aspect word extraction task is:

[0032]

[0033] In the loss function of the aspect word extraction task, represents the feature output of the BART decoder before time step t, A J Represents the generated aspect word sequence, whose length is O=2n+1, where 2n represents the number of start and end indexes of the aspect word, and 1 represents a special participle <eot>The number of t represents the feature output of the current time step t, E M～D Represents the expectation of the multimodal data M on the entire dataset D, which is used to calculate the average loss.

[0034] According to a method for constructing a multimodal aspect-level sentiment analysis model provided by the present invention, the first multimodal fusion data and the second multimodal fusion data are obtained by using a first cross-modal multi-head attention mechanism and a second cross-modal multi-head attention mechanism through an MDFF network according to predicted aspect word data, image sample features, spliced ​​text features, and annotated aspect word features, including:

[0035] According to the predicted aspect word data, image sample features, and concatenated text features, the first multimodal fusion data is obtained through the MDFF network and the first cross-modal multi-head attention mechanism;

[0036] According to the image sample features, spliced ​​text features, and annotated word features, the second multimodal fusion data is obtained through the MDFF network and the second cross-modal multi-head attention mechanism.

[0037] According to a method for constructing a multimodal aspect-level sentiment analysis model provided by the present invention, the first multimodal fusion data is obtained by using a first cross-modal multi-head attention mechanism through an MDFF network according to predicted aspect word data, image sample features, and spliced ​​text features, including:

[0038] Masking the concatenated text features using the predicted aspect word data to obtain a first masked text feature sequence;

[0039] Perform layer normalization and linear transformation on the first masked text feature sequence to obtain the query of the first cross-modal multi-head attention mechanism;

[0040] Perform layer normalization and linear transformation on the image sample features to obtain the keys and values ​​of the first cross-modal multi-head attention mechanism;

[0041] According to the query, key and value of the first cross-modal multi-head attention mechanism, the fusion of the image modality and the text modality is repeatedly deepened in combination with the first cross-modal multi-head attention mechanism and the first feedforward neural network to obtain the first multimodal fusion data.

[0042] According to a method for constructing a multimodal aspect-level sentiment analysis model provided by the present invention, the second multimodal fusion data is obtained by using a second cross-modal multi-head attention mechanism through an MDFF network according to image sample features, spliced ​​text features, and annotated aspect word features, including:

[0043] Using the annotated aspect word features to mask the concatenated text features, obtaining a second masked text feature sequence;

[0044] Perform layer normalization and linear transformation on the second mask text feature sequence to obtain the query of the second cross-modal multi-head attention mechanism;

[0045] Perform layer normalization and linear transformation on the image sample features to obtain the keys and values ​​of the second cross-modal multi-head attention mechanism;

[0046] According to the query, key and value of the second cross-modal multi-head attention mechanism, the second cross-modal multi-head attention mechanism and the second feedforward neural network are combined to repeatedly deepen the fusion of the image modality and the text modality to obtain the second multimodal fusion data.

[0047] According to a method for constructing a multimodal aspect-level sentiment analysis model provided by the present invention, the multimodal aspect-level sentiment analysis model is trained based on the first multimodal fusion data and the second multimodal fusion data using the first attention mechanism and the second attention mechanism, comprising:

[0048] The predicted aspect word data is used as the query of the first attention mechanism, and the first multimodal fusion data is used as the key and value of the first attention mechanism. Through the first attention mechanism, the sentiment polarity prediction result based on the predicted aspect word is obtained;

[0049] The features of the annotated aspect words are used as the query of the second attention mechanism, and the second multimodal fusion data is used as the key and value of the second attention mechanism. Through the second attention mechanism, the sentiment polarity prediction result based on the annotated aspect words is obtained;

[0050] Combining the sentiment polarity prediction results based on predicted aspect words and the sentiment polarity prediction results based on labeled aspect words, a multimodal aspect-level sentiment analysis model is trained using the cross entropy function.

[0051] According to a method for constructing a multimodal aspect-level sentiment analysis model provided by the present invention, the cross entropy function is:

[0052]

[0053] In the cross entropy function, k represents the number of aspect words, Y i C represents the true label of the i-th aspect word, is the predicted probability of the i-th aspect word.

[0054] The present invention also provides a multimodal aspect-level sentiment analysis method, comprising:

[0055] receiving images and text to be analyzed;

[0056] According to the image and text to be analyzed, the multimodal aspect-level sentiment analysis model obtained by the construction method of the multimodal aspect-level sentiment analysis model described in any of the above items obtains the sentiment polarity prediction results of the image and text to be analyzed.

[0057] The present invention also provides a multimodal aspect-level sentiment analysis system, comprising:

[0058] The data receiving module is used to: receive images and texts to be analyzed;

[0059] The prediction module is used to obtain the sentiment polarity prediction results of the images and texts to be analyzed based on the multimodal aspect-level sentiment analysis model obtained by the construction method of the multimodal aspect-level sentiment analysis model described in any of the above items.

[0060] The present invention also provides an electronic device, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, the method for constructing a multimodal aspect-level sentiment analysis model and / or the multimodal aspect-level sentiment analysis method described above is implemented.

[0061] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for constructing a multimodal aspect-level sentiment analysis model and / or the multimodal aspect-level sentiment analysis method.

[0062] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute any of the above-mentioned methods for constructing a multimodal aspect-level sentiment analysis model and / or the multimodal aspect-level sentiment analysis method.

[0063] The present invention provides a multimodal aspect-level sentiment analysis method, system, device and medium combining dual-granularity image information, which can bring at least the following beneficial effects:

[0064] In the multimodal aspect word extraction task, in order to enhance the interaction between the image modality and the text modality, the ClipCap network is used to generate coarse-grained feature description text of the image as image prompt information, which effectively improves the precision and accuracy of the model in predicting aspect words and their attributes in the text;

[0065] In multimodal word sentiment classification, in order to capture rich fine-grained sentiment features of images, the cross-modal multi-head attention mechanism is used to deeply interact the underlying features of the image with the original sentiment semantics with the masked text, effectively strengthening the fusion of image features to text features.

[0066] The present invention significantly enhances the deep interaction between images and texts by introducing dual-granularity image features and combining them with a cross-modal multi-head attention mechanism. In particular, before modal fusion, a pairwise modal semantic enhancement strategy is adopted to effectively improve the accuracy of feature alignment. Compared with existing methods, the present invention not only makes breakthrough progress in the utilization of fine-grained image features, but also achieves improvement in the deep fusion of multimodal information, providing new solutions and methods for multimodal aspect-level sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 A flowchart of a multimodal aspect-level sentiment analysis method combining dual-granularity image information provided by the present invention.

[0069] Figure 2 Schematic diagram of the structure of the multimodal aspect-level sentiment analysis model.

[0070] Figure 3 A schematic diagram of generating text based on image samples through the ClipCap network.

[0071] Figure 4 Schematic diagram of part of the structure of the AE network.

[0072] Figure 5 Schematic diagram of the principle of cross-modal multi-head attention mechanism.

[0073] Figure 6 A structural schematic diagram of a multimodal aspect-level sentiment analysis system combining dual-granularity image information provided by the present invention.

[0074] Figure 7 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0076] Figure 1 A flowchart of a multimodal aspect-level sentiment analysis method for combined dual-granularity image information provided by the present invention. The execution subject of the multimodal aspect-level sentiment analysis method for combined dual-granularity image information provided by the present invention can be any applicable terminal side device or network side device, such as a multimodal aspect-level sentiment analysis device for combined dual-granularity image information.

[0077] The present invention provides a multimodal aspect-level sentiment analysis method combining dual-granularity image information, which includes two parts: construction and application of a multimodal aspect-level sentiment analysis model.

[0078] Figure 2 A structural schematic diagram of the multimodal aspect-level sentiment analysis model in this embodiment is shown. The multimodal aspect-level sentiment analysis model in this embodiment includes four parts: multimodal data input and feature extraction (Multimodal data Input and Feature Extraction, MIFE), aspect term extraction (Aspect term Extraction, AE), multimodal data feature fusion (Multimodal Data Feature Fusion, MDFF) and sentiment classification (Sentiment classification, SC). Among them, the MIFE part is used to obtain the embedding of image features and text features, AE is the task of extracting aspect words in text, the MDFF part aims to repeatedly deepen the fusion of image and text modalities through multi-layer cross-modal multi-head attention, and the SC part predicts the sentiment polarity of aspect words in JMASA and MASC tasks through attention mechanism and softmax function.

[0079] Regarding the construction of a multimodal aspect-level sentiment analysis model, the method steps may include:

[0080] S110, acquiring image sample data and text sample data, wherein each text in the text sample data is annotated with aspect words representing sentiment polarity to form annotated aspect word data, and the sentiment polarity may include positive, negative, and neutral;

[0081] S120, obtaining image sample features, concatenated text features, and annotated aspect word features through the MIEF network according to the image sample data, the text sample data, and the annotated aspect word data;

[0082] S130, obtaining predicted aspect word data through an AE network according to the image sample features and the concatenated text features;

[0083] S140, according to the predicted aspect word data, the image sample features, the spliced ​​text features, and the annotated aspect word features, through the MDFF network, using the first cross-modal multi-head attention mechanism and the second cross-modal multi-head attention mechanism, obtain the first multimodal fusion data and the second multimodal fusion data;

[0084] S150. According to the first multimodal fusion data and the second multimodal fusion data, a multimodal aspect-level sentiment analysis model is trained using a first attention mechanism and a second attention mechanism.

[0085] In one embodiment, S120 may include:

[0086] S1201. According to the image sample data, a plurality of image region features whose confidences meet preset requirements are obtained through the Faster R-CNN network as image sample features;

[0087] Among them, this embodiment is based on the excellent image feature extraction capability of the Faster R-CNN network. The input image sample is preprocessed by Faster R-CNN, the features of all candidate regions are extracted from the image sample, and only the 36 regions with the highest confidence are retained. At the same time, the semantic category of each region is retained. By retaining the regions with higher confidence, the interference from low-quality regions (which may contain noise or irrelevant information) in the model during sentiment analysis can be reduced. Specifically, this embodiment uses mean pooling convolution features as the input image features, and uses E V ={v1,...,v 36 } represents the obtained image features, where v i ∈R 2048 represents the image features of the i-th region. To be consistent with the text representation, this embodiment uses a batch of linear transformation layers to project the image features onto a d-dimensional vector. The formula is as follows:

[0088] E I =W I E v +b,E I ∈R d×36 (1)

[0089] Where W I is a d×2048 weight matrix;

[0090] This embodiment also adds and special participles indicate the beginning and end of image features, and obtain image embedding I E =[ ,E I ,];

[0091] S1202: Based on the image sample data, the coarse-grained features in the image are used through the ClipCap network to generate text data. For details, see Figure 3 In order to enhance the relevance between images and texts and further improve the accuracy of aspect word recognition, this embodiment uses the ClipCap network to generate image description text from the coarse-grained features of image samples to form generated text data, denoted as C:

[0092] C=ClipCap(V) (2)

[0093] S1203, concatenate the generated text data and the text sample data, and obtain the concatenated text features through the Text Embedding network. Specifically, in order to effectively map the text to a high-dimensional semantic space to capture its deep features, this embodiment uses a BART dedicated word segmenter to segment the text data, converts the word segmentation into an index in the vocabulary, and then uses an embedding matrix to convert the word segmentation index into an embedding vector. In order to distinguish between the input text and the image description text, this embodiment adds <txt> 、< / txt> and <cap> 、< / cap> Special word segmentation is used to indicate the beginning and end of the embedding of the input text (i.e., text sample data) and the image description text (i.e., generated text data), as shown below:

[0094]

[0095]

[0096] Where T is the input text, l txt is the length of the input text feature sequence, l cap is the length of the image description text feature sequence.

[0097] In this embodiment, E C With E T Concatenate to get text embedding T E =[ <txt> ,E T ,< / txt> , <cap> ,E C ,< / cap> ];

[0098] S1204 , obtaining the labeled aspect word features through the Aspect Embedding network according to the labeled aspect word data.

[0099] In one embodiment, S130 may include:

[0100] According to the image sample features and the concatenated text features, the encoding features are obtained through the BART encoder of the AE network combined with the text mask strategy;

[0101] According to the encoding features, the BART decoder of the AE network is used to set the aspect word extraction task as a special word segmentation index generation task to obtain the predicted aspect word data;

[0102] Among them, the BART encoder is a multi-layer bidirectional Transformer network, and the BART decoder is a multi-layer unidirectional Transformer network.

[0103] Specifically, this embodiment obtains the embedding M of multimodal data by splicing image and text features. E ,T E ], when inputting text, a text masking strategy is adopted, that is, the word segmentation of the input text is randomly masked with a probability of 15%, so that the image and masked text can regenerate the original text, which is used to build the consistency of text and image features.

[0104] Among them, the loss function of the text mask strategy is:

[0105]

[0106] In the loss function of the text mask strategy, e i and denote the i-th word segmentation and mask multimodal input of the concatenated text features, respectively, and e ＜i represents the context information before the current time step i, P(·) represents the predicted probability distribution, and E M～D Represents the expectation of the multimodal data M on the entire dataset D, which is used to calculate the average loss, and K represents the length of the concatenated text feature.

[0107] See also Figure 4 , this embodiment will As the input of the BART encoder, Figure 4 As shown, the characteristic output of the multimodal encoder is obtained:

[0108]

[0109] Among them, l m = l img +l txt +l cap +l token , l img is the length of the image feature sequence, l token is the length of the special word, and d is the hidden dimension.

[0110] This embodiment inserts <bot>The word segmentation indicates the start of decoding generation, and inserts <aot>Segmentation is used to represent the aspect word extraction task. <eot>The end of decoding is indicated by the word segmentation. This embodiment also sets the aspect word extraction task as a special word segmentation index generation task. Given an input text as a source text sequence, the goal is to generate an aspect word index sequence, which consists of the start and end indexes of all aspect word segmentations, that is, in They represent the starting index and ending index of the i-th aspect word respectively, and n represents the number of aspect words.

[0111] Here, this embodiment assumes that Indicates H e The decoder outputs H with the multimodal encoder e and the output D before the decoder ＜t As the input of the decoder, it predicts the word segmentation probability distribution P(a t ).

[0112] Among them, the expression of BART decoder is:

[0113]

[0114] In the expression of BART decoder, H e represents the feature output of the BART encoder, i.e., the encoding feature, l m = l img +l txt +l cap +l token , l img Represents the length of the image feature sequence, l token represents the length of a special word, d represents the hidden dimension, l txt Represents the length of the text feature sequence of the text sample data, l cap Indicates the length of the text feature sequence that generates text data, Represents a masked multimodal input.

[0115] The expression of the BART decoder is:

[0116]

[0117]

[0118]

[0119] In the expression of BART decoder, Indicates H e The text part, H e represents the characteristic output of the BART encoder, E t represents the embedding of word segmentation, P(a t ) represents the predicted word segmentation probability distribution, represents the feature output of the BART decoder before time step t, C d Special end participle indicating the end of decoding <eot>The embedding vector of .

[0120] The loss function of the aspect word extraction task is:

[0121]

[0122] In the loss function of the aspect word extraction task, A J Represents the generated aspect word sequence, whose length is O=2n+1, where 2n represents the number of start and end indexes of the aspect word, and 1 represents a special participle <eot>The number of t represents the feature output of the current time step t, E M～D Represents the expectation of the multimodal data M on the entire dataset D, which is used to calculate the average loss.

[0123] In one embodiment, S140 may include:

[0124] According to the predicted aspect word data, image sample features, and concatenated text features, the first multimodal fusion data is obtained through the MDFF network and the first cross-modal multi-head attention mechanism;

[0125] According to the image sample features, spliced ​​text features, and annotated word features, the second multimodal fusion data is obtained through the MDFF network and the second cross-modal multi-head attention mechanism.

[0126] In order to improve the accuracy of multimodal sentiment analysis and reduce the interference of irrelevant areas. This embodiment adopts multi-layer cross-modal multi-head attention dominated by text to effectively capture and align fine-grained features between images and text. In the multi-layer cross-modal attention architecture, the attention mechanism of each layer can gradually refine information, so that the model can selectively focus on key information in different modalities in each layer, thereby filtering out irrelevant and noisy information layer by layer.

[0127] Repeated multi-layer cross-modal attention not only helps to improve the robustness of the model to noise, but also this embodiment uses the parallel processing capabilities of multi-head attention to better handle modal differences and achieve deep fusion of information, thereby obtaining stronger expressive capabilities. In order to accurately obtain the sentiment information related to aspect words, this embodiment masks the aspect words in the text. Subsequently, the image information and the masked text information are fused to obtain more comprehensive aspect word sentiment information. By masking the aspect word mask, the model needs to rely more on the image modality to fill in the missing semantic gaps, thereby better integrating information from different modalities.

[0128] Since the underlying features of non-text information are conducive to the model retaining its original sentiment semantics, such as Figure 5 As shown, this embodiment adopts an L-layer cross-modal multi-head attention (CMA) structure to achieve potential cross-modal adaptation. The multi-head attention mechanism can capture fine-grained interaction information between different modalities in parallel in multiple attention heads. Each layer of CMA first calculates the correlation weights between text and image features, thereby selectively incorporating image features into the text representation. Through repeated cross-modal attention mechanisms, the model can not only make full use of image information to supplement the implicit emotional information in the text, but also strengthen the alignment of images and texts at different levels to generate more accurate emotional representations.

[0129] In this embodiment, the structures and principles of the first cross-modal multi-head attention mechanism and the second cross-modal multi-head attention mechanism can be consistent. Under the first cross-modal multi-head attention mechanism, the concatenated text features are masked using the predicted aspect word data to obtain a first masked text feature sequence; the first masked text feature sequence is layer-normalized and linearly transformed to obtain the query of the first cross-modal multi-head attention mechanism; the image sample features are layer-normalized and linearly transformed to obtain the keys and values ​​of the first cross-modal multi-head attention mechanism; according to the query, key and value of the first cross-modal multi-head attention mechanism, the fusion of the image modality and the text modality is repeatedly deepened in combination with the first cross-modal multi-head attention mechanism and the first feedforward neural network to obtain the first multi-modal fusion data. Similarly, under the second cross-modal multi-head attention mechanism, the concatenated text features are masked using the annotated aspect word features to obtain a second masked text feature sequence; the second masked text feature sequence is layer-normalized and linearly transformed to obtain the query of the second cross-modal multi-head attention mechanism; the image sample features are layer-normalized and linearly transformed to obtain the keys and values ​​of the second cross-modal multi-head attention mechanism; based on the query, keys and values ​​of the second cross-modal multi-head attention mechanism, the second cross-modal multi-head attention mechanism and the second feedforward neural network are combined to repeatedly deepen the fusion of the image modality and the text modality to obtain the second multimodal fusion data.

[0130] See also Figure 5 In order to obtain the multimodal fusion feature data of the JMASA task, this embodiment uses the aspect word A predicted by the AE part J For the text feature sequence T E Mask it, and then use the masked text feature sequence After layer normalization and linear transformation, the image feature sequence I is used as the query Q of cross-modal multi-head attention. E After layer normalization and linear transformation, the key K and value V of the cross-modal multi-head attention are as follows:

[0131]

[0132]

[0133]

[0134] in represents the weight matrix; i represents the number of layers currently fused, Represents the text feature sequence of the i-th layer, when i=1 LN(·) represents layer normalization.

[0135] After obtaining the query Q, key K, and value V sequence, this embodiment uses cross-modal multi-head attention to integrate image information into text:

[0136]

[0137] in represents the text sequence after image enhancement, i represents the number of layers currently fused, l t = l txt +l cap Indicates the length of the text feature sequence.

[0138] After the above CMA, each word vector in the text sequence can receive information from all elements in the image sequence, and then the enhanced text sequence is concatenated with the text sequence before enhancement:

[0139]

[0140] This embodiment allows F i After layer normalization, a feedforward neural network is performed, and then a residual connection is performed to obtain the text output after the layer is strengthened. The formula is as follows:

[0141]

[0142] where f(·) represents a feedforward neural network, f θ Represents the parameters of a feedforward neural network.

[0143] After L layers of fusion, the final representation of the image information injected into the text is obtained, that is, the fused multimodal feature sequence

[0144] For the MASC task, this embodiment adopts the same method to mask the text feature sequence of the annotated aspect words. and image feature sequence I E As input, we get the corresponding multimodal fusion feature sequence

[0145] It should be noted that in the multi-layer cross-modal attention, the image features of each layer are fixed pre-trained visual features extracted by Faster R-CNN. These features can be regarded as the underlying representation of the image, and the text features of each layer are the fused features of the previous layer.

[0146] In one embodiment, S150 may include:

[0147] The predicted aspect word data is used as the query of the first attention mechanism, and the first multimodal fusion data is used as the key and value of the first attention mechanism. Through the first attention mechanism, the sentiment polarity prediction result based on the predicted aspect word is obtained;

[0148] The features of the annotated aspect words are used as the query of the second attention mechanism, and the second multimodal fusion data is used as the key and value of the second attention mechanism. Through the second attention mechanism, the sentiment polarity prediction result based on the annotated aspect words is obtained;

[0149] The sentiment polarity prediction results based on predicted aspect words and the sentiment polarity prediction results based on labeled aspect words are combined with the cross entropy function to train a multimodal aspect-level sentiment analysis model.

[0150] Specifically, in form, this embodiment makes A C Represented as a feature sequence of aspect words, C∈{J,M}, where A J A represents the feature sequence of aspect words extracted from the AE part, M Represents the feature sequence of the aspect words of the annotation.

[0151] Since the attention mechanism can dynamically adjust the degree of attention to feature information according to the different aspect words, in order to better capture the sentiment characteristics of each aspect word, this embodiment will be aspect word A C Considered as query Q, multimodal fusion feature sequence As the key K and value V in the attention mechanism, the attention mechanism is used to obtain the sentiment polarity of the aspect word. The formula is as follows:

[0152]

[0153]

[0154]

[0155] S C =softmax(Att C (Q C ,K C ,V C )) (twenty one) in Represents the weight matrix of sentiment classification, S C It is the predicted output of the model, which represents the probability distribution of the model's sentiment tendency for various aspects of words.

[0156] In order to measure the emotional label distribution S predicted by the model C The difference between the distribution of the real sentiment labels and the actual distribution of the sentiment labels is evaluated using the cross entropy function in this example:

[0157]

[0158] Among them, k is the number of aspect words, Y i C is the true label of the i-th aspect word, is the predicted probability of the i-th aspect word.

[0159] After obtaining the multimodal aspect-level sentiment analysis model through the above steps, the image and text to be analyzed can be input into the multimodal aspect-level sentiment analysis model to obtain the aspect-level sentiment polarity prediction results of the image and text to be analyzed, such as the probability that the aspect-level sentiment polarity of the image and text to be analyzed is close to positive, negative or neutral.

[0160] On the other hand, in order to evaluate the performance of the multimodal aspect-level sentiment analysis model obtained in this embodiment, this embodiment adopts two annotated benchmark datasets, namely, Twitter-2015 and Twitter-2017, two multimodal datasets, where each sample contains a Twitter text, a corresponding picture, a labeled aspect face and its corresponding sentiment polarity label.

[0161] The statistical information of the two datasets is shown in Table 1. There are three types of sentiment polarity labels: Positive, Neutral, and Negative.

[0162] Table 1 Statistics of the dataset

[0163]

[0164] Among them, ML represents the maximum length of a text sentence.

[0165] This embodiment uses F1 score (F1), precision (P) and recall (R) as evaluation indicators to measure the performance of the model for the three subtasks of MABSA. For the MASC task, in order to compare fairly with other methods, this embodiment also uses accuracy (ACC).

[0166] All models in this embodiment are implemented based on Pytorch, and the experiments are run on the NVIDIA RTX 3090 GPU. In order to verify the performance of the model and accurately evaluate the experimental results, this embodiment conducted repeated experiments on two public datasets, Twitter-2015 and Twitter-2017, to obtain the optimal parameter combination for the subtasks. The results are shown in Table 2.

[0167] Table 2 Experimental parameter settings

[0168]

[0169] This example compares the obtained multimodal aspect-level sentiment analysis model with the baseline models in the three subtasks listed below.

[0170] 1. MATE task baseline model (UMT, RAN, OSCGA, AOM).

[0171] 2. Baseline models for the MASC task (TomBERT, ESAFN, CapTrBERT, IAMFN, HF-EKCL).

[0172] 3. JMASA task baseline models (UMT+TomBERT and OSCGA+TomBERT; UMT-collapse and OSCGA-collaspe; JML; VLP-MABSA).

[0173] 1. MATE result analysis

[0174] According to the results in Table 3, this embodiment believes that the model trained in this embodiment can better extract and fuse the coarse-grained information of the image, and help improve various evaluation indicators of the aspect word extraction task.

[0175] Table 3 Experimental results of MATE tasks

[0176]

[0177] 2. Analysis of MASC results

[0178] From the results in Table 4, the model of this embodiment has the best performance on both datasets, which is due to the fact that the model of this embodiment effectively retains the underlying feature information and original emotional semantics of the image through multi-layer cross-modal information fusion. The introduction of the cross-modal multi-head attention mechanism enables the model to better align aspect words with the fused multi-modal information, thereby improving the accuracy and robustness of classification.

[0179] Table 4 Results of MASC task experiments

[0180]

[0181] 3. JMASA results analysis

[0182] According to the results in Table 5, the model of this embodiment is slightly inferior to VLP-MABSA in F1 value on Twitter-2017 dataset, but its overall performance is still superior. In Twitter-2017 dataset, each sample has more aspect words and sentiment information, which may be because the sentiment information of other aspect words is classified during sentiment classification. Overall, the model of this embodiment is highly competitive.

[0183] Table 5 Results of JMASA on Twitter dataset

[0184]

[0185]

[0186] The present invention provides a multimodal aspect-level sentiment analysis method, system, device and medium combining dual-granularity image information, which can bring at least the following beneficial effects:

[0187] In the multimodal aspect word extraction task, in order to enhance the interaction between the image modality and the text modality, the ClipCap network is used to generate coarse-grained feature description text of the image as image prompt information, which effectively improves the precision and accuracy of the model in predicting aspect words and their attributes in the text;

[0188] In multimodal word sentiment classification, in order to capture rich fine-grained sentiment features of images, the cross-modal multi-head attention mechanism is used to deeply interact the underlying features of the image with the original sentiment semantics with the masked text, effectively strengthening the fusion of image features to text features.

[0189] The present invention significantly enhances the deep interaction between images and texts by introducing dual-granularity image features and combining them with a cross-modal multi-head attention mechanism. In particular, before modal fusion, a pairwise modal semantic enhancement strategy is adopted to effectively improve the accuracy of feature alignment. Compared with existing methods, the present invention not only makes breakthrough progress in the utilization of fine-grained image features, but also achieves improvement in the deep fusion of multimodal information, providing new solutions and methods for multimodal aspect-level sentiment analysis.

[0190] The multimodal aspect-level sentiment analysis system provided by the present invention is described below. The multimodal aspect-level sentiment analysis system described below and the multimodal aspect-level sentiment analysis method described above can be referred to each other.

[0191] like Figure 6 As shown, a multimodal aspect-level sentiment analysis system provided by the present invention may include:

[0192] The data receiving module is used to: receive images and texts to be analyzed;

[0193] The prediction module is used to obtain the sentiment polarity prediction results of the images and texts to be analyzed based on the multimodal aspect-level sentiment analysis model obtained by the construction method of the multimodal aspect-level sentiment analysis model described in any of the above items.

[0194] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute any of the multimodal aspect-level sentiment analysis methods described above.

[0195] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0196] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute any of the multimodal aspect-level sentiment analysis methods described above.

[0197] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform any of the multimodal aspect-level sentiment analysis methods described above.

[0198] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0199] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / eot> < / eot> < / eot> < / aot> < / bot> < / eot> < / eot>

Claims

1. A method for constructing a multimodal aspect-level sentiment analysis model, characterized in that: include: Acquire image sample data and text sample data, wherein each text in the text sample data is annotated with aspect words representing sentiment polarity to form annotated aspect word data; According to the image sample data, text sample data, and annotated aspect word data, the image sample features, the spliced ​​text features, and the annotated aspect word features are obtained through the MIEF network; According to the image sample features and the concatenated text features, the predicted aspect word data is obtained through the AE network; According to the predicted aspect word data, image sample features, spliced ​​text features, and annotated aspect word features, the first multimodal fusion data and the second multimodal fusion data are obtained through the MDFF network by using the first cross-modal multi-head attention mechanism and the second cross-modal multi-head attention mechanism; According to the first multimodal fusion data and the second multimodal fusion data, a multimodal aspect-level sentiment analysis model is trained by utilizing the first attention mechanism and the second attention mechanism.

2. The method for constructing a multimodal aspect-level sentiment analysis model according to claim 1, characterized in that: The method of obtaining image sample features, concatenated text features, and annotated aspect word features through the MIEF network based on the image sample data, the text sample data, and the annotated aspect word data includes: According to the image sample data, several image region features whose confidences meet the preset requirements are obtained through the Faster R-CNN network as image sample features; Based on the image sample data, the coarse-grained features in the image are used through the ClipCap network to generate text data; The generated text data and the text sample data are concatenated and passed through the Text Embedding network to obtain the concatenated text features; According to the labeled aspect word data, the labeled aspect word features are obtained through the Aspect Embedding network.

3. The method for constructing a multimodal aspect-level sentiment analysis model according to claim 2, characterized in that: The method of obtaining predicted aspect word data through the AE network based on the image sample features and the spliced ​​text features includes: According to the image sample features and the concatenated text features, the encoding features are obtained through the BART encoder of the AE network combined with the text mask strategy; According to the encoding features, the BART decoder of the AE network is used to set the aspect word extraction task as a special word segmentation index generation task to obtain the predicted aspect word data; Among them, the BART encoder is a multi-layer bidirectional Transformer network, and the BART decoder is a multi-layer unidirectional Transformer network.

4. The method for constructing a multimodal aspect-level sentiment analysis model according to claim 3, characterized in that: The method obtains first multimodal fusion data and second multimodal fusion data according to the predicted aspect word data, image sample features, spliced ​​text features, and annotated aspect word features through the MDFF network, using the first cross-modal multi-head attention mechanism and the second cross-modal multi-head attention mechanism, including: According to the predicted aspect word data, image sample features, and concatenated text features, the first multimodal fusion data is obtained through the MDFF network and the first cross-modal multi-head attention mechanism; According to the image sample features, spliced ​​text features, and annotated word features, the second multimodal fusion data is obtained through the MDFF network and the second cross-modal multi-head attention mechanism.

5. The method for constructing a multimodal aspect-level sentiment analysis model according to claim 4, characterized in that: The method of obtaining first multimodal fusion data based on the predicted aspect word data, image sample features, and spliced ​​text features through the MDFF network and the first cross-modal multi-head attention mechanism includes: Masking the concatenated text features using the predicted aspect word data to obtain a first masked text feature sequence; Perform layer normalization and linear transformation on the first masked text feature sequence to obtain the query of the first cross-modal multi-head attention mechanism; Perform layer normalization and linear transformation on the image sample features to obtain the keys and values ​​of the first cross-modal multi-head attention mechanism; According to the query, key and value of the first cross-modal multi-head attention mechanism, the fusion of the image modality and the text modality is repeatedly deepened in combination with the first cross-modal multi-head attention mechanism and the first feedforward neural network to obtain the first multimodal fusion data.

6. The method for constructing a multimodal aspect-level sentiment analysis model according to claim 5, characterized in that: The method obtains the second multimodal fusion data by using the second cross-modal multi-head attention mechanism through the MDFF network according to the image sample features, the spliced ​​text features, and the annotated aspect word features, including: Using the annotated aspect word features to mask the concatenated text features, obtaining a second masked text feature sequence; Perform layer normalization and linear transformation on the second mask text feature sequence to obtain the query of the second cross-modal multi-head attention mechanism; Perform layer normalization and linear transformation on the image sample features to obtain the keys and values ​​of the second cross-modal multi-head attention mechanism; According to the query, key and value of the second cross-modal multi-head attention mechanism, the second cross-modal multi-head attention mechanism and the second feedforward neural network are combined to repeatedly deepen the fusion of the image modality and the text modality to obtain the second multimodal fusion data.

7. The method for constructing a multimodal aspect-level sentiment analysis model according to claim 6, characterized in that: The method of training a multimodal aspect-level sentiment analysis model based on the first multimodal fusion data and the second multimodal fusion data using the first attention mechanism and the second attention mechanism includes: The predicted aspect word data is used as the query of the first attention mechanism, and the first multimodal fusion data is used as the key and value of the first attention mechanism. Through the first attention mechanism, the sentiment polarity prediction result based on the predicted aspect word is obtained; The features of the annotated aspect words are used as the query of the second attention mechanism, and the second multimodal fusion data is used as the key and value of the second attention mechanism. Through the second attention mechanism, the sentiment polarity prediction result based on the annotated aspect words is obtained; Combining the sentiment polarity prediction results based on predicted aspect words and the sentiment polarity prediction results based on labeled aspect words, a multimodal aspect-level sentiment analysis model is trained using the cross entropy function.

8. A multimodal aspect-level sentiment analysis method, characterized in that: include: receiving images and text to be analyzed; According to the image and text to be analyzed, the multimodal aspect-level sentiment analysis model obtained by the method for constructing a multimodal aspect-level sentiment analysis model described in any one of claims 1-7 obtains the aspect-level sentiment polarity prediction results of the image and text to be analyzed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the method for constructing a multimodal aspect-level sentiment analysis model as described in any one of claims 1 to 7 and / or the multimodal aspect-level sentiment analysis method as described in claim 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for constructing a multimodal aspect-level sentiment analysis model as described in any one of claims 1 to 7 and / or the multimodal aspect-level sentiment analysis method as described in claim 8.

Images