Multi-modal aspect-level sentiment analysis method based on global-local attention

Abstract

The invention discloses a multi-modal aspect level sentiment analysis method based on global-local attention multi-granularity semantic fusion, belongs to the technical field of multi-modal aspect level sentiment analysis, and uses a local-global text image interaction module to realize alignment of text features and visual features from coarse granularity to fine granularity. A visually enhanced text representation is obtained. A local-global text aspect interaction module is used for capturing interaction information of different granularities between aspects and contexts, information interaction of the aspects and the texts on different granularities is considered, and global interaction between the texts and the aspects and word-level interaction between aspect words and text words are captured through global and local attention. The problem that information is lost due to the fact that a rough attention mechanism is used for aspect modeling in the past is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal aspect-level sentiment analysis, and specifically to a multimodal aspect-level sentiment analysis method based on global-local attention. Background Art

[0002] With the rapid growth of web applications and the widespread and low-cost availability of the Internet, people are gradually participating in expressing their opinions on the Internet. Due to the ubiquity of multimodal data such as text and images in current social platforms, the methods of multimodal aspect-level sentiment analysis are constantly developing and have become a research hotspot in the field of sentiment computing.

[0003] However, they failed to consider exploring both coarse-grained and fine-grained information in visual features at the same time. Typical methods in multimodal aspect-level sentiment analysis usually utilize cross-modal transformers to capture the interaction between text and visual modalities. These methods mainly rely on coarse-grained information extracted from images, such as features of the entire image, and achieve alignment between images and text to a certain extent. However, it is common in multimodal data that images and texts have inconsistent information about the target. Coarse-grained features are not sufficient for accurate classification. At the same time, many methods also consider exploring fine-grained interactions about aspects, but they only consider sentiment information at the word level, which can lead to prediction errors when the sentiment polarity of aspect words in text is unclear. At the same time, most existing methods regard aspect words as a whole, ignoring that different words in aspect phrases have different contributions to the interaction between aspects and text. Summary of the invention

[0004] The present invention provides a multimodal aspect-level sentiment analysis method based on global-local attention, which solves the problem of prediction errors caused when the sentiment polarity of aspect words in a text is unclear.

[0005] To achieve the above object, the present invention provides the following technical solutions: A multimodal aspect-level sentiment analysis method based on global-local attention, including: Acquire multimodal data and preprocess them; Extract features from preprocessed multimodal data; Based on the extracted features, a local-global multi-granularity text-image graph convolution module is constructed to obtain text-image fusion features; Based on the extracted features, a local-global multi-granularity text-aspect interaction module is constructed to obtain text-aspect fusion features; The text image fusion feature and the text aspect fusion feature are connected to obtain the final fusion feature, and the prediction result is obtained based on the final fusion feature; Calculate the loss value between the predicted result and the true value, obtain the multimodal aspect-level sentiment analysis network, and perform sentiment analysis based on the multimodal aspect-level sentiment analysis network.

[0006] Preferably, the steps of obtaining multimodal data and preprocessing are specifically as follows: Obtain multimodal data, extract text data from each multimodal data, obtain the dependency arc probability matrix through the dependency parse tree, use it as the text adjacency matrix, generate corresponding image descriptions and scene descriptions from the image data in the multimodal data, use the caption transformer to generate the image caption, encapsulate the global visual information of the image, and use the scene graph to generate the scene graph description of the image We select triplets of multiple sets of object-to-object and object-to-image relations in the scene description, convert them into serialized sentences separated by “,”, and then connect the triplets by the token [ts].

[0007] Preferably, the extracted features include scene description, text and aspect representation.

[0008] Preferably, a local-global multi-granularity text-image graph convolution module is constructed based on the extracted features, and the steps of obtaining text-image fusion features are specifically as follows: Perform multi-head attention interaction on text and scene description, use text representation as query, scene description representation as key and value, pass the output of multi-head attention mechanism through a fully connected layer and layer normalization to obtain fine-grained image-text fusion features, perform multi-head attention on text and image description to obtain coarse-grained image-text fusion features; Calculate the similarity between each element in the fusion feature to obtain the similarity matrix, and calculate the product of the adjacency matrix and the similarity matrix to obtain the fusion adjacency matrix; The fused adjacency matrix is ​​used to perform graph convolution operations, and the results of the graph convolution operations are average pooled to obtain the text image fusion features.

[0009] Preferably, a local-global multi-granularity text-aspect interaction module is constructed based on the extracted features, and the steps of obtaining the text aspect fusion features are specifically as follows: Use the multi-head attention mechanism, the aspect representation as the query, the text representation as the key and value, and obtain the local text aspect representation; The aspect representation and the text representation are average pooled, and cross-attention operations are performed on the aspect pooled representation and the text representation. At the same time, cross-attention operations are performed on the text pooled representation and the aspect representation to calculate the representation of the global impact of the aspect on the text and the representation of the global impact of the text on the aspect. The pooled representation is used as the query, and the two global representations are concatenated and passed through a linear layer to obtain the global feature representation of the text aspect.

[0010] The global text aspect representation and the local text aspect representation are concatenated to obtain the final text aspect fusion representation.

[0011] Preferably, the final fused features are passed through a fully connected layer to obtain a prediction result.

[0012] Preferably, a cross entropy loss function is used to calculate the loss value between the predicted label and the true value, the network is trained using a gradient descent method, the network parameters are updated, and a trained multimodal aspect-level sentiment analysis network is obtained.

[0013] A multimodal aspect-level sentiment analysis system based on global-local attention, including: Preprocessing module: used to obtain multimodal data and perform preprocessing; Extraction module: used to extract features from preprocessed multimodal data; The first construction module is used to construct a local-global multi-granularity text-image graph convolution module based on the extracted features to obtain text-image fusion features; The second building module is used to build a local-global multi-granularity text-aspect interaction module based on the extracted features to obtain text-aspect fusion features; Prediction module: used to connect the text image fusion feature and the text aspect fusion feature to obtain the final fusion feature, and obtain the prediction result based on the final fusion feature; Analysis module: used to calculate the loss value between the predicted result and the true value, obtain the multimodal aspect-level sentiment analysis network, and perform sentiment analysis based on the multimodal aspect-level sentiment analysis network.

[0014] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a multimodal aspect-level sentiment analysis method based on global-local attention when executing the computer program.

[0015] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a multimodal aspect-level sentiment analysis method based on global-local attention.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a multimodal aspect-level sentiment analysis method based on global-local attention, and uses a local-global text-image interaction module to achieve alignment of text features and visual features from coarse-grained to fine-grained to obtain visually enhanced text representation. The local-global text-aspect interaction module is used to capture the interactive information of different granularities between aspects and contexts, and the information interaction between aspects and texts at different granularities is considered. The global and local attention is used to capture the global interaction between text and aspects, as well as the word-level interaction between aspect words and text words. This overcomes the problem of information loss caused by the previous use of a coarse attention mechanism to model aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the multimodal aspect-level sentiment analysis method based on global-local attention of the present invention; Figure 2 This is a flow chart of a multimodal aspect-level sentiment analysis method based on global-local attention according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the network structure of the present invention; Figure 4 This is a block diagram of the multimodal aspect-level sentiment analysis system based on global-local attention of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0021] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0022] like Figure 1As shown, the embodiment of the present invention provides a multimodal aspect-level sentiment analysis method based on global-local attention, including: S101 acquires multimodal data and performs preprocessing; Parse the text in the data and use the probability matrix obtained by parsing the dependency tree as the initial adjacency matrix A. Generate text using the images in the data: generate image captions and image object-level description scene graphs respectively. The text, image captions, and scene graphs together form the data set of the model.

[0023] S102 extracts features from the preprocessed multimodal data; The pre-trained model Bert is used to extract feature representations of text, caption, and scene graph, and the extracted feature representations are used as input for subsequent modules.

[0024] S103 constructs a local-global multi-granularity text-image graph convolution module based on the extracted features to obtain text-image fusion features; 3.1 In the local-global multi-granularity text-image graph convolution module, the interaction between text and scene description is used to obtain the local text-image fusion representation, and the interaction between text and image description is used to obtain the global text-image fusion representation.

[0025] 3.2 First, multi-head attention interaction is performed on the text and scene description, and the text representation is used as the query, and the scene description representation is used as the key and value. The output result of the multi-head attention mechanism is passed through a fully connected layer and layer normalization, and finally a fine-grained image-text fusion feature is obtained. Similarly, multi-head attention is performed on the text and image description to obtain a coarse-grained image-text fusion feature.

[0026] 3.3 Calculate the similarity between each element in the fused feature to obtain the similarity matrix S_global and S_local. Calculate the product of the adjacency matrix and the similarity matrix to obtain the fused adjacency matrix A_global and A_local.

[0027] 3.4 Use the fused adjacency matrix to perform graph convolution operations. Perform average pooling on the results of graph convolution operations.

[0028] S104 constructs a local-global multi-granularity text-aspect interaction module based on the extracted features to obtain text-aspect fusion features; 4.1 Use the multi-head attention mechanism, aspect representation as query, text representation as key and value to obtain local text aspect representation.

[0029] 4.2 Perform average pooling on the aspect representation and text representation, perform cross-attention operation on the aspect pooled representation and text representation, and perform cross-attention operation on the text pooled representation and aspect representation, and calculate the representation of the global impact of the aspect on the text and the representation of the global impact of the text on the aspect. The pooled representation is used as the query. The two global representations are concatenated and passed through a linear layer to obtain the global feature representation of the text aspect.

[0030] 4.3 Concatenate the global text aspect representation and the local text aspect representation to obtain the final text aspect fusion representation.

[0031] S105 connects the text image fusion feature and the text aspect fusion feature to obtain a final fusion feature, and obtains a prediction result based on the final fusion feature; The text-aspect fusion feature and the text-visual fusion feature are connected as the final fusion feature. After the fully connected layer, the final prediction result is obtained. S106 calculates the loss value between the prediction result and the true value, obtains a multimodal aspect-level sentiment analysis network, and performs sentiment analysis based on the multimodal aspect-level sentiment analysis network.

[0032] The cross entropy loss function is used to calculate the loss value between the predicted label and the true value, the gradient descent method is used to train the network, and the network parameters are updated to obtain a trained multimodal aspect-level sentiment analysis network.

[0033] We use global and local visual information to interact with text. We extract coarse and fine visual information from the image and fuse it with the text information to effectively achieve multi-granular alignment of text and image. The extraction of multi-granular visual sentiment information effectively bridges the semantic gap between different modalities.

[0034] We consider the information interaction between aspects and text at different granularities, and use global and local attention to capture the global interaction between text and aspects as well as the word-level interaction between aspect words and text words, overcoming the problem of information loss caused by using a coarse attention mechanism to model aspects.

[0035] like Figure 2 As shown, another embodiment of the present invention provides a multimodal aspect-level sentiment analysis method based on global-local attention, comprising: Step 1. Data preprocessing: Obtain multimodal data. In this example, the multimodal data comes from TWITTER-15 and TWITTER-17. Extract the text data in each multimodal data, obtain the dependency arc probability matrix through the dependency parse tree, and use it as the text adjacency matrix A. Compared with the final discrete output of the dependency parser, the dependency probability matrix represents the correlation between words and can capture rich structural information by providing all potential syntactic information.

[0036] The image data in the multimodal data is converted into corresponding image descriptions C and scene descriptions G. The caption transformer is used to generate the image caption to encapsulate the global visual information of the image; the scene graph is used to generate the scene graph description of the image, which describes the local fine-grained features of the image.

[0037] Since the scene description G generates several triplets representing object-to-object and object-to-image relationships, we select 5 groups of object-to-object and 5 groups of object-to-image relationship triplets, convert them into serialized sentences separated by “,”, and then connect the triplets through the tag [ts].

[0038] Step 2: Extract features Use the text pre-training model Bert as the network structure for extracting features. Get feature representation: , the output dimension is 768.

[0039] Step 3: Local-global multi-granularity image-text convolution interaction module A local-global image-text convolution interaction module is constructed, and the number of layers of the two GCN convolutions is 2.

[0040] First, we obtain the local and global fusion features of text and image. We mainly rely on the multi-head attention mechanism to obtain fusion features. We perform multi-head attention interaction on text and scene description, use the text representation as the query, and the scene description representation as the key and value, and normalize the output of the multi-head attention mechanism through a layer, and finally obtain fine-grained image-text fusion features.

[0041] The calculation process of multi-head attention interaction is:

[0042] ,

[0043] in , , , is a randomly selected learnable weight parameter, and CONCAT represents a concatenation operation. Then its output passes through a fully connected layer and layer normalization to obtain the local image-text fusion feature:

[0044] Similarly, multi-head attention is performed on the text and image caption to obtain coarse-grained image-text fusion features.

[0045] ,

[0046] Calculate the similarity between each element in the fused feature to obtain the similarity matrix S. Calculate the product of the adjacency matrix and the similarity matrix to obtain the fused adjacency matrix and.

[0047]

[0048] Used to calculate the similarity between two feature vectors, where , .

[0049] Calculate the dot product of the text connection matrix A and the similarity matrix S to obtain the updated adjacency matrix.

[0050]

[0051] The updated adjacency matrix and fused features are used to perform graph convolution operations to establish the connection between image and text information based on grammatical relations.

[0052] The calculation process of graph convolution is as follows:

[0053]

[0054] in, and is the feature representation of the l-th layer output of the i-th node, and is the trainable weight matrix of the lth layer of GCN. and Respectively represent the similarity between nodes in the local and global fusion adjacency matrices. Therefore, after graph convolution, the final graph-based fusion representation is ,

[0055] Apply average pooling function on these two output results to obtain a global representation of text-image fusion and .

[0056] Step 4: Local-global multi-granularity aspect text interaction module Construct an aspect-text interaction module based on local-global attention. Use the pre-trained model Bert to obtain the contextual representation of aspect words , the output dimension is 768.

[0057] The local attention mechanism captures word-level interactions to enhance the relational information between aspects and contexts.

[0058] We use a multi-head attention mechanism to compute a fused representation at the local granularity of textual aspects, where the aspect representation is used as the query and the textual representation is used as the key and value.

[0059]

[0060] The global attention mechanism explores the interactions between context and aspects and generates a coarse-grained representation of the relations.

[0061] First, calculate the representation of the global impact of the aspect on the text, and perform average pooling on the aspect representation The aspect pooled representation and the text representation are cross-attentioned, with the aspect pooled representation as the query and the text representation as the key and value.

[0062]

[0063] Then calculate the representation of the global impact of the text. Perform average pooling on the text representation , perform cross-attention operations on the text pooling representation and aspect representation:

[0064] The two global representations are concatenated and passed through a linear layer to obtain the global feature representation of the text.

[0065]

[0066] The global text aspect representation and the local text aspect representation are concatenated to obtain the final text aspect fusion representation.

[0067]

[0068] Step 5: Train the model The fusion features of the text image are combined with the fusion features of the text to obtain the final result H.

[0069]

[0070] The result H after a linear layer is sent to softmax to obtain the final emotion label. is a learnable parameter.

[0071]

[0072] In order to optimize the parameters in the model, the present invention adopts the cross entropy loss constructed by the predicted value and the true value of the aspect entity sentiment label as the training loss function of the model sentiment analysis task:

[0073] like Figure 4 As shown, another embodiment of the present invention provides a multimodal aspect-level sentiment analysis system based on global-local attention, comprising: Preprocessing module: used to obtain multimodal data and perform preprocessing; Extraction module: used to extract features from preprocessed multimodal data; The first construction module is used to construct a local-global multi-granularity text-image graph convolution module based on the extracted features to obtain text-image fusion features; The second building module is used to build a local-global multi-granularity text-aspect interaction module based on the extracted features to obtain text-aspect fusion features; Prediction module: used to connect the text image fusion feature and the text aspect fusion feature to obtain the final fusion feature, and obtain the prediction result based on the final fusion feature; Analysis module: used to calculate the loss value between the predicted result and the true value, obtain the multimodal aspect-level sentiment analysis network, and perform sentiment analysis based on the multimodal aspect-level sentiment analysis network.

[0074] A terminal device is provided in one embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0075] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0076] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0077] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0078] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0079] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0080] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields, and the above-mentioned specific embodiments are only illustrative and instructive, rather than restrictive. Under the guidance of the specification, a person skilled in the art can also make many forms without departing from the scope of protection of the claims of the present invention, all of which belong to the scope of protection of the present invention.

Claims

1. A multimodal aspect-level sentiment analysis method based on global-local attention, characterized in that: include: Acquire multimodal data and preprocess them; Extract features from preprocessed multimodal data; Based on the extracted features, a local-global multi-granularity text-image graph convolution module is constructed to obtain text-image fusion features; Based on the extracted features, a local-global multi-granularity text-aspect interaction module is constructed to obtain text-aspect fusion features; The text image fusion feature and the text aspect fusion feature are connected to obtain the final fusion feature, and the prediction result is obtained based on the final fusion feature; Calculate the loss value between the predicted result and the true value, obtain the multimodal aspect-level sentiment analysis network, and perform sentiment analysis based on the multimodal aspect-level sentiment analysis network.

2. The multimodal aspect-level sentiment analysis method based on global-local attention according to claim 1, characterized in that: The specific steps for obtaining multimodal data and preprocessing are: Obtain multimodal data, extract text data from each multimodal data, obtain the dependency arc probability matrix through the dependency parse tree, use it as the text adjacency matrix, generate corresponding image descriptions and scene descriptions from the image data in the multimodal data, use the caption transformer to generate the image caption, encapsulate the global visual information of the image, and use the scenegraph to generate the scene graph description of the image; We select triplets of multiple sets of object-to-object and object-to-image relations in the scene description, convert them into serialized sentences separated by ",", and then connect the triplets by the token [ts].

3. The multimodal aspect-level sentiment analysis method based on global-local attention according to claim 2, characterized in that: The extracted features include scene description, text, and aspect representation.

4. The multimodal aspect-level sentiment analysis method based on global-local attention according to claim 3, characterized in that: Based on the extracted features, a local-global multi-granularity text-image graph convolution module is constructed. The steps to obtain the text-image fusion features are as follows: Perform multi-head attention interaction on text and scene description, use text representation as query, scene description representation as key and value, pass the output of multi-head attention mechanism through a fully connected layer and layer normalization to obtain fine-grained image-text fusion features, perform multi-head attention on text and image description to obtain coarse-grained image-text fusion features; Calculate the similarity between each element in the fusion feature to obtain the similarity matrix, and calculate the product of the adjacency matrix and the similarity matrix to obtain the fusion adjacency matrix; The fused adjacency matrix is ​​used to perform graph convolution operations, and the results of the graph convolution operations are average pooled to obtain the text image fusion features.

5. The multimodal aspect-level sentiment analysis method based on global-local attention according to claim 1, characterized in that: Based on the extracted features, a local-global multi-granularity text-aspect interaction module is constructed. The steps to obtain the text-aspect fusion features are as follows: Use the multi-head attention mechanism, the aspect representation as the query, the text representation as the key and value, and obtain the local text aspect representation; Perform average pooling on the aspect representation and the text representation, perform cross-attention operation on the aspect pooled representation and the text representation, and perform cross-attention operation on the text pooled representation and the aspect representation, and calculate the representation of the global impact of the aspect on the text and the representation of the global impact of the text on the aspect. The pooled representation is used as the query, and the two global representations are concatenated and passed through a linear layer to obtain the global feature representation of the text aspect. The global text aspect representation and the local text aspect representation are concatenated to obtain the final text aspect fusion representation.

6. The multimodal aspect-level sentiment analysis method based on global-local attention according to claim 1, characterized in that: The final fusion features are passed through the fully connected layer to obtain the prediction result.

7. The multimodal aspect-level sentiment analysis method based on global-local attention according to claim 1, characterized in that: The cross entropy loss function is used to calculate the loss value between the predicted label and the true value, the gradient descent method is used to train the network, the network parameters are updated, and a trained multimodal aspect-level sentiment analysis network is obtained.

8. A multimodal aspect-level sentiment analysis system based on global-local attention, characterized in that include: Preprocessing module: used to obtain multimodal data and perform preprocessing; Extraction module: used to extract features from preprocessed multimodal data; The first construction module is used to construct a local-global multi-granularity text-image graph convolution module based on the extracted features to obtain text-image fusion features; The second building module is used to build a local-global multi-granularity text-aspect interaction module based on the extracted features to obtain text-aspect fusion features; Prediction module: used to connect the text image fusion feature and the text aspect fusion feature to obtain the final fusion feature, and obtain the prediction result based on the final fusion feature; Analysis module: used to calculate the loss value between the predicted result and the true value, obtain the multimodal aspect-level sentiment analysis network, and perform sentiment analysis based on the multimodal aspect-level sentiment analysis network.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multimodal aspect-level sentiment analysis method based on global-local attention are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multimodal aspect-level sentiment analysis method based on global-local attention are implemented as described in any one of claims 1 to 7.

Images