Multimodal aspect-level sentiment analysis method based on multi-view gated fusion network

Abstract

The present invention provides a multimodal aspect-level sentiment analysis method, system, storage medium and electronic device based on a multi-view gated fusion network, and relates to the technical field of aspect-level sentiment analysis. The present invention views multimodal information from different perspectives, can effectively expand the information content of multimodal data, and can achieve relatively good sentiment prediction effects even when there are fewer multimodal data samples; based on a dynamic gated self-attention module, a gated adjustment vector is constructed using aspect word information representation, and is embedded in a self-attention mechanism to achieve dynamic adjustment of weight calculation, thereby effectively filtering out noise information unrelated to aspect words, improving the information quality of fine-grained features, and facilitating later feature fusion; a three-view decomposition high-order pooling mechanism simultaneously fuses three different view features, and captures the complex interactions between different views and different modal features through multiple expansion and compression stages, achieving an effective fusion effect.

Description

Technical Field

[0001] The present invention relates to the technical field of aspect-level sentiment analysis, and in particular to a multimodal aspect-level sentiment analysis method, system, storage medium and electronic device based on a multi-view gated fusion network. Background Art

[0002] With the new round of industrial revolution and technological innovation, new energy vehicles and intelligent connected vehicles have become the main development directions of the global automotive industry. Seizing this major development opportunity will also provide beneficial assistance to my country in addressing social issues such as road congestion, emission pollution, and traffic safety. In the user experience and feedback regarding the use of new energy vehicles, user-generated content (UGC; such as online reviews and social blogs) is the main channel for obtaining product market response. As consumers accumulate more and more, the emotional biases contained in it provide reliable data support for exploring user needs, understanding user preferences, determining product advantages and disadvantages, and promoting product upgrades. Therefore, analyzing the emotional preferences of user-generated content is of great significance from the perspective of both customers and manufacturers.

[0003] Currently, sentiment analysis methods for user-generated content (UGC) operate at three granularity levels: paragraph-level, sentence-level, and aspect-level. While paragraph-level and sentence-level sentiment analysis can only comprehensively assess the sentiment of the entire text, aspect-level sentiment analysis can make individual sentiment judgments based on different aspect words within the text. This provides a more in-depth analysis that is more consistent with authentic emotional thinking. Furthermore, thanks to advances in multimedia technology, UGC is shifting towards multimodal forms, encompassing not only text data but also other modal data such as images. The interrelationships between the different modalities within UGC contain more valuable information, necessitating their full exploration. However, current research on multimodal aspect-level sentiment analysis is relatively limited. Therefore, effectively leveraging the characteristics of multimodal data to establish aspect-level sentiment analysis models remains an urgent challenge. Patent CN115293170A (application name: A method for aspect-level multimodal sentiment analysis based on collaborative attention fusion [3], applicant: Guilin University of Electronic Technology, application date: 2022.08.12) uses the attention mechanism to generate aspect-guided global text feature representation and aspect-guided global image feature representation, and the cross-modal feature interaction mechanism to simultaneously generate local text feature representation integrated with vision and local visual feature representation integrated with text. Finally, a gated multimodal fusion mechanism is constructed to effectively capture the global semantic association within the modality and the local semantic alignment between modalities.

[0004] However, the above solution fails to fully consider the potential presence of noise in multimodal data that is unrelated to a given aspect word. Generally, this noise falls into two categories: first, meaningless information in text or images that is unrelated to the aspect word itself; second, when a single data sample contains multiple aspect words, the sentiment polarity of each aspect word may differ. This means that information useful for aspect word A may become noise for aspect word B. If these two types of noise are not filtered out during feature representation and participate in modal interaction fusion, they will significantly interfere with the model's judgment of the sentiment polarity of the aspect word. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides a multimodal aspect-level sentiment analysis method, system, storage medium and electronic device based on a multi-view gated fusion network, which solves the technical problem of considering the presence of noise information irrelevant to a given aspect word in multimodal data.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A multimodal aspect-level sentiment analysis method based on a multi-view gated fusion network, wherein the multi-view gated fusion network includes an input layer, a multi-view feature extraction layer, a feature fusion layer, and an output layer; the multimodal aspect-level sentiment analysis method includes:

[0010] S1. Acquire a multimodal image-text dataset, wherein the multimodal image-text dataset includes a text review, a set of related images, and an aspect word sequence, wherein the aspect word sequence is a subsequence of the text review;

[0011] S2. At the input layer, the text comments, associated images, and aspect word sequences are encoded respectively, and the context word embedding matrix, image embedding matrix, and aspect word embedding matrix are obtained accordingly; sentence-level features and overall image features are obtained;

[0012] and obtaining an adjacency matrix representing syntactic dependencies based on the text comments;

[0013] S3. In the multi-view feature extraction layer, according to the context word embedding matrix, aspect word features that take into account the context are obtained, and the sentence-level features and the overall image features are combined to obtain the context-aware global view features of the text and the global view features of the image;

[0014] And according to the aspect word embedding matrix, noise filtering is performed on the context word embedding matrix and the image embedding matrix based on a dynamic gated self-attention module to obtain text word order view features and image region view features; and the text syntax view features are obtained by combining the adjacency matrix;

[0015] S4. In a feature fusion layer, the text global view feature, the image global view feature, the text word order view feature, the text syntax view feature, and the image region view feature are fused to obtain a final fused feature;

[0016] S5. At the output layer, the prediction result of the aspect-level sentiment category is obtained based on the final fusion feature.

[0017] Preferably, in S2, the BERT model is used to encode the text comments and aspect word sequences, and the ResNet50 model is used to encode the associated images; and / or the spacy tool is used to obtain the adjacency matrix A;

[0018] Among them, the BERT model outputs the word-level context word embedding matrix H S , aspect word embedding matrix H A The feature information output by the penultimate layer of the ResNet50 model is then subjected to dimensionality reduction to obtain the image embedding matrix H I ;

[0019] And use the [CLS] feature information of the BERT model as sentence-level features The feature information output by the last layer of the ResNet50 model is then subjected to dimensionality reduction to obtain the overall image features.

[0020] Preferably, the process of acquiring the text global view feature and the image global view feature in S3 includes:

[0021] According to the length of the aspect word sequence and its position in the review text, the corresponding subsequence in the context word embedding matrix is extracted

[0022] Average pooling the corresponding subsequences to obtain aspect word features;

[0023]

[0024] in, Indicates aspect word features; n indicates aspect word index; Indicates H S The k+nth word in the aspect word sequence, k represents the number of the word before the starting position of the aspect word sequence; r refers to the length of the aspect word sequence;

[0025] The feature vector obtained by adding the sentence-level feature and the aspect word feature through the self-attention mechanism is used as the global view feature of the text;

[0026]

[0027] in, Represents the global view feature of the text; Attention(·) is the attention mechanism;

[0028] The feature vector obtained by adding the overall image feature and the aspect word feature through the self-attention mechanism is used as the global view feature of the image;

[0029]

[0030] in, Represents the global view features of the image.

[0031] Preferably, the process of acquiring the text word order view features, the image area view features, and the text syntax view features in S3 includes:

[0032] constructing a gated adjustment vector according to the aspect word embedding matrix;

[0033] G A =expand(σ(W G ×AvgPool(H A )+b G )) (4)

[0034] Among them, G A represents the gated adjustment vector; expand(·) is the dimension of the feature vector expanded along the length dimension; σ(·) is the Sigmoid activation function; AvgPool(·) is the average pooling operation; W G with b G All are trainable parameters;

[0035] The gated adjustment vector is embedded in the multi-head self-attention mechanism of the dynamic gated self-attention module to dynamically adjust the attention weight calculation process between the key-query pairs, perform noise filtering, and obtain the aspect-sensitive text and image feature representation H SA and H IA ;

[0036] Passing the aspect-word-sensitive text feature representation through Bi-LSTM to obtain the text word order view feature;

[0037]

[0038]

[0039] Among them, View Seq Represents the text word order view features; Indicates H SA The node vector in ; Represent the forward LSTM network and the backward LSTM network respectively;

[0040] The aspect-word-sensitive text feature representation is passed through L layers of GCN and combined with the adjacency matrix to obtain the text syntactic view feature;

[0041]

[0042]

[0043] Among them, View Sny Represents text syntactic view features; Represents a View Sny The node vector in; ReLU(·) is the ReLU activation function; A represents the adjacency matrix; W C is a trainable parameter; represents the degree matrix of A;

[0044] performing normalization adjustment on the feature representation of the image that is sensitive to the aspect words to obtain the image region view feature;

[0045] View Reg =Norm(H IA ) (9)

[0046] Among them, View Reg represents the image region view feature; Norm(·) represents the normalization operation.

[0047] Preferably, the process of obtaining the aspect-word-sensitive text and image feature representation includes:

[0048] The context word embedding matrix and the image embedding matrix are used as the input of the multi-head self-attention mechanism, and the Key, Query, and Value keys are obtained through projection.

[0049]

[0050]

[0051]

[0052] in, Indicates H S Input the query, key, and value feature matrices obtained by projection after the self-attention mechanism; Indicates H IThe query, key, and value feature matrices are obtained by projection after inputting the self-attention mechanism.

[0053] According to the gate adjustment vector of the j-th attention mechanism head Get the Key feature matrix in each attention head And the Query feature matrix

[0054]

[0055]

[0056] in, Indicates that if part of the feature vector is not activated by the gated adjustment vector, the original feature is retained; is the Hadamard dot product; Respectively represent H S 、H I Query feature matrix in the jth self-attention mechanism head; Respectively represent H S 、H I Key feature matrix in the j-th self-attention mechanism head;

[0057] According to the gated adjustment vector Adjusted Key feature matrix And the Query feature matrix Obtaining text and image feature representations that are sensitive to the aspect words based on the self-attention mechanism;

[0058]

[0059]

[0060]

[0061]

[0062] in, They represent the aspect-sensitive text and image feature representations output by the j-th self-attention mechanism head respectively; and is a trainable parameter; h represents the number of heads of the multi-head attention mechanism; concat represents the feature concatenation operation.

[0063] Preferably, a two-layer three-view decomposition high-order pooling mechanism TFH is introduced in the feature fusion layer. Each layer of TFH is composed of one three-view decomposition bilinear pooling mechanism TFB. Each TFB includes two stages: feature expansion and feature squeezing. S4 includes:

[0064] In the first layer TFH, the text word order view features, text syntax view features and image region view features are fused to obtain fine-grained fusion features;

[0065]

[0066]

[0067] Among them, exp corresponds to the feature expansion stage, and sqz corresponds to the feature compression stage; and is a learnable parameter; Represents the fine-grained output features of the i-th layer TFB in the feature expansion stage; View F，i represents the fine-grained output feature of the i-th layer TFB in the feature compression stage; prj represents the feature projection operation, tanh represents the tanh activation function; SumPool represents the sum pooling operation; View F Represents fine-grained fusion features;

[0068] In the second layer TFH, the fine-grained fusion features are further fused with the text global view features and the image global view features to obtain the final fusion features;

[0069]

[0070]

[0071] in, and is a learnable parameter; Represents the final output feature of the i-th layer TFB in the feature expansion stage; View W，i Represents the final output feature of the i-th layer TFB in the feature compression stage; View W Represents the final fusion feature.

[0072] Preferably, the S5 specifically includes:

[0073] At the output layer, the final fusion feature is normalized by the Softmax classifier to obtain the probability value of the corresponding aspect word sentiment category, where the emotion corresponding to the maximum probability value is used as the prediction result of the aspect-level sentiment category. The calculation process is as follows:

[0074]

[0075] Among them, W p is a trainable parameter, b p For bias.

[0076] A multimodal aspect-level sentiment analysis system based on a multi-view gated fusion network, wherein the multi-view gated fusion network includes an input layer, a multi-view feature extraction layer, a feature fusion layer, and an output layer; the multimodal aspect-level sentiment analysis system includes:

[0077] An acquisition module is used to acquire a multimodal image-text dataset, wherein the multimodal image-text dataset includes a text review, a set of related images, and an aspect word sequence, wherein the aspect word sequence is a subsequence of the text review;

[0078] An encoding module is used to encode the text comments, associated images, and aspect word sequences at the input layer, respectively, to obtain a context word embedding matrix, an image embedding matrix, and an aspect word embedding matrix; and to obtain sentence-level features and overall image features;

[0079] and obtaining an adjacency matrix representing syntactic dependencies based on the text comments;

[0080] An extraction module is configured to obtain context-aware aspect word features based on the context word embedding matrix at the multi-view feature extraction layer, and to obtain context-aware global view features of the text and global view features of the image in combination with the sentence-level features and the overall image features.

[0081] And according to the aspect word embedding matrix, noise filtering is performed on the context word embedding matrix and the image embedding matrix based on a dynamic gated self-attention module to obtain text word order view features and image region view features; and the text syntax view features are obtained by combining the adjacency matrix;

[0082] A fusion module is used to fuse the text global view features, the image global view features, the text word order view features, the text syntax view features and the image region view features at the feature fusion layer to obtain a final fusion feature;

[0083] The prediction module is used to obtain the prediction result of the aspect-level sentiment category based on the final fusion feature at the output layer.

[0084] A storage medium stores a computer program for multimodal aspect-level sentiment analysis based on a multi-view gated fusion network, wherein the computer program enables a computer to execute the multimodal aspect-level sentiment analysis method as described above.

[0085] An electronic device, comprising:

[0086] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for performing the multimodal aspect-level sentiment analysis method as described above.

[0087] (3) Beneficial effects

[0088] The present invention provides a multimodal aspect-level sentiment analysis method, system, storage medium, and electronic device based on a multi-view gated fusion network. Compared with the existing technology, it has the following advantages:

[0089] The multimodal aspect-level sentiment analysis method provided by the present invention is based on a multi-view gated fusion network. It views multimodal information from five different perspectives: a global text view, a global image view, an image region view, a text word order view, and a text syntax view. This effectively expands the information content of multimodal data and achieves relatively good sentiment prediction results even when there are fewer multimodal data samples. In addition, a dynamic gated self-attention module is introduced. Based on the self-attention mechanism, it adjusts the internal importance of features by calculating weights. A gated adjustment vector is constructed using aspect word information representation and embedded in the self-attention mechanism to achieve dynamic adjustment of weight calculations, effectively filtering out noise information unrelated to aspect words, improving the information quality of fine-grained features, facilitating later feature fusion, and helping to accurately predict aspect-level sentiment categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0091] Figure 1 A schematic diagram of the structure of a multi-view gated fusion network provided by an embodiment of the present invention;

[0092] Figure 2 A block diagram of a multimodal aspect-level sentiment analysis method based on a multi-view gated fusion network provided by an embodiment of the present invention;

[0093] Figure 3 A schematic diagram of the structure of a dynamic gated self-attention module provided by an embodiment of the present invention;

[0094] Figure 4 A schematic diagram of the structure of a three-view decomposition high-order pooling mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0095] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0096] The embodiments of the present application provide a multimodal aspect-level sentiment analysis method, system, storage medium and electronic device based on a multi-view gated fusion network, which solves the technical problem of considering the presence of noise information irrelevant to a given aspect word in multimodal data and achieves accurate prediction of aspect-level sentiment.

[0097] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0098] like Figure 1 As shown in FIG, the multimodal aspect-level sentiment analysis method provided by the present invention is based on a multi-view gated fusion network. Specifically:

[0099] First, in order to reduce the negative impact of irrelevant information in multimodal data on the sentiment judgment of aspect words, the present invention designs a dynamic gated self-attention module with an embedded gating mechanism in the early stage of image and text feature generation to denoise text features and image features. By participating in the weight calculation process in the self-attention mechanism, it weakens information irrelevant to the aspect word as much as possible, dynamically emphasizes the more important parts of each feature vector, and improves the quality of feature generation.

[0100] Secondly, in order to fully explore the deep information of different dimensions in multimodal data, such as syntactic information, semantic information, location information, etc., the present invention expands text and image features into five different view features, namely image global view features, image area view features, text global view features, text word order view features, and text syntax view features, so as to capture data information from multiple perspectives, expand the data volume, and improve the aspect-level sentiment prediction effect.

[0101] Finally, in order to capture the complex interactions between different views and different modal features, including the consistency and heterogeneity between views, the present invention designs a triple-view factorized high-order pooling mechanism (TFH) that is extended for multi-view features for the final feature fusion. This fusion mechanism is composed of n triple-view factorized bilinear pooling mechanisms (TFB). Ordinary bilinear pooling mechanisms can only fuse two feature inputs in application, and the model complexity and computational cost are high when processing multimodal feature fusion. However, the triple-view factorized high-order pooling mechanism used in the present invention can fuse three view features, has good scalability, fully exploits the complex interactions between multiple view features, and is highly efficient.

[0102] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0103] Example:

[0104] like Figure 2 As shown, an embodiment of the present invention provides a multimodal aspect-level sentiment analysis method based on a multi-view gated fusion network, wherein the multi-view gated fusion network includes an input layer, a multi-view feature extraction layer, a feature fusion layer, and an output layer; the multimodal aspect-level sentiment analysis method includes:

[0105] S1. Acquire a multimodal image-text dataset, wherein the multimodal image-text dataset includes a text review, a set of related images, and an aspect word sequence, wherein the aspect word sequence is a subsequence of the text review;

[0106] S2. At the input layer, the text comments, associated images, and aspect word sequences are encoded respectively, and the context word embedding matrix, image embedding matrix, and aspect word embedding matrix are obtained accordingly; sentence-level features and overall image features are obtained;

[0107] and obtaining an adjacency matrix representing syntactic dependencies based on the text comments;

[0108] S3. In the multi-view feature extraction layer, according to the context word embedding matrix, aspect word features that take into account the context are obtained, and the sentence-level features and the overall image features are combined to obtain the context-aware global view features of the text and the global view features of the image;

[0109] And according to the aspect word embedding matrix, noise filtering is performed on the context word embedding matrix and the image embedding matrix based on a dynamic gated self-attention module to obtain text word order view features and image region view features; and the text syntax view features are obtained by combining the adjacency matrix;

[0110] S4. In a feature fusion layer, the text global view feature, the image global view feature, the text word order view feature, the text syntax view feature, and the image region view feature are fused to obtain a final fused feature;

[0111] S5. At the output layer, the prediction result of the aspect-level sentiment category is obtained based on the final fusion feature.

[0112] The embodiments of the present invention view multimodal information from five different perspectives: text global view, image global view, image region view, text word order view, and text syntax view. This can effectively expand the amount of information in multimodal data and achieve relatively good sentiment prediction effects even when there are fewer multimodal data samples.

[0113] In addition, a dynamic gated self-attention module is introduced, which adjusts the internal importance of features by calculating weights based on the self-attention mechanism. A gated adjustment vector is constructed using aspect word information representation and embedded in the self-attention mechanism to achieve dynamic adjustment of weight calculation, effectively filtering out noise information irrelevant to aspect words, improving the information quality of fine-grained features, facilitating later feature fusion, and helping to accurately predict aspect-level sentiment categories.

[0114] In fact, this module is also applicable to other related tasks, such as visual question answering tasks and review usefulness prediction tasks. It has great application value and will not be explained in detail here.

[0115] The following are the steps of the above technical solution:

[0116] In step S1 , a multimodal image-text dataset is obtained, where the multimodal image-text dataset includes a text review, a set of associated images, and an aspect word sequence, wherein the aspect word sequence is a subsequence of the text review.

[0117] In this step, a set of multimodal image-text datasets D is given, assuming that each sample d∈D contains a text comment S: {w1, w2, ..., w n} and a set of associated images I: {v1, v2, ..., v m} and an aspect word sequence A: {a1, a2, ..., a r}, where A is a subsequence of S. The goal of this task is to use (S, I) to predict the sentiment polarity of aspect word A. Here, n represents the length of the text, m represents the number of images, and r represents the length of the aspect word.

[0118] For example, the model can be verified and evaluated using public datasets annotated by Yu et al., including Twitter-2015 and Twitter-2017. The two datasets contain multimodal tweets published on Twitter in 2014-2015 and 2016-2017, respectively. In the Twitter-2015 and Twitter-2017 datasets, there are a total of 5338 and 5972 aspect-tweet pairs, and each sample includes tweet text, tweet entity (i.e. aspect word), sentiment label value, and image id. Among them, the aspect word of the sample is the aspect entity, and there are 3 sentiment labels, namely -1, 0, and 1, corresponding to negative, neutral, and positive emotions, respectively. This set of datasets is divided into three parts: training set, validation set, and test set, with a ratio of 3:1:1. The label statistics for the two datasets are shown in Table 1 below:

[0119] Table 1 Dataset label data statistics

[0120]

[0121]

[0122] The specific parameter settings are shown in Table 2 below:

[0123] Table 2 Experimental parameter settings

[0124] Hyperparameters Twitter-2015 Twitter-2017 Number of training rounds 30 30 Batch size 32 32 Learning rate 1e-5 2e-5 Hidden layer dimensions 768 768 Abandonment rate 0.1 0.1 Maximum encoding length of text 50 50 Maximum encoding length of aspect words 8 8

[0125] In step S2:

[0126] S21. At the input layer, the text comments, associated images, and aspect word sequences are encoded respectively, and a context word embedding matrix, an image embedding matrix, and an aspect word embedding matrix are obtained accordingly; and sentence-level features and overall image features are obtained.

[0127] like Figure 1 As shown, the BERT model is used to encode text comments and aspect word sequences, while the ResNet50 model is used to encode associated images. For text comments and aspect words, they must first be converted into the formats "[CLS] + text + [SEP]" and "[CLS] + aspect word + [SEP]." For images, they must be converted into RGB format and resized to 224×224 pixels before using ResNet50 for feature extraction.

[0128] According to the above description, this embodiment sets the dimension to 768. After the data enters the input layer, five information representations will be obtained, namely: the BERT model outputs the word-level context word embedding matrix Aspect word embedding matrix The feature information output by the penultimate layer of the ResNet50 model is then subjected to dimensionality reduction to obtain the image embedding matrix And use the [CLS] feature information of the BERT model as sentence-level features The feature information output by the last layer of the ResNet50 model is then subjected to dimensionality reduction to obtain the overall image features. Among them, n′ and r′ represent the maximum encoding length of text and aspect words respectively.

[0129] In particular, to ensure that all data can be processed in batches by the GPU, a fixed maximum text encoding length needs to be set. Text exceeding this length will be directly truncated, and text less than this length will be padded with zeros: n′ represents the maximum encoding length of the selected text, which is set to 50 in the experiment; 49 represents the number of sub-regions an image is divided into; r′ represents the maximum encoding length of aspect words, which is set to 8 in the experiment.

[0130] S22, and according to the text comments, obtain an adjacency matrix representing syntactic dependencies; Figure 1 As shown, the adjacency matrix A is obtained using the spacy tool.

[0131] In step S3, due to the granularity of feature view extraction, the multi-view feature extraction layer includes extracting coarse-grained view features, namely, text and image global view features, and fine-grained view features, namely, image region view features, text word order view features, and text syntax view features. Before extracting fine-grained view features, in order to filter out information irrelevant to aspect words, the fine-grained original information representation must pass through a dynamic gated self-attention module (such as Figure 3 As shown in the figure), aspect words are used to generate gated adjustment vectors, which are embedded in the self-attention mechanism. By adjusting the weight calculation process, the information noise is dynamically filtered.

[0132] Accordingly, the S3 includes:

[0133] S31. In the multi-view feature extraction layer, aspect word features that take context into consideration are obtained according to the context word embedding matrix. The context-aware text global view features and image global view features are correspondingly obtained by combining the sentence-level features and the overall image features.

[0134] like Figure 1 As shown, according to the length of the aspect word sequence and its position in the review text, the corresponding subsequence in the context word embedding matrix is extracted.

[0135] Average pooling the corresponding subsequences to obtain aspect word features;

[0136]

[0137] in, Indicates aspect word features; n indicates aspect word index; Indicates H S The k+nth word in the aspect word sequence, k represents the number of the word before the starting position of the aspect word sequence; r refers to the length of the aspect word sequence;

[0138] The feature vector obtained by adding the sentence-level feature and the aspect word feature through the self-attention mechanism is used as the global view feature of the text;

[0139]

[0140] in, Represents the global view feature of the text; Attention(·) is the attention mechanism;

[0141] The feature vector obtained by adding the overall image feature and the aspect word feature through the self-attention mechanism is used as the global view feature of the image;

[0142]

[0143] in, Represents the global view features of the image.

[0144] S32, and according to the aspect word embedding matrix, based on the dynamic gated self-attention module, respectively perform noise filtering on the context word embedding matrix and the image embedding matrix to obtain text word order view features and image area view features; and combine with the adjacency matrix to obtain text syntax view features.

[0145] like Figure 1 As shown, according to the aspect word embedding matrix, a gated adjustment vector is constructed;

[0146] G A =expand(σ(W G ×AvgPool(H A )+b G )) (4)

[0147] Among them, G A represents the gated adjustment vector; expand(·) is the dimension of the feature vector expanded along the length dimension; σ(·) is the Sigmoid activation function; AvgPool(·) is the average pooling operation; W G with b G All are trainable parameters;

[0148] The gated adjustment vector is embedded in the multi-head self-attention mechanism of the dynamic gated self-attention module to dynamically adjust the attention weight calculation process between the key-query pairs, perform noise filtering, and obtain the aspect-sensitive text and image feature representation H SA and H IA ; Specifically include:

[0149] First, the context word embedding matrix and the image embedding matrix are used as the input of the multi-head self-attention mechanism, and the Key, Query, and Value keys are obtained through projection.

[0150]

[0151]

[0152]

[0153] in, Indicates H S Input the query, key, and value feature matrices obtained by projection after the self-attention mechanism; Indicates H I The query, key, and value feature matrices are obtained by projection after inputting the self-attention mechanism.

[0154] Secondly, according to the gate adjustment vector of the j-th attention mechanism head Get the Key feature matrix in each attention head And the Query feature matrix

[0155]

[0156] in, Indicates that if part of the feature vector is not activated by the gated adjustment vector, the original feature is retained; is the Hadamard dot product; Respectively represent H S 、H I Query feature matrix in the jth self-attention mechanism head; Respectively represent H S 、H I Key feature matrix in the j-th self-attention mechanism head;

[0157] Then, according to the gated adjustment vector Adjusted Key feature matrix And the Query feature matrix Obtaining text and image feature representations that are sensitive to the aspect words based on the self-attention mechanism;

[0158]

[0159]

[0160]

[0161]

[0162] in, They represent the aspect-sensitive text and image feature representations output by the j-th self-attention mechanism head respectively; and is a trainable parameter; h represents the number of heads of the multi-head attention mechanism; concat represents the feature concatenation operation.

[0163] In terms of obtaining word-sensitive text and image feature representation H SA and H IA , respectively execute:

[0164] (1) The aspect-word-sensitive text feature representation is passed through Bi-LSTM to obtain the text word order view feature;

[0165]

[0166]

[0167] Among them, View Seq Represents the text word order view features; Indicates H SA The node vector in ; They represent the forward LSTM network and the backward LSTM network respectively.

[0168] (2) Since the graph convolutional network (GCN) solves the problem of lack of syntactic constraints and long-range word dependencies in the model through syntactic information and word dependencies, it can process non-Euclidean structure data and effectively extract spatial features. Therefore, the aspect-word-sensitive text feature representation is passed through L layers of GCN and combined with the adjacency matrix to obtain the text syntactic view features;

[0169]

[0170]

[0171] Among them, View Sny Represents text syntactic view features; Represents a View Sny The node vector in ; ReLU(·) is the ReLU activation function; represents the adjacency matrix; W C is a trainable parameter; represents the degree matrix of A.

[0172] (3) performing normalization adjustment on the feature distribution of the image feature representation sensitive to the aspect words to obtain the image region view feature;

[0173] View Reg =Norm(H IA ) (18)

[0174] Among them, View Reg represents the image region view feature; Norm(·) represents the normalization operation.

[0175] In step S4, at the feature fusion layer, the text global view features, the image global view features, the text word order view features, the text syntax view features and the image region view features are fused to obtain the final fused features.

[0176] In particular, such as Figure 1 and 4 As shown in Figure 2, this step introduces a two-layer three-view decomposition high-order pooling mechanism (TFH) at the feature fusion layer to rationally fuse features from different modalities and views. Each TFH layer consists of a three-view decomposition bilinear pooling mechanism (TFB), each of which includes two stages: feature expansion and feature squeezing.

[0177] Accordingly, the S4 includes:

[0178] S41. In the first TFH layer, the text word order view feature, the text syntax view feature, and the image region view feature are fused to obtain a fine-grained fusion feature.

[0179]

[0180]

[0181] Among them, exp corresponds to the feature expansion stage, and sqz corresponds to the feature compression stage; and is a learnable parameter; Represents the fine-grained output features of the i-th layer TFB in the feature expansion stage; View F，i represents the fine-grained output feature of the i-th layer TFB in the feature compression stage; prj represents the feature projection operation, tanh represents the tanh activation function; SumPool represents the sum pooling operation; View F Represents fine-grained fusion features.

[0182] S42. In the second TFH layer, further fusing the fine-grained fusion features with the text global view features and the image global view features to obtain the final fusion features;

[0183]

[0184]

[0185]

[0186] in, and is a learnable parameter; Represents the final output feature of the i-th layer TFB in the feature expansion stage; View W，i Represents the final output feature of the i-th layer TFB in the feature compression stage; View W Represents the final fusion feature.

[0187] The embodiment of the present invention utilizes the above-mentioned three-view decomposition high-order pooling mechanism (TFH) for the final feature fusion. This fusion mechanism can simultaneously fuse three different view features, and through multiple expansion and compression stages, capture the complex interactions between different views and different modal features to achieve an effective fusion effect.

[0188] In step S5, at the output layer, the prediction result of the aspect-level sentiment category is obtained based on the final fusion feature.

[0189] like Figure 1 As shown in FIG, at the output layer, the final fusion feature is normalized by the Softmax classifier to obtain the probability value of the corresponding aspect word sentiment category, where the emotion corresponding to the maximum probability value is used as the prediction result of the aspect-level sentiment category. The calculation process is as follows:

[0190]

[0191] Among them, W p is a trainable parameter, b p For bias.

[0192] An embodiment of the present invention provides a multimodal aspect-level sentiment analysis system based on a multi-view gated fusion network, wherein the multi-view gated fusion network includes an input layer, a multi-view feature extraction layer, a feature fusion layer, and an output layer; the multimodal aspect-level sentiment analysis system includes:

[0193] An acquisition module is used to acquire a multimodal image-text dataset, wherein the multimodal image-text dataset includes a text review, a set of related images, and an aspect word sequence, wherein the aspect word sequence is a subsequence of the text review;

[0194] An encoding module is used to encode the text comments, associated images, and aspect word sequences at the input layer, respectively, to obtain a context word embedding matrix, an image embedding matrix, and an aspect word embedding matrix; and to obtain sentence-level features and overall image features;

[0195] and obtaining an adjacency matrix representing syntactic dependencies based on the text comments;

[0196] An extraction module is configured to obtain context-aware aspect word features based on the context word embedding matrix at the multi-view feature extraction layer, and to obtain context-aware global view features of the text and global view features of the image in combination with the sentence-level features and the overall image features.

[0197] And according to the aspect word embedding matrix, noise filtering is performed on the context word embedding matrix and the image embedding matrix based on a dynamic gated self-attention module to obtain text word order view features and image region view features; and the text syntax view features are obtained by combining the adjacency matrix;

[0198] A fusion module is used to fuse the text global view features, the image global view features, the text word order view features, the text syntax view features and the image region view features at the feature fusion layer to obtain a final fusion feature;

[0199] The prediction module is used to obtain the prediction result of the aspect-level sentiment category based on the final fusion feature at the output layer.

[0200] An embodiment of the present invention provides a storage medium storing a computer program for multimodal aspect-level sentiment analysis based on a multi-view gated fusion network, wherein the computer program enables a computer to execute the multimodal aspect-level sentiment analysis method as described above.

[0201] An embodiment of the present invention provides an electronic device, including:

[0202] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for performing the multimodal aspect-level sentiment analysis method as described above.

[0203] It can be understood that the multimodal aspect-level sentiment analysis system, storage medium and electronic device based on the multi-view gated fusion network provided in the embodiments of the present invention correspond to the multimodal aspect-level sentiment analysis method based on the multi-view gated fusion network provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the multimodal aspect-level sentiment analysis method, and will not be repeated here.

[0204] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0205] 1. The embodiments of the present invention view multimodal information from five different perspectives: text global view, image global view, image region view, text word order view, and text syntax view. This can effectively expand the amount of information in multimodal data and achieve better sentiment prediction effects even when there are fewer multimodal data samples.

[0206] 2. Furthermore, a dynamic gated self-attention module is introduced. Based on the self-attention mechanism, it adjusts the internal importance of features by calculating weights. A gated adjustment vector is constructed using aspect word information and embedded in the self-attention mechanism to dynamically adjust weight calculations. This effectively filters out noise information unrelated to aspect words, improving the information quality of fine-grained features, facilitating later feature fusion and helping accurately predict aspect-level sentiment categories. This module is also applicable to other related tasks, such as visual question answering and review helpfulness prediction, and has great application value, so we will not elaborate on it here.

[0207] 3. The embodiment of the present invention utilizes a three-view decomposition high-order pooling mechanism (TFH) for the final feature fusion. This fusion mechanism can simultaneously fuse three different view features, and capture the complex interactions between different views and different modal features through multiple expansion and compression stages to achieve an effective fusion effect.

[0208] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0209] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multimodal aspect-level sentiment analysis method based on a multi-view gated fusion network, characterized in that: The multi-view gated fusion network includes an input layer, a multi-view feature extraction layer, a feature fusion layer and an output layer; The multimodal aspect-level sentiment analysis method includes: S1. Acquire a multimodal image-text dataset, wherein the multimodal image-text dataset includes a text review, a set of related images, and an aspect word sequence, wherein the aspect word sequence is a subsequence of the text review; S2. At the input layer, the text comments, associated images, and aspect word sequences are encoded respectively, and the context word embedding matrix, image embedding matrix, and aspect word embedding matrix are obtained accordingly; sentence-level features and overall image features are obtained; and obtaining an adjacency matrix representing syntactic dependencies based on the text comments; S3. In the multi-view feature extraction layer, according to the context word embedding matrix, aspect word features that take into account the context are obtained, and the sentence-level features and the overall image features are combined to obtain the context-aware global view features of the text and the global view features of the image; And according to the aspect word embedding matrix, the context word embedding matrix and the image embedding matrix are noise filtered based on the dynamic gated self-attention module to obtain text word order view features and image area view features; and combined with the adjacency matrix, the text syntax view feature is obtained. S4. In a feature fusion layer, the text global view feature, the image global view feature, the text word order view feature, the text syntax view feature, and the image region view feature are fused to obtain a final fused feature; S5. At the output layer, the prediction result of the aspect-level sentiment category is obtained based on the final fusion feature.

2. The multimodal aspect-level sentiment analysis method according to claim 1, wherein: In S2, the BERT model is used to encode the text comments and aspect word sequences, and the ResNet50 model is used to encode the associated images; and / or the spacy tool is used to obtain the adjacency matrix A; Among them, the BERT model outputs the word-level context word embedding matrix H S , aspect word embedding matrix H A The feature information output by the penultimate layer of the ResNet50 model is then subjected to dimensionality reduction to obtain the image embedding matrix H I ; And use the [CLS] feature information of the BERT model as sentence-level features The feature information output by the last layer of the ResNet50 model is then subjected to dimensionality reduction to obtain the overall image features.

3. The multimodal aspect-level sentiment analysis method according to claim 2, wherein: The process of acquiring the text global view feature and the image global view feature in S3 includes: According to the length of the aspect word sequence and its position in the review text, the corresponding subsequence in the context word embedding matrix is extracted Average pooling the corresponding subsequences to obtain aspect word features; in, Indicates aspect word features; n indicates aspect word index; Indicates H S The k+nth word in the aspect word sequence, k represents the number of the word before the starting position of the aspect word sequence; r refers to the length of the aspect word sequence; The feature vector obtained by adding the sentence-level feature and the aspect word feature through the self-attention mechanism is used as the global view feature of the text; in, Represents the global view feature of the text; Attention(·) is the self-attention mechanism; The feature vector obtained by adding the overall image feature and the aspect word feature through the self-attention mechanism is used as the global view feature of the image; in, Represents the global view features of the image.

4. The multimodal aspect-level sentiment analysis method according to claim 3, wherein: The process of acquiring the text word order view features, image region view features, and text syntax view features in S3 includes: constructing a gated adjustment vector according to the aspect word embedding matrix; G A =exp a nd(σ(W G ×AvgPool(H A )+b G )) (4) Among them, G A represents the gated adjustment vector; expand(·) is the dimension of the feature vector expanded along the length dimension; σ(·) is the Sigmoid activation function; AvgPool(·) is the average pooling operation; W G with b G All are trainable parameters; The gated adjustment vector is embedded in the multi-head self-attention mechanism of the dynamic gated self-attention module to dynamically adjust the attention weight calculation process between the key-query pairs, perform noise filtering, and obtain the aspect-sensitive text and image feature representation H SA and H IA ; Passing the aspect-word-sensitive text feature representation through Bi-LSTM to obtain the text word order view feature; Among them, View Seq Represents the text word order view features; Indicates H SA The node vector in ; Represent the forward LSTM network and the backward LSTM network respectively; The aspect-word-sensitive text feature representation is passed through L layers of GCN and combined with the adjacency matrix to obtain the text syntactic view feature; Among them, View Sny Represents text syntactic view features; Represents a View Sny The node vector in; ReLU(·) is the ReLU activation function; A represents the adjacency matrix; W C is a trainable parameter; represents the degree matrix of A; performing normalization adjustment on the feature representation of the image that is sensitive to the aspect words to obtain the image region view feature; View Reg =Norm(H IA ) (9) Among them, View Reg represents the image region view feature; Norm(·) represents the normalization operation.

5. The multimodal aspect-level sentiment analysis method according to claim 4, wherein: The process of obtaining the aspect-word-sensitive text and image feature representation includes: The context word embedding matrix and the image embedding matrix are used as the input of the multi-head self-attention mechanism, and the Key, Query, and Value keys are obtained through projection. in, Indicates H S Input the query, key, and value feature matrices obtained by projection after the self-attention mechanism; Indicates H I Input the query, key, and value feature matrices obtained by projection after the self-attention mechanism; According to the gate adjustment vector of the j-th attention mechanism head Get the Key feature matrix in each attention head And the Query feature matrix in, Indicates that if part of the feature vector is not activated by the gated adjustment vector, the original feature is retained; is the Hadamard dot product; Respectively represent H S 、H I Query feature matrix in the jth self-attention mechanism head; Respectively represent H S 、H I Key feature matrix in the j-th self-attention mechanism head; According to the gated adjustment vector Adjusted Key feature matrix And the Query feature matrix Obtaining text and image feature representations that are sensitive to the aspect words based on the self-attention mechanism; in, They represent the aspect-sensitive text and image feature representations output by the j-th self-attention mechanism head respectively; and is a trainable parameter; h represents the number of heads of the multi-head attention mechanism; concat represents the feature concatenation operation.

6. The multimodal aspect-level sentiment analysis method according to claim 4, wherein: A two-layer three-view decomposition high-order pooling mechanism TFH is introduced in the feature fusion layer. Each TFH layer consists of a three-view decomposition bilinear pooling mechanism TFB. Each TFB includes two stages: feature expansion and feature squeezing. The S4 includes: In the first layer TFH, the text word order view features, text syntax view features and image region view features are fused to obtain fine-grained fusion features; View F =TFH l (View Reg ,View Seq ,View Syn ) (21) =View F，1 +…+View F,l Among them, exp corresponds to the feature expansion stage, and sqz corresponds to the feature compression stage; and is a learnable parameter; Represents the fine-grained output features of the i-th layer TFB in the feature expansion stage; View F，i represents the fine-grained output feature of the i-th layer TFB in the feature compression stage; prj represents the feature projection operation, tanh represents the tanh activation function; SumPool represents the sum pooling operation; View F Represents fine-grained fusion features; In the second layer TFH, the fine-grained fusion features are further fused with the text global view features and the image global view features to obtain the final fusion features; in, and is a learnable parameter; Represents the final output feature of the i-th layer TFB in the feature expansion stage; View W，i Represents the final output feature of the i-th layer TFB in the feature compression stage; View W Represents the final fusion feature.

7. The multimodal aspect-level sentiment analysis method according to any one of claims 1 to 6, characterized in that: The S5 specifically includes: In the output layer, the final fusion feature View W After normalization by the Softmax classifier, the probability value of the corresponding aspect word sentiment category is obtained, where the emotion corresponding to the maximum probability value is used as the prediction result of the aspect-level sentiment category. The calculation process is as follows: Among them, W p is a trainable parameter, b p For bias.

8. A multimodal aspect-level sentiment analysis system based on a multi-view gated fusion network, characterized in that: The multi-view gated fusion network includes an input layer, a multi-view feature extraction layer, a feature fusion layer, and an output layer; the multimodal aspect-level sentiment analysis system includes: An acquisition module is used to acquire a multimodal image-text dataset, wherein the multimodal image-text dataset includes a text review, a set of related images, and an aspect word sequence, wherein the aspect word sequence is a subsequence of the text review; An encoding module is used to encode the text comments, associated images, and aspect word sequences at the input layer, respectively, to obtain a context word embedding matrix, an image embedding matrix, and an aspect word embedding matrix; and to obtain sentence-level features and overall image features; and obtaining an adjacency matrix representing syntactic dependencies based on the text comments; An extraction module is configured to obtain context-aware aspect word features based on the context word embedding matrix at the multi-view feature extraction layer, and to obtain context-aware global view features of the text and global view features of the image in combination with the sentence-level features and the overall image features. And according to the aspect word embedding matrix, noise filtering is performed on the context word embedding matrix and the image embedding matrix based on a dynamic gated self-attention module to obtain text word order view features and image region view features; and the text syntax view features are obtained by combining the adjacency matrix; A fusion module is used to fuse the text global view features, the image global view features, the text word order view features, the text syntax view features and the image region view features at the feature fusion layer to obtain a final fusion feature; The prediction module is used to obtain the prediction result of the aspect-level sentiment category based on the final fusion feature at the output layer.

9. A storage medium, characterized in that: It stores a computer program for multimodal aspect-level sentiment analysis based on a multi-view gated fusion network, wherein the computer program enables a computer to execute the multimodal aspect-level sentiment analysis method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including a program for executing the multimodal aspect-level sentiment analysis method according to any one of claims 1 to 7.

Images