A multi-modal aspect-level sentiment analysis method, device, equipment and storage medium

Abstract

The application discloses a kind of multi-modal aspect-level sentiment analysis method, device, equipment and storage medium.The application includes: obtaining multi-modal input data;The multi-modal input data includes input sentence and input image;The input image is input into pre-training conversion model, and the image caption of the input image is output;The context text representation of the input sentence and the context image caption description representation of the image caption are generated;Based on attention mechanism, the context text representation and the context image caption description representation are used to generate semantic information;The syntax mask matrix is constructed using the semantic information;Aspect word representation is obtained by performing graph convolution operation on the syntax mask matrix;The aspect word representation includes text representation and image representation;The text representation and image representation are interactively predicted to obtain the sentiment classification of the multi-modal input data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sentiment analysis, and particularly relates to a multi-modal aspect-level sentiment analysis method, device, equipment and storage medium. BACKGROUND

[0002] Sentiment analysis, also known as opinion mining, aims to determine the opinions and attitudes of a speaker, author or other subject object towards a certain topic, document or event. The application of sentiment analysis involves all aspects, and its field is very broad, whether in the industrial, academic or daily life, there is sentiment analysis. At present, the related research of sentiment analysis in the single modal field has been relatively mature, among which the application in the three modalities of text, image and voice is the most widely used, and has produced a lot of impressive research results.

[0003] However, the sentiment expression ability of single modal is limited, and it is not enough to identify complex sentiment, and it may also produce ambiguity. At the same time, more and more people share daily life, express opinions and emotions through social media, and establish more extensive contact between people. On these platforms, users express their emotions and sentiments through text, images, sounds and other ways. Based on the inseparable correlation and complementarity between multi-modalities, in recent years, the research focus has also shifted from various sentiment analysis based on single modal features to multi-modal sentiment analysis considering the mutual influence between different modalities in practical application occasions.

[0004] The purpose of multi-modal sentiment analysis (MSA) is to use images, audio and text to predict overall sentiment orientation. In recent years, multi-modal sentiment analysis has attracted widespread attention in the research community. Existing research mainly focuses on two sub-tasks, including MSA in conversation and MSA in social media posts or product reviews. For MSA in conversation, previous methods mainly focus on exploring the ability to build interaction between different modalities using different neural network models (such as CNN, LSTM and GRU), and have been proven to achieve satisfactory performance in many multi-modal SA tasks (such as sentiment classification). Secondly, for MSA in social media posts or product reviews, some existing research focuses on using text labels for sentiment or user and project-oriented features for visual sentiment analysis. In addition, some subsequent research aims to combine text and images to predict the overall sentiment orientation of each multi-modal social post or product review.

[0005] Multimodal aspect-based sentiment analysis (MABSA) is a fine-grained variant of multimodal sentiment analysis, which aims to predict the sentiment polarity of opinion targets mentioned in a sentence and image pair. In one example, researchers collect a benchmark Chinese dataset for multimodal aspect-based sentiment analysis from a digital product review platform, and propose a multi-interactive memory network to iteratively fuse text and visual representations. In another example, researchers annotate two datasets on Twitter to implement multimodal aspect-based sentiment classification, and effectively combine text and visual modalities using BERT as the backbone. There are also researchers who propose a target-sensitive attention and fusion network to address text and multimodal-based target-oriented sentiment classification. In addition, researchers also introduce an input space translation architecture to construct auxiliary sentences from images and fuse these sentences with text through BERT.

[0006] However, most of the existing multimodal aspect-based sentiment analysis methods use separately pre-trained visual and text models or general pre-training tasks to extract features, but the granularity of opinion targets under the two modalities is inconsistent, resulting in visual attention sometimes failing to capture the corresponding visual representation of the target, and both text and visual modalities are not sensitive to aspect extraction and sentiment detection. At the same time, the existing methods do not effectively mine syntactic dependency information, ignoring that effective fusion of syntactic structure and semantic association can obtain more rich information. Therefore, it greatly affects the correctness of sentiment polarity judgment. In addition, the existing subtitle generation is trained in a direct way, that is, all training data are uniformly and equally input into the model, ignoring that different training images may contain different levels of information. SUMMARY

[0007] The present application provides a multimodal aspect-based sentiment analysis method, device, equipment and storage medium, which is used to solve the technical problem of low accuracy of the existing multimodal aspect-based sentiment analysis method.

[0008] The present application provides a multimodal aspect-based sentiment analysis method, comprising:

[0009] Obtaining multimodal input data; the multimodal input data comprises an input sentence and an input image;

[0010] Inputting the input image into a pre-trained conversion model to output an image caption of the input image;

[0011] Generating a context text representation of the input sentence and a context image caption description representation of the image caption;

[0012] generate semantic information based on an attention mechanism using the context text representation and the context image caption description representation;

[0013] construct a syntax mask matrix using the semantic information;

[0014] perform graph convolution operation on the syntax mask matrix to obtain aspect word representation; the aspect word representation includes text representation and image representation;

[0015] perform interactive prediction on the text representation and the image representation to obtain sentiment classification of the multi-modal input data.

[0016] Optionally, the training process of the pre-training conversion model includes:

[0017] obtain an image training data set; the image training data set includes a plurality of image-text pairs;

[0018] divide the image data set into M first sub-data sets;

[0019] train an initial conversion model using the first sub-data set;

[0020] calculate a first difficulty score of each image-text pair;

[0021] calculate a second difficulty score of each image-text pair in the first sub-data set according to the difficulty score;

[0022] sort the image-text pairs according to the second difficulty score to obtain a first sequence;

[0023] divide the first sequence into a plurality of second sub-data sets;

[0024] extract sample data from each sub-data set according to a preset number of stages to generate sample data sets of each preset stage;

[0025] train the initial conversion model using the sample data set to obtain a pre-training conversion model.

[0026] Optionally, the step of generating semantic information based on an attention mechanism using the context text representation and the context image caption description representation includes:

[0027] obtain a first weight of the context text representation;

[0028] obtain a second weight of the context image caption description representation;

[0029] calculate aspect perception attention scores and self-attention scores using the context text representation, the context image caption description representation, the first weight, and the second weight;

[0030] adding the aspect-aware attention score and the self-attention score to obtain semantic information of the multi-modal input data.

[0031] Optionally, the step of constructing the syntax mask matrix using the semantic information comprises:

[0032] obtaining distances between words in the semantic information;

[0033] generating a thresholded syntax mask matrix according to the distances;

[0034] constructing the syntax mask matrix using the thresholded syntax mask matrix and the semantic information.

[0035] The application further provides a multi-modal aspect-level sentiment analysis device, comprising:

[0036] a multi-modal input data acquisition module configured to acquire multi-modal input data; the multi-modal input data comprising an input sentence and an input image;

[0037] an image caption output module configured to input the input image into a pre-trained conversion model and output an image caption of the input image;

[0038] a context representation generation module configured to generate a context text representation of the input sentence and a context image caption description representation of the image caption;

[0039] a semantic information generation module configured to generate semantic information based on an attention mechanism using the context text representation and the context image caption description representation;

[0040] a syntax mask matrix construction module configured to construct a syntax mask matrix using the semantic information;

[0041] a graph convolution operation module configured to perform graph convolution operation on the syntax mask matrix to obtain aspect word representations; the aspect word representations comprising text representations and image representations;

[0042] an interaction prediction module configured to perform interaction prediction on the text representations and the image representations to obtain sentiment classification of the multi-modal input data.

[0043] Optionally, the training process of the pre-trained conversion model comprises:

[0044] an image training data set acquisition module configured to acquire an image training data set; the image training data set comprising a plurality of image-text pairs;

[0045] a first sub-data set division module configured to divide the image data set into M first sub-data sets;

[0046] an initial conversion model training module configured to train an initial conversion model using the first sub-data set;

[0047] a first difficulty score calculation module configured to calculate a first difficulty score of each of the image-text pairs;

[0048] a second difficulty score calculation module configured to calculate a second difficulty score of each of the image-text pairs in the first sub-data set according to the difficulty scores;

[0049] a first sequence generation module configured to sort the image-text pairs according to the second difficulty scores to obtain a first sequence;

[0050] a second sub-data set division module configured to divide the first sequence into a plurality of second sub-data sets;

[0051] a sample data set generation module configured to extract sample data from each of the sub-data sets according to a preset number of stages to generate sample data sets of the preset number of stages;

[0052] a pre-trained conversion model training module configured to train the initial conversion model using the sample data sets to obtain a pre-trained conversion model.

[0053] Optionally, the semantic information generation module comprises:

[0054] a first weight acquisition sub-module configured to acquire a first weight of the context text representation;

[0055] a second weight acquisition sub-module configured to acquire a second weight of the context image caption description representation;

[0056] an aspect-aware attention score and self-attention score calculation sub-module configured to calculate an aspect-aware attention score and a self-attention score using the context text representation, the context image caption description representation, the first weight and the second weight;

[0057] a semantic information generation sub-module configured to calculate a sum of the aspect-aware attention score and the self-attention score to obtain semantic information of the multi-modal input data.

[0058] Optionally, the syntax mask matrix construction module comprises:

[0059] a distance acquisition sub-module configured to acquire a distance between each word in the semantic information;

[0060] a syntax mask matrix generation sub-module configured to generate a syntax mask matrix with a threshold according to the distance;

[0061] The syntax mask matrix construction submodule is configured to construct a syntax mask matrix by using the syntax mask matrix with threshold and the semantic information.

[0062] The application further provides an electronic device, which comprises a processor and a memory:

[0063] The memory is configured to store program code and transmit the program code to the processor.

[0064] The processor is configured to execute the multi-modal aspect-level sentiment analysis method according to the instructions in the program code.

[0065] The application further provides a computer readable storage medium, which is configured to store program code, and the program code is configured to execute the multi-modal aspect-level sentiment analysis method.

[0066] From the above technical solutions, the application has the following advantages: the application discloses a multi-modal aspect-level sentiment analysis method, which comprises the following steps: obtaining multi-modal input data; the multi-modal input data comprises an input sentence and an input image; inputting the input image into a pre-trained conversion model to output an image caption of the input image; generating a context text representation of the input sentence and a context image caption description representation of the image caption; generating semantic information based on an attention mechanism by using the context text representation and the context image caption description representation; constructing a syntax mask matrix by using the semantic information; performing graph convolution operation on the syntax mask matrix to obtain aspect word representation; the aspect word representation comprises a text representation and an image representation; and performing interactive prediction on the text representation and the image representation to obtain sentiment classification of the multi-modal input data. The application can obtain visual information at different levels and effectively mine syntax dependency information from local to global and aspect-related semantic information, thereby improving the accuracy of multi-modal aspect-level sentiment analysis. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0068] Figure 1 A step flow chart of a multi-modal aspect-level sentiment analysis method provided by the application;

[0069] Figure 2 A Caption Transformer structure schematic diagram;

[0070] Figure 3 is a course learning strategy flowchart;

[0071] Figure 4 is a context text representation or context image caption description representation generation flowchart;

[0072] Figure 5 is a cross-attention mechanism flowchart;

[0073] Figure 6 is a structural block diagram of a multi-modal aspect-level sentiment analysis device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0074] The embodiment of the present application provides a multi-modal aspect-level sentiment analysis method, device, equipment and storage medium, and is used for solving the technical problem that the accuracy of the existing multi-modal aspect-level sentiment analysis method is low.

[0075] In order to make the application purpose, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the following described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0076] Please refer to Figure 1 , Figure 1 is a step flowchart of a multi-modal aspect-level sentiment analysis method provided by an embodiment of the present application.

[0077] The multi-modal aspect-level sentiment analysis method provided by the embodiment of the present application can specifically include the following steps:

[0078] Step 101, acquiring multi-modal input data; the multi-modal input data includes an input sentence and an input image;

[0079] Multimodal aspect-level sentiment analysis (MABSA) is a fine-grained variant task in multi-modal sentiment analysis, and the purpose is to predict the sentiment polarity of the opinion target mentioned in the sentence and image pair.

[0080] When performing multi-modal aspect-level sentiment analysis, first, multi-modal input data needs to be acquired, wherein the multi-modal input data can include an input sentence and an input image.

[0081] Step 102, inputting the input image into a pre-training conversion model to output an image caption of the input image;

[0082] Image Captioning is a cross-modal task that combines computer vision and natural language processing. The goal is to generate a syntactically and semantically correct description for a given single image, that is, a translation from image to language.

[0083] In a specific implementation, given a target T i , an input sentence S i and an image I i form a multi-modal input data Mi=(T i , S i , I i ). The input image can be encoded by a pre-trained conversion model ResNet as an image encoder. Then the encoded image is converted into an image caption A=(w1, w2, … w m ) by a pre-trained conversion model (such as Caption Transformer), and 3, H, W represent the channel number, width and height of the input image respectively. The specific process is shown in Figure 2

[0084] Given the input image , the output of the last convolutional layer is obtained as a feature map f∈C×H×W by a pre-trained model ResNet, where C=2048, and Then, 1x1 convolution is performed on the channel dimension C to reduce it to d<<C to obtain a new feature map z0, where d=256, Then, the spatial dimension is tiled, and the final feature map is represented as The position information of the feature map is enhanced by fixed positional encoding, and these feature maps are fed into a stack of DETR encoder layers. Then the decoder is used to generate non-autoregressive text by predicting the description of the input image in one forward pass.

[0085] In an example, the training process of the pre-trained conversion model includes:

[0086] S11, obtaining an image training data set; the image training data set includes a plurality of image-text pairs;

[0087] S12, dividing the image data set into M first sub-data sets;

[0088] S13, training an initial conversion model using the first sub-data set;

[0089] S14, calculating a first difficulty score of each image-text pair;

[0090] ​S15, calculating a second difficulty score of each image-text pair in the first sub-data set according to the difficulty score;

[0091] S16, ranking the image-text pairs according to the second difficulty score to obtain a first sequence;

[0092] S17, dividing the first sequence into a plurality of second sub-data sets;

[0093] S18, extracting sample data from each sub-data set according to a preset number of stages to generate sample data sets of each preset stage;

[0094] S19, training an initial conversion model using the sample data sets to obtain a pre-trained conversion model.

[0095] In a specific implementation, inspired by the learning process from easy to difficult, the embodiment of the application adopts curriculum learning as the training strategy of subtitle generation, and then introduces a cross-examination mechanism to distinguish the difficulty of the training image, and the specific process is as shown in Figure 3 .

[0096] Curriculum learning is a training strategy that simulates the learning process of human beings and advocates that the model should start learning from easy samples and gradually progress to complex samples and knowledge.

[0097] In order to obtain more comparable and stable difficulty scores, first, the image training data set is uniformly divided into M parts to construct M first sub-data sets, denoted as D i . Then, a CaptionTransformer model θ i is trained on each first sub-data set. Each model can only use training data set for training. The parameters of these models can be learned through the following optimization, and the specific formula is as follows:

[0098]

[0099] where (I, S) is an image-text pair in the i-th first sub-data set. The cross-entropy loss function between the probability prediction of θ i (I) and the corresponding ground truth title S is defined.

[0100] Since image captioning is a cross-modal matching task from image to text, the difficulty evaluation should be consistent with the corresponding evaluation indicators such as BLEU-1 and BLEU-2. Therefore, the cross-examination mechanism is adopted to determine the difficulty level of all training examples in the embodiment of the present application. After training the Caption Transformer model on M first sub-datasets, the difficulty level of each training example is evaluated. Each image-text example (I, S) ∈ D i has been seen by the model θ i during training. Therefore, another model θ k is used to evaluate the difficulty of (I, S), and the formula is as follows:

[0101] ε k (I, S) = 1 - Metric (θ k (I), S)

[0102] where ε k (I, S) is the difficulty score of the image-text pair (I, S). Metric represents a formula, which can be one of the image captioning indicators such as BLEU-1, BLEU-2, BLEU-3 and BLEU-4. The average of the BLEU-1 score is used to represent the difficulty in the model. The sum of the evaluation scores of other Caption Transformer models, and the formula is as follows:

[0103]

[0104] where DS((I, S)) is the difficulty score of the text-image pair (I, S) in the sub-dataset D i .

[0105] First, all image-texts are sorted according to the difficulty score DS, and then they are divided into M second sub-datasets U i . Then the training dataset is arranged from U1 (easiest) to U M (the most difficult). The number of samples in each category is defined as |U1|, |U2|,..., |U M |. The model adopts M stages for training, which can be defined as C i (i = 1,..., M). In each stage C i , the image-to-text examples are still shuffled to maintain local randomness, and each example in different stages does not overlap to prevent overfitting. For each learning stage C i , examples are selected from the above categories according to the difficulty in a certain proportion, and the specific number is shown in the following formula:

[0106]

[0107] When in CM When reaching the training stage, the model should be ready to train on the image samples of the entire dataset, so another learning stage C is added M+1 .

[0108] After training the Caption Transformer model through the curriculum learning strategy, it can be used to translate the input image into a natural language description of the image, generating the caption A=(w1, w2,... w m ) of the input image, where m is the number of words in the generated sentence.

[0109] Step 103, generating context text representation of input sentence and context image caption description representation of image caption;

[0110] In the embodiment of the present application, the understanding of context is crucial for sentiment analysis task. Since the pre-trained language model RoBERTa has been proven to achieve competitive performance on various NLP tasks, including aspect-based sentiment analysis (ABSA), it is used as the text encoder and image caption description encoder of the model in the embodiment of the present application, to encode the input sentence and image caption description, and construct word context representation. The specific process is as shown in Figure 4 .

[0111] Two special tokens are inserted in each input sentence, namely the token <s>Adding a marker at the end of the sentence< / s> is added at the beginning of the sentence and the token <s>< / s> <s>x1, x2,..., x n x < / s> is added at the end of the sentence. Formally, the input is represented as X=(x <s>< / s> <s>x1, x2,..., x n x < / s> , and each token in X is represented as xi, which is the sum of the word embedding and position embedding of the i-th token. The word embedding and position embedding are obtained from the word embedding matrix and position embedding matrix in the pre-trained RoBERTa model. Based on the input embedding, X is fed into the RoBERTa encoder to obtain the context text representation of the context or the context image caption description representation of the context. The formula is as follows:

[0112] (c <s>< / s> <s>, c1, c2,..., c n , c < / s> )=RoBERTa(x <s>< / s> <s>x1, x2,..., x n x < / s> )

[0113] Where C=(c1, c2,..., c n ) is the generated word context representation (context text representation or context image caption description representation), d is the dimension of the word context representation.

[0114] In step 104, based on the attention mechanism, the semantic information is generated by using the context text representation and the context image caption description representation.

[0115] Unlike the sentence-level sentiment classification task, aspect-based sentiment analysis aims to judge the sentiment of specific aspect terms in the context sentence, so it is necessary to establish a specific semantic correlation model based on different aspect terms. Therefore, the combination of aspect-aware attention and self-attention can not only learn the semantic correlation related to aspects, but also learn the global semantics of the sentence.

[0116] In one example, the step of generating semantic information based on the attention mechanism by using the context text representation and the context image caption description representation includes:

[0117] S41, obtaining a first weight of the context text representation;

[0118] S42, obtaining a second weight of the context image caption description representation;

[0119] S43, calculating aspect-aware attention scores and self-attention scores by using the context text representation, the context image caption description representation, the first weight and the second weight;

[0120] S44, calculating the sum of the aspect-aware attention scores and the self-attention scores to obtain semantic information of the multi-modal input data.

[0121] In a specific implementation, the aspect-aware attention score calculation formula is as follows:

[0122]

[0123] Wherein, the aspect word is used as Q1, K and V are equal to the text representation or image caption representation generated by the feature extraction layer, and are learnable weights. The model uses the aspect-aware attention of the P head to obtain an aspect-aware attention score matrix of a sentence, wherein, represents the aspect-aware attention score obtained by the i-th attention head.

[0124] The self-attention score calculation formula is as follows:

[0125]

[0126] Wherein, Q2, K and V are equal to the text representation or image caption representation generated by the feature extraction layer, and are learnable weights.

[0127] Then the aspect-aware attention score is combined with the self-attention score, as shown in the following formula, to obtain semantic information of the multi-modal input data.

[0128]

[0129] wherein, Used as input for the syntax mask layer computation that follows. For each A i It represents a fully connected graph.

[0130] Step 105, constructing a syntax mask matrix using the semantic information;

[0131] Syntax mask matrix is a technique used in natural language processing to identify and extract grammatical information from sentences. It converts each sentence into a matrix where each word corresponds to its grammatical role. Syntax mask matrix can make it easier to perform syntactic analysis, named entity recognition, and other natural language processing tasks.

[0132] In the embodiments of the present application, step 105 can include the following sub-steps:

[0133] S51, obtaining the distance between each word in the semantic information;

[0134] S52, generating a thresholded syntax mask matrix according to the distance;

[0135] S53, constructing a syntax mask matrix using the thresholded syntax mask matrix and the semantic information.

[0136] In specific implementations, in the syntax mask layer, the syntax dependency tree is regarded as an undirected graph, and each token is regarded as a node. Then, the distance between nodes v i and v j is defined as d(v i , v j ). Since there are multiple paths between nodes on the syntax dependency tree, the distance of the shortest path can be defined as D, as follows:

[0137] D(i, j) = min d(v i , v j )

[0138] An attention mechanism is applied to find the words most relevant to each aspect. However, if attention is applied directly, each aspect will inevitably receive the highest attention score. To mitigate this issue, a mask is introduced. A syntactic mask matrix for the sentence is constructed based on the different syntactic distances between words. In the P-head attention mechanism described earlier, P adjacency matrices can be obtained. Therefore, the number of syntactic mask matrices based on different syntactic distances can be set to be the same as the number of attention heads. When the syntactic distance is small, the model can learn local information; conversely, if the syntactic distance is relatively large, global structural information will be considered. The syntactic mask matrix M with a threshold k is then used. k The calculation is shown in the following formula:

[0139]

[0140] Where k∈[1, p]. To obtain both global information and local features, the attention scope is limited by different syntactic distances:

[0141] M = {M 1 M k M p}

[0142]

[0143] Wherein, the syntactic mask matrix based on distance i is denoted as

[0144] Step 106: Perform graph convolution operation on the syntactic mask matrix to obtain aspect word representations; aspect word representations include text representations and image representations;

[0145] Since there are P distinct syntax mask matrices, it is necessary to... Perform P-fold graph convolution operations. Then h... l-1 Let h represent the input state of the l-th layer. 1 Represented as the output state of the l-th layer, h 0 This refers to the output of the sentence encoding layer. Each node in the l-th GCN layer is updated based on the hidden representations of its neighborhood, as shown in the following formula.

[0146]

[0147] Among them, W l It is a linear transformation weight, b l σ is the bias term, and σ is a nonlinear function. The final output of an l-layer GCN is represented as... After summing the node representations from each layer, the final feature representation can be obtained. Non-aspectual terms in the output representation learned by the GCN layer are masked out to obtain aspectual term representations (including text and image representations). Furthermore, average pooling is used to preserve most of the information in the aspectual term representations. As shown in the formula below:

[0148]

[0149] Here, f is an average pooling function, suitable for the augmented aspect representation of the GCN layer.

[0150] Step 107: Perform interactive prediction on the text representation and image representation to obtain the sentiment classification of the multimodal input data.

[0151] In this embodiment of the invention, after obtaining the text representation and the image representation, interactive prediction can be performed on the text representation and the image representation to obtain the sentiment classification of the multimodal input data.

[0152] In specific implementations, such as Figure 5 As shown, there are two existing features representing H. T and H I The model learns the interaction between text and image captions by applying a cross-attention mechanism. The text is represented as H... T As query Q, the image representation H I The purpose of using K as the key and V as the value is to automatically learn the attention weights for each region in the image given each word as a query, adaptively assigning higher attention values ​​to more important features, and thus obtaining the final representation. The formula is as follows:

[0153]

[0154] Where Q3 represents text, K1 and V1 represent image representations, and W... q W k W v It is the weight matrix of query, key, and value in cross-attention.

[0155] The model further stacks two sub-layers on top of the fused representation, as shown in the following formula:

[0156]

[0157]

[0158] Where LN represents the normalization of the layer, and FFN represents the feedforward network.

[0159] Then, H fThe linear layer is fed into, and the sentiment probability y for a specific target is obtained by using a Softmax function.

[0160] y = Softmax(W a H f +b)

[0161] where W a and b are learnable weights and biases.

[0162] The model is trained by using a cross-entropy error method, and the loss function is as follows:

[0163]

[0164] After the sentiment probability y for a specific target is calculated, the sentiment classification of the specific target can be determined according to the sentiment probability.

[0165] The subtitle generation module of the embodiment of the application combines a curriculum learning strategy, and introduces a cross-examination mechanism to distinguish the difficulty of the training image. The curriculum learning strategy from easy to difficult greatly improves the ability of the subtitle generation module and obtains more rich visual information. In addition, the embodiment of the application constructs a syntactic mask matrix calculated by different distances between words in the syntactic dependency structure of a sentence to learn the structure information from the local to the whole. The adjacency matrix is combined with the syntactic mask matrix to enhance the graph convolution network. Aspect-oriented features are obtained through multi-layer graph convolution operation, which are used for aspect-level sentiment classification, can effectively mine the syntactic dependency information from the local to the whole and the aspect-related semantic information, and thus improve the accuracy of the multi-modal aspect-level sentiment analysis.

[0166] Please refer to Figure 6 , Figure 6 The structural block diagram of a multi-modal aspect-level sentiment analysis device provided by the embodiment of the application is shown.

[0167] The embodiment of the application provides a multi-modal aspect-level sentiment analysis device, which comprises:

[0168] A multi-modal input data acquisition module 601 is configured to acquire multi-modal input data, wherein the multi-modal input data comprises an input sentence and an input image.

[0169] An image subtitle output module 602 is configured to input the input image into a pre-trained conversion model and output an image subtitle of the input image.

[0170] A context representation generation module 603 is configured to generate a context text representation of the input sentence and a context image subtitle description representation of the image subtitle.

[0171] The semantic information generation module 604 is configured to generate semantic information based on an attention mechanism, using the context text representation and the context image caption description representation.

[0172] The syntax mask matrix construction module 605 is configured to construct a syntax mask matrix using the semantic information.

[0173] The graph convolution operation module 606 is configured to perform graph convolution operation on the syntax mask matrix to obtain aspect word representation; the aspect word representation includes text representation and image representation.

[0174] The interaction prediction module 607 is configured to perform interaction prediction on the text representation and the image representation to obtain sentiment classification of the multi-modal input data.

[0175] In the embodiment of the application, the training process of the pre-training conversion model includes:

[0176] The image training data set acquisition module is configured to acquire an image training data set; the image training data set includes a plurality of image-text pairs.

[0177] The first sub-data set division module is configured to divide the image data set into M first sub-data sets.

[0178] The initial conversion model training module is configured to train an initial conversion model using the first sub-data set.

[0179] The first difficulty score calculation module is configured to calculate a first difficulty score of each image-text pair.

[0180] The second difficulty score calculation module is configured to calculate a second difficulty score of each image-text pair in the first sub-data set according to the difficulty score.

[0181] The first sequence generation module is configured to sort the image-text pairs according to the second difficulty score to obtain a first sequence.

[0182] The second sub-data set division module is configured to divide the first sequence into a plurality of second sub-data sets.

[0183] The sample data set generation module is configured to extract sample data from each sub-data set according to a preset number of stages to generate sample data sets of each preset stage.

[0184] The pre-training conversion model training module is configured to train the initial conversion model using the sample data set to obtain a pre-training conversion model.

[0185] In the embodiment of the application, the semantic information generation module 604 includes:

[0186] The first weight acquisition sub-module is configured to acquire a first weight of the context text representation.

[0187] a second weight obtaining sub-module, configured to obtain a second weight of the context image caption description representation;

[0188] an aspect-aware attention score and self-attention score calculating sub-module, configured to calculate the aspect-aware attention score and the self-attention score by using the context text representation, the context image caption description representation, the first weight and the second weight;

[0189] a semantic information generating sub-module, configured to calculate a sum of the aspect-aware attention score and the self-attention score to obtain semantic information of the multi-modal input data.

[0190] In the embodiment of the present application, the syntax mask matrix constructing module 605 comprises:

[0191] a distance obtaining sub-module, configured to obtain distances between words in the semantic information;

[0192] a syntax mask matrix generating sub-module with threshold, configured to generate a syntax mask matrix with threshold according to the distances;

[0193] a syntax mask matrix constructing sub-module, configured to construct a syntax mask matrix by using the syntax mask matrix with threshold and the semantic information.

[0194] The embodiment of the present application further provides an electronic device, which comprises a processor and a memory:

[0195] The memory is configured to store program code and transmit the program code to the processor;

[0196] The processor is configured to execute the multi-modal aspect-level sentiment analysis method according to the instructions in the program code.

[0197] The embodiment of the present application further provides a computer readable storage medium, which is configured to store program code, and the program code is configured to execute the multi-modal aspect-level sentiment analysis method.

[0198] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, and will not be described here.

[0199] Each embodiment in the specification describes in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to.

[0200] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, apparatus, or computer program product. Accordingly, embodiments of the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer program instructions.

[0201] Embodiments of the present application are described herein with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0202] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing terminal apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal apparatus to cause a series of operational steps to be performed on the computer or other programmable terminal apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0204] While preferred embodiments of the present application have been described, modifications and alterations thereto will occur to those skilled in the art upon reading the preceding description. In particular, it will be apparent to those skilled in the art that parts can be added to, or substituted for, parts of the described embodiments of the present application. Accordingly, the application is intended to be

[0205] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be used to distinguish one entity or action from another entity or action, without necessarily requiring or implying that there is any such actual relationship or order between these entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof are intended to cover non-exclusive inclusions, so that a process, method, article, or terminal device including a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article, or terminal device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article, or terminal device including the element.

[0206] The above description and the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-modal aspect-level sentiment analysis method, characterized in that, The method comprises the following steps: obtaining multi-modal input data; the multi-modal input data comprises an input sentence and an input image; inputting the input image into a pre-trained conversion model to output an image caption of the input image; generating a context text representation of the input sentence and a context image caption description representation of the image caption; generating semantic information based on an attention mechanism using the context text representation and the context image caption description representation; constructing a syntax mask matrix using the semantic information; performing graph convolution operation on the syntax mask matrix to obtain aspect word representation; the aspect word representation comprises a text representation and an image representation; performing interactive prediction on the text representation and the image representation to obtain sentiment classification of the multi-modal input data; wherein the training process of the pre-trained conversion model comprises: obtaining an image training data set; the image training data set comprises a plurality of image-text pairs; dividing the image training data set into M first sub-data sets; training an initial conversion model using the first sub-data set; calculating a first difficulty score of each image-text pair; calculating a second difficulty score of each image-text pair in the first sub-data set according to the difficulty score; sorting the image-text pairs according to the second difficulty score to obtain a first sequence; dividing the first sequence into a plurality of second sub-data sets; extracting sample data from each sub-data set according to a preset number of stages to generate sample data sets of each preset stage; training the initial conversion model using the sample data set to obtain a pre-trained conversion model.

2. The method of claim 1, wherein, The step of generating semantic information based on an attention mechanism using the context text representation and the context image caption description representation comprises: obtaining a first weight of the context text representation; obtaining a second weight of the context image caption description representation; calculating aspect perception attention scores and self-attention scores using the context text representation, the context image caption description representation, the first weight and the second weight; calculating the sum of the aspect perception attention scores and the self-attention scores to obtain the semantic information of the multi-modal input data.

3. The method of claim 1, wherein, The step of constructing a syntax mask matrix using the semantic information comprises: obtaining the distance between each word in the semantic information; generating a threshold syntax mask matrix according to the distance; constructing a syntax mask matrix using the threshold syntax mask matrix and the semantic information.

4. A multi-modal aspect-level sentiment analysis apparatus, characterized in that, The method comprises the following steps: a multi-modal input data acquisition module for acquiring multi-modal input data; the multi-modal input data comprises an input sentence and an input image; an image caption output module for inputting the input image into a pre-trained conversion model to output an image caption of the input image; a context representation generation module for generating a context text representation of the input sentence and a context image caption description representation of the image caption; a semantic information generation module for generating semantic information based on an attention mechanism using the context text representation and the context image caption description representation; The syntax mask matrix construction module is configured to construct a syntax mask matrix based on the semantic information. The graph convolution operation module is configured to perform graph convolution operation on the syntax mask matrix to obtain aspect word representation. The interaction prediction module is configured to perform interaction prediction on the text representation and the image representation to obtain sentiment classification of the multi-modal input data. The training process of the pre-training conversion model includes: The image training data set acquisition module is configured to acquire an image training data set. The first sub-data set division module is configured to divide the image training data set into M first sub-data sets. The initial conversion model training module is configured to train an initial conversion model based on the first sub-data sets. The first difficulty score calculation module is configured to calculate a first difficulty score of each image-text pair. The second difficulty score calculation module is configured to calculate a second difficulty score of each image-text pair in the first sub-data set based on the difficulty score. The first sequence generation module is configured to sort the image-text pairs based on the second difficulty score to obtain a first sequence. The second sub-data set division module is configured to divide the first sequence into a plurality of second sub-data sets. The sample data set generation module is configured to extract sample data from each sub-data set according to a preset number of stages to generate sample data sets of each preset stage. The pre-training conversion model training module is configured to train the initial conversion model based on the sample data sets to obtain a pre-training conversion model.

5. The apparatus of claim 4, wherein, The semantic information generation module includes: The first weight acquisition submodule is configured to acquire a first weight of the context text representation. The second weight acquisition submodule is configured to acquire a second weight of the context image caption description representation. The aspect-aware attention score and self-attention score calculation submodule is configured to calculate an aspect-aware attention score and a self-attention score based on the context text representation, the context image caption description representation, the first weight, and the second weight. The semantic information generation submodule is configured to calculate the sum of the aspect-aware attention score and the self-attention score to obtain the semantic information of the multi-modal input data.

6. The apparatus of claim 4, wherein, The syntax mask matrix construction module includes: The distance acquisition submodule is configured to acquire the distance between each word in the semantic information. The threshold-based syntax mask matrix generation submodule is configured to generate a threshold-based syntax mask matrix based on the distance. The syntax mask matrix construction submodule is configured to construct a syntax mask matrix based on the threshold-based syntax mask matrix and the semantic information.

7. An electronic device, comprising: The device includes a processor and a memory: The memory is configured to store program code and transmit the program code to the processor. The processor is configured to execute the multi-modal aspect-level sentiment analysis method according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program code for performing the multi-modal aspect-level sentiment analysis method according to any one of claims 1-3.

Images