Emotion analysis method and system based on multi-source knowledge and multi-granularity image-text features

Abstract

The invention relates to a sentiment analysis method based on multi-source knowledge and multi-granularity image-text features, which comprises the following steps of: A, acquiring text and picture data, identifying and marking aspect words and sentiment polarity thereof in the text, and forming a training set; step B, constructing a deep learning network model based on external multi-source knowledge and multi-granularity image-text features, obtaining knowledge-enhanced text feature representation and image feature representation by using an AMR image and an image tag, and obtaining a knowledge-enhanced text feature representation and a knowledge-enhanced image feature representation by using a syntactic dependency relationship of a text, a component tree structure of the text and a fine-granularity image-text relationship matrix; a multi-granularity text-visual fusion feature representation is obtained in combination with a multi-layer graph attention network; using the training set to train a deep learning network model; and C, sequentially inputting the text data and the picture data into the trained deep learning network model, extracting aspect words in the text data, and predicting the emotion polarity corresponding to the aspect words. The method and the system are beneficial to improving the accuracy of emotion polarity analysis.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically relates to a sentiment analysis method and system based on multi-source knowledge and multi-granularity graphic and text features. Background Art

[0002] Sentiment analysis, also known as opinion mining, is a major branch of natural language processing. It is the process of processing and analyzing texts with emotional colors and then determining their emotional polarity. Different from text-level sentiment analysis, aspect-level sentiment analysis transfers the relevant goal of expressing emotions from the entire document or sentence to an entity or an aspect of an entity. For example, when customers buy clothes, they need to understand the price, quality, after-sales service and other aspects of the clothes, so they need to conduct sentiment analysis on these aspects to meet the needs of customers.

[0003] Traditional sentiment analysis mainly uses text data for sentiment classification and sentiment recognition, while multimodal sentiment analysis is a technology that combines multiple modal data (such as text, images, audio, etc.) to perform sentiment analysis. With the widespread application of social media and multimedia content, more and more information appears in multimodal form, such as the combination of text and images in social media posts, and videos with audio comments. By fusing multiple data sources, it is possible to complement information from different perspectives and enhance sentiment analysis of specific aspects. For example, the emotions expressed in text information are relatively vague, while the emotional tendency of image information is relatively clear, so the user's expressed emotions can be judged more accurately, and vice versa. Therefore, the extraction and fusion of sentiment features of different modalities are crucial for the final sentiment word extraction and sentiment polarity prediction tasks.

[0004] In recent years, researchers have used DNN methods such as CNN, RNN, and LSTM to obtain feature representations of text. With the development of attention mechanisms, the combination of DNN methods and attention mechanisms has performed well in capturing semantic information; however, syntactic relations have also been ignored. Therefore, graph convolutional networks have begun to be applied to aspect-level sentiment analysis to help the model further identify sentiment information. As multimodal data has gradually become the main form of data, capturing aspect-related information from different modalities (such as text and images) for sentiment analysis has gradually become a major research issue in this field. In order to better explore the connection between text modality and image modality, researchers have interacted with image information in unimodal text sentiment analysis methods, and have achieved certain results. However, the above work still has certain limitations. For example, simply interacting text information and image information will introduce visual noise that is irrelevant to aspect words. Therefore, in order to reduce the introduction of visual noise, the fine-grained information of the image should be fully considered and aligned with the text information. Secondly, most of the existing multimodal aspect-level sentiment analysis uses graph attention networks to model the syntactic dependencies and semantic relationships of texts. Although this method has achieved remarkable results, it only considers the fine-grained relationship of the text, that is, the word-level relationship, and pays more attention to the local features of the text, without fully considering the global features of the text.

[0005] The existing multimodal sentiment analysis is to model the text information and image information separately, and then fuse them to obtain the final feature representation for sentiment analysis. Or before obtaining the feature representation, the two modal information is spliced ​​and input into the pre-trained model for modeling. The above two methods have certain problems. If the image information and text information are fused too late, the feature information of the two modalities will not be able to interact deeply. If they are fused too early, the information of the two modalities will not be sensitive enough to the information of their own modalities. Both methods may lead to the propagation of erroneous sentiment information and affect the final sentiment prediction accuracy.

[0006] In summary, the graph attention network has made certain achievements in the fusion of text and image information, but it still has shortcomings in the control of image information noise, the global features of text information and the fusion of image and text information, and has not fully utilized the global features of text and the fine-grained information of images. Summary of the invention

[0007] The purpose of the present invention is to provide a sentiment analysis method and system based on multi-source knowledge and multi-granularity graphic features, which method and system are conducive to improving the accuracy of sentiment polarity analysis.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is: a sentiment analysis method based on multi-source knowledge and multi-granularity image and text features, comprising the following steps:

[0009] Step A: Collect text and image data, identify and mark aspect words and their sentiment polarity in the text, and form a training set TS containing original text, related images, aspect items and their sentiment polarity, where the aspect item and sentiment polarity of each sample are used as true values ​​to train the model to identify and understand the emotional expression in text and images;

[0010] Step B: construct a deep learning network model G based on external multi-source knowledge and multi-granularity image-text features, wherein the deep learning network model G uses AMR images and image labels to obtain knowledge-enhanced text feature representation and image feature representation, and uses the syntactic dependency of the text, the component tree structure of the text, and the fine-grained image-text relationship matrix, combined with a multi-layer graph attention network to obtain a multi-granularity text-visual fusion feature representation; use the training set TS to train the deep learning network model G based on external multi-source knowledge and multi-granularity image-text features;

[0011] Step C: Input the text data and image data into the trained deep learning network model G in sequence, extract the aspect words in the text data, and predict the sentiment polarity corresponding to the aspect words.

[0012] Furthermore, the step B specifically includes the following steps:

[0013] Step B1: Perform syntactic dependency analysis on the text data of each sample in the training set TS to obtain the syntactic dependency relationship DA between different nodes; obtain the text division of the sentence with the help of component tree analysis to obtain the component tree structure CA of the text;

[0014] Step B2: For each sample in the training set TS, obtain the AMR image G corresponding to the text data in the sample AMR And the image label V corresponding to the image data in the sample tag ; Encode the text data and image data to obtain the text feature representation H t And image feature representation H v ; For the acquired AMR image G AMR The feature representation H of the AMR image is obtained by processing AMR , and obtain the image label V tag Processing is performed to obtain the image label feature representation H vtag ;

[0015] Step B3: Represent the text feature obtained in step B2 as H t And image feature representation H v Perform similarity calculation to obtain a fine-grained image-text relationship matrix SA; and fuse it with the syntactic dependency relationship DA and component tree structure CA obtained in step B1 to obtain a heterogeneous graph matrix EA;

[0016] Step B4: Based on the heterogeneous graph matrix EA of step B3, a multi-layer graph attention network is used to obtain the interaction of multi-granular text features and image features, and finally a multi-granular text-visual fusion feature representation is obtained.

[0017] Step B5: Based on the text feature representation H obtained in step B2 t and AMR graph feature representation H AMR , get the noun feature representation H of the text nt And use the interactive attention mechanism to fuse the noun feature representation and the AMR graph feature representation to obtain the knowledge-related noun feature representation H kt , and in order to eliminate the noise brought by the AMR graph, the knowledge-related noun feature representation H kt Perform semantic purification to obtain knowledge-enhanced text feature representation H et' ;

[0018] Step B6: Represent H based on the image feature obtained in step B2 v and image label feature representation H vtag , using the interactive attention mechanism to fuse the knowledge-related image feature representation H ev In order to reduce the semantic conflict caused by the fusion of image features with image label features, the knowledge-related image features are represented by H ev Heterogeneity elimination is performed to obtain the knowledge-enhanced image feature representation H ev' ;

[0019] Step B7: Represent the multi-granularity text-visual fusion feature obtained in step B4 The knowledge-enhanced text feature representation H obtained in step B5 et' And the knowledge-enhanced image feature representation H obtained in step B6 ev' The semantically enhanced multimodal feature representation H is obtained by fusion f , extract the aspect words of the text data in each sample and predict the sentiment polarity of the aspect words;

[0020] Step B8: Compare the predicted value and the true value obtained in step B7 to calculate the loss, and the semantically enhanced multimodal feature representation H obtained in step B7 f Perform image-text comparison learning loss, use the back-propagation algorithm to calculate the gradients of each parameter in the deep network, and use the stochastic gradient descent algorithm to update the parameters;

[0021] Step B9: When the loss value generated by the deep learning network model is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning model G is terminated.

[0022] Furthermore, the specific implementation method of step B1 is: perform syntactic dependency analysis on the text data of each sample in the training set TS to obtain the syntactic dependency relationship DA between different nodes; obtain the text division of the sentence by means of component tree analysis to obtain the component tree structure CA of the text;

[0023]

[0024] Among them, DA ij Represents the relationship between the i-th node and the j-th node of the text in the syntactic dependency tree, CA ij It represents the relationship between the i-th node and the j-th node of the text in the component tree structure. Dep.Tree represents the syntactic dependency relationship; Con.Tree represents the component tree structure;

[0025] The step B2 specifically comprises the following steps:

[0026] Step B21: For the text data T of each sample in the training set TS, use the parser APRING to obtain the AMR graph G of the text data in each sample AMR ; In order to obtain the coarse-grained information of the image, the pre-trained model ResNet_101 is used to extract the image label set V of the image data V in each sample tag , if the extracted image labels are irrelevant to the image or aspect, it will lead to error propagation problems, so 10 image labels are extracted for the image:

[0027] G AMR =APRING(T)

[0028]

[0029] Step B22: G obtained in step B21 AMR and V tag Add to the training set, that is, the training set is represented as TS = {T, V, G AMR ,V tag}, where T represents text data, V represents image data, and G AMR Indicates the AMR image corresponding to the text, V tag Indicates the image label corresponding to the image; use FasterR_CNN to encode the image data to obtain the image object feature E V , and retain the 36 objects with the highest confidence:

[0030]

[0031] Step B23: Obtain image object feature E from step B22 V And use the pre-trained model BART to obtain the text embedding representation E S, input into BART encoder for encoding, and obtain multimodal representation H tv :

[0032] E=" "+E V +" "+" <bos> "+E S +"" <eos>"

[0033] H tv =[H v ,H t ]=BART-Encoder(E)

[0034] Step B24: AMR image G obtained in step B22 AMR , use BART encoder to encode it to get AMR image feature representation H AMR :

[0035] H AMR =BART-Encoder(G AMR )

[0036] Step B25: The image label set V obtained in step B22 tag , use BART's encoder to encode it to get the image label feature representation H vtag :

[0037]

[0038] Furthermore, the step B3 specifically includes the following steps:

[0039] Step B31: Represent the text feature obtained in step B2 as H t And image feature representation H v Calculate the similarity to obtain the fine-grained image-text relationship matrix SA, and set a threshold τ. If the similarity is greater than the threshold, then there is a relationship; otherwise, there is no relationship.

[0040]

[0041] in, represents the feature representation of the i-th word in the text, represents the feature representation of the jth object in the image, is the text-guided attention score, is the image-guided attention score, τ is the threshold, and f() is the dot product operation; and By adding them together, we can get the similarity value between the i-th word in the text and the j-th object in the image. ij Represents the relationship between the i-th word in the text and the j-th object in the image;

[0042] Step B32: The syntactic dependency tree relationship DA and component tree structure CA obtained in step B1 and the fine-grained image-text relationship matrix SA obtained in step B31 are fused to obtain the heterogeneous graph matrix EA as the relationship adjacency matrix of the graph attention network.

[0043] Furthermore, the step B4 specifically includes the following steps:

[0044] Step B41: Based on the multimodal feature representation H obtained in step B2 tv And the heterogeneous graph matrix EA obtained in step B3, using the multi-layer graph attention network, obtain the feature representation of the i-th node in the l-th layer

[0045]

[0046]

[0047] in, is the adjacent node of the ith node in the heterogeneous graph matrix EA, is the final output of the i-th node in the l-th layer of the graph attention network, is the result of the masked self-attention mechanism on the i-th node of the l-th layer. is the normalized result of the similarity values ​​between the ith node and the jth node in the lth layer, LN() is the normalization layer, || represents vector concatenation, Z is the number of attention heads, σ is the activation function, is the trainable parameter of the lth layer of the zth attention head, and f() is a function that measures the similarity between two words;

[0048] Step B42: Repeat step B41 and use the previous layer as the input of the next layer, that is, Attention network layer for the second graph, resulting in By analogy, by stacking multiple layers of graph attention network layers, the granularity of the text will gradually increase from fine-grained to coarse-grained, and finally a multi-granular text-visual fusion feature representation will be obtained.

[0049] Furthermore, the step B5 specifically includes the following steps:

[0050] Step B51: Create a noun mask matrix to retain the text feature representation of the noun:

[0051]

[0052] Step B52: Based on the text feature representation H obtained in step B2 t Based on the noun mask matrix obtained in step B51, the noun feature representation H is obtained. nt :

[0053]

[0054] Step B53: The feature representation H of the AMR image obtained in step B22 AMR The AMR graph global feature representation H is obtained by further encoding through the graph neural network. GAMR :

[0055] H GAMR =GNN(H AMR )

[0056] Step B54: Based on the noun feature representation H obtained in step B52 nt The AMR image global feature representation H obtained in step B53 GAMR Perform interactive attention mechanism to obtain knowledge-related noun feature representation H kt :

[0057]

[0058]

[0059] in, represents the global feature representation of the i-th node in the AMR graph, represents the feature representation of the jth noun, and f() is a function that measures the similarity between two words;

[0060] Step B55: Based on the knowledge-related noun feature representation H obtained in step B54 kt In order to reduce the noise between multi-source information, the semantic purification module is applied to remove the noise; the knowledge-related noun feature representation H obtained in step B54 is kt and noun feature representation H nt After two fully connected layers, the semantic parameter matrix is ​​obtained:

[0061]

[0062] Among them, MLP1() and MLP2() are two different fully connected layers, f(·) represents the Leaky_ReLU activation function, and W1 and W2 are semantic parameter matrices;

[0063] Step B56: Based on the semantic parameter matrices W1 and W2 obtained in step B55, the final knowledge-enhanced noun feature representation H is obtained. kn :

[0064] H kn =W2(W1×H kt +b1)+b2

[0065] Among them, b1 and b2 are trainable parameters;

[0066] Step B57: The noun feature representation H is enhanced according to the knowledge obtained in step B56 kn And the text feature representation H obtained in step B23 t , use the multi-head attention mechanism to obtain knowledge-related text feature representation H ktn :

[0067] H ktn =MultiAtt(Q,K,V)

[0068] =Concat(head1,…,head2)W v

[0069]

[0070] Among them, Q is the text feature representation H t , K and V are knowledge-enhanced noun feature representations H kn , and W v is a trainable parameter;

[0071] Step B58: Represent the text features H based on the knowledge obtained in B57 ktn , using a gating mechanism to dynamically fuse it with the text feature representation H t Get the knowledge-enhanced text feature representation H et' :

[0072]

[0073] H et' =g×H t +(1-g)×H ktn

[0074] Among them, g is the gating coefficient automatically generated according to the features, and W3 and b3 are trainable parameters.

[0075] Furthermore, the step B6 specifically includes the following steps:

[0076] Step B61: Based on the image label feature representation H obtained in step B2 vtag , using noun features to represent H nt The guided attention mechanism dynamically learns the correlation between nouns and each image label, and obtains the noun-guided image label feature representation H vtag' :

[0077] Hv tag‘ =MultiAtt(Q,K,V)

[0078] =Concat(head1,…,head2)W v

[0079]

[0080] Among them, Q is the noun feature representation H nt , K and V are image label features representing H vtag , and W v is a trainable parameter;

[0081] Step B62: Based on the noun-guided image label feature representation H obtained in step B61 vtag' The image feature representation H obtained in step B23 v Perform interactive attention mechanism to obtain knowledge-related image feature representation H ev :

[0082]

[0083] in, represents the feature representation of the jth image object, and f() is a function that measures the similarity between two words;

[0084] Step B63: Obtain knowledge-related image feature representation H based on step B62 ev In order to reduce the semantic conflict between image labels and image features, the heterogeneity elimination module is applied to remove the semantic conflict. First, the knowledge-related image feature representation H obtained in step B62 is ev Map to high-dimensional space and obtain high-dimensional feature representation H high :

[0085] H high =f(W4H ev +b4)

[0086] Where f(·) represents the Leaky_ReLU activation function, W4 and b4 are trainable parameters;

[0087] Step B64: The high-dimensional features obtained in B63 are fed to the activation function layer, and the high-dimensional features are dynamically fused through the gating mechanism to represent H high Knowledge-related image feature representation H ev Obtain knowledge-enhanced image feature representation H ev' :

[0088] θ=σ(H high )

[0089] H ev' =W5(θ×H high +(1-θ)×H ev )+b5

[0090] Among them, σ(·) represents the sigmoid activation function used to generate the mean vector, and W5 and b5 are trainable parameters.

[0091] Furthermore, the step B7 specifically includes the following steps:

[0092] Step B71: Represent the features obtained in step B4 The knowledge-enhanced text feature representation H obtained in step B5 et' And the knowledge-enhanced image feature representation H obtained in step B6 ev' The semantically enhanced multimodal feature representation H is obtained by fusion f :

[0093]

[0094] Step B72: The semantically enhanced multimodal feature representation H obtained in step B71 f Input into the BART decoder to get the output at the tth moment

[0095]

[0096] Among them, Y <t represents the output of the Decoder at the first t moments, It represents the output of Decoder at the tth moment;

[0097] Step B73: Based on step B72, the output of the decoder at the tth moment is obtained, and the embedding representation C of the sentiment polarity is used d And step B2 obtains the text feature representation and uses softmax for classification to obtain aspect words and corresponding sentiment polarity:

[0098]

[0099] Furthermore, the step B8 specifically includes the following steps:

[0100] Step B81: Compare the predicted value obtained in step B7 with the true value to calculate the L2 regularized cross entropy loss:

[0101]

[0102] Among them, S is the number of training samples in a batchB, and C is the number of classes; is the true value of the aspect word; Θ represents all trainable parameters; λ is the coefficient of the L2 regularization term;

[0103] Step B82: The semantically enhanced multimodal feature representation H obtained in step B71 f Obtain contrastive multimodal representations for image-text contrastive learning using stacked attention networks:

[0104]

[0105]

[0106] in, is the output of the lth layer of point-by-point convolutional network, PWTcpnv(·) is the point-by-point convolutional network, and the output of the last layer in the stacked point-by-point convolutional network is

[0107] Step B83: Comparing the multimodal representations obtained in step B82 Divide into images and text and And do a picture-text comparison to learn the loss:

[0108]

[0109] In each sample is a positive sample, and is a negative sample, τ is the temperature coefficient;

[0110] Step B84: Add the losses obtained in step B81 and step B83, use the back propagation algorithm to calculate the gradient of each parameter in the deep network, and use the stochastic gradient descent algorithm to update the parameters:

[0111]

[0112] Among them, λ1 and λ2 are controllable parameters.

[0113] The present invention also provides a multimodal sentiment analysis system based on multi-source knowledge and multi-granularity graphic and text features, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0114] Compared with the prior art, the present invention has the following beneficial effects: Firstly, the present invention aims at the problem of modal alignment between different modalities. Instead of introducing the information of the whole image, the present invention uses fine-grained images and fine-grained texts to calculate the similarity, and combines the component tree to obtain different granularities of the text. The multi-granularity text-visual fusion feature representation is obtained through the multi-layer graph attention network, thereby realizing the alignment of the image modality and the text modality, and reducing the introduction of noise and redundant information irrelevant to aspect words. Secondly, the present invention introduces image labels and AMR images as external knowledge for images and texts respectively. The model can fully explore the semantic information of images and texts by using the multi-source knowledge of images and texts, and obtain the feature representation enhanced by knowledge. Finally, in order to further reduce the semantic gap between the image modality and the text modality, the present invention introduces image-text contrast learning, makes full use of the multi-granularity text-visual fusion feature representation, the knowledge-enhanced text feature representation and the knowledge-enhanced image feature representation, narrows the distance of semantic information between different modalities in the same sample, and helps the model to learn the correlation and consistency between different modalities, thereby improving the accuracy of aspect word extraction and aspect word sentiment polarity prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] Figure 1 is a flow chart of a method implementation of an embodiment of the present invention;

[0116] Figure 2 It is an architecture diagram of a deep learning network model G based on external multi-source knowledge and multi-granularity graphic features in an embodiment of the present invention. DETAILED DESCRIPTION

[0117] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0118] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0119] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0120] like Figure 1 As shown, this embodiment provides a sentiment analysis method based on multi-source knowledge and multi-granularity graphic features, including the following steps:

[0121] Step A: Collect text and image data, identify and mark aspect words and their sentiment polarity in the text, and form a training set TS containing original text, related images, aspect items and their sentiment polarity, where the aspect item and sentiment polarity of each sample are used as true values ​​to train the model to identify and understand the emotional expression in text and images;

[0122] Step B: construct a deep learning network model G based on external multi-source knowledge and multi-granularity image-text features, wherein the deep learning network model G uses AMR images and image labels to obtain knowledge-enhanced text feature representation and image feature representation, and uses the syntactic dependency of the text, the component tree structure of the text, and the fine-grained image-text relationship matrix, combined with a multi-layer graph attention network to obtain a multi-granularity text-visual fusion feature representation; use the training set TS to train the deep learning network model G based on external multi-source knowledge and multi-granularity image-text features;

[0123] Step C: Input the text data and image data into the trained deep learning network model G in sequence, extract the aspect words in the text data, and predict the sentiment polarity corresponding to the aspect words.

[0124] Figure 2 It is an architecture diagram of the deep learning network model G based on external multi-source knowledge and multi-granularity graphic features in this embodiment. In this embodiment, the specific implementation steps of step B are as follows.

[0125] Step B1: Perform syntactic dependency analysis on the text data of each sample in the training set TS to obtain the syntactic dependency relationship DA between different nodes; obtain the text division of the sentence with the help of component tree analysis to obtain the component tree structure CA of the text.

[0126] The specific implementation method of step B1 is: performing syntactic dependency analysis on the text data of each sample in the training set TS to obtain the syntactic dependency relationship DA between different nodes; obtaining the text division of the sentence by means of component tree analysis to obtain the component tree structure CA of the text;

[0127]

[0128] Among them, DA ij Represents the relationship between the i-th node and the j-th node of the text in the syntactic dependency tree, CA ij It represents the relationship between the i-th node and the j-th node of the text in the component tree structure. Dep.Tree represents the syntactic dependency relationship; Con.Tree represents the component tree structure.

[0129] Step B2: For each sample in the training set TS, obtain the AMR (Abstract Semantic Representation) graph G corresponding to the text data in the sample AMR And the image label V corresponding to the image data in the sample tag ; Encode the text data and image data to obtain the text feature representation H t And image feature representation H v ; For the acquired AMR image G AMR The feature representation H of the AMR image is obtained by processing AMR , and obtain the image label V tag Processing is performed to obtain the image label feature representation H vtag .

[0130] The step B2 specifically comprises the following steps:

[0131] Step B21: For the text data T of each sample in the training set TS, use the parser APRING to obtain the AMR graph G of the text data in each sample AMR ; In order to obtain the coarse-grained information of the image, the pre-trained model ResNet_101 is used to extract the image label set V of the image data V in each sample tag , if the extracted image labels are irrelevant to the image or aspect, it will lead to error propagation problems, so 10 image labels are extracted for the image:

[0132] G AMR =APRING(T)

[0133]

[0134] Step B22: G obtained in step B21 AMR and V tag Add to the training set, that is, the training set is represented as TS = {T, V, G AMR ,V tag }, where T represents text data, V represents image data, and G AMR Indicates the AMR image corresponding to the text, V tag Indicates the image label corresponding to the image; use FasterR_CNN to encode the image data to obtain the image object feature E V , and retain the 36 objects with the highest confidence:

[0135]

[0136] Step B23: Obtain image object feature E from step B22 V And use the pre-trained model BART to obtain the text embedding representation E S , input into BART encoder for encoding, and obtain multimodal representation H tv :

[0137] E=" "+E V +" "+" <bos> "+E S +"" <eos>"

[0138] H tv =[H v ,H t ]=BART-Encoder(E)

[0139] Step B24: AMR image G obtained in step B22 AMR , use BART encoder to encode it to get AMR image feature representation H AMR :

[0140] H AMR =BART-Encoder(G AMR )

[0141] Step B25: The image label set V obtained in step B22 tag , use BART's encoder to encode it to get the image label feature representation H vtag :

[0142]

[0143] Step B3: Represent the text feature obtained in step B2 as H t And image feature representation H v Similarity calculation is performed to obtain a fine-grained image-text relationship matrix SA; and it is fused with the syntactic dependency relationship DA and component tree structure CA obtained in step B1 to obtain a heterogeneous graph matrix EA.

[0144] The step B3 specifically comprises the following steps:

[0145] Step B31: Represent the text feature obtained in step B2 as H t And image feature representation H v Calculate the similarity to obtain the fine-grained image-text relationship matrix SA, and set a threshold τ. If the similarity is greater than the threshold, then there is a relationship; otherwise, there is no relationship.

[0146]

[0147] in, represents the feature representation of the i-th word in the text, represents the feature representation of the jth object in the image, is the text-guided attention score, is the image-guided attention score, τ is the threshold, and f() is the dot product operation; and By adding them together, we can get the similarity value between the i-th word in the text and the j-th object in the image. ij Represents the relationship between the i-th word in the text and the j-th object in the image.

[0148] Step B32: The syntactic dependency tree relationship DA and component tree structure CA obtained in step B1 and the fine-grained image-text relationship matrix SA obtained in step B31 are fused to obtain the heterogeneous graph matrix EA as the relationship adjacency matrix of the graph attention network.

[0149] Step B4: Based on the heterogeneous graph matrix EA of step B3, a multi-layer graph attention network is used to obtain the interaction of multi-granular text features and image features, and finally a multi-granular text-visual fusion feature representation is obtained.

[0150] The step B4 specifically comprises the following steps:

[0151] Step B41: Based on the multimodal feature representation H obtained in step B2 tv And the heterogeneous graph matrix EA obtained in step B3, using the multi-layer graph attention network, obtain the feature representation of the i-th node in the l-th layer

[0152]

[0153] in, is the adjacent node of the ith node in the heterogeneous graph matrix EA, is the final output of the i-th node in the l-th layer of the graph attention network, is the result of the masked self-attention mechanism on the i-th node of the l-th layer. is the normalized result of the similarity values ​​between the ith node and the jth node in the lth layer, LN() is the normalization layer, || represents vector concatenation, Z is the number of attention heads, σ is the activation function, is the trainable parameter of the lth layer of the zth attention head, and f() is a function that measures the similarity between two words.

[0154] Step B42: Repeat step B41 and use the previous layer as the input of the next layer, that is, Attention network layer for the second graph, resulting in By analogy, by stacking multiple layers of graph attention network layers, the granularity of the text will gradually increase from fine-grained to coarse-grained, and finally a multi-granular text-visual fusion feature representation will be obtained.

[0155] Step B5: Based on the text feature representation H obtained in step B2 t and AMR graph feature representation H AMR , get the noun feature representation H of the text nt And use the interactive attention mechanism to fuse the noun feature representation and the AMR graph feature representation to obtain the knowledge-related noun feature representation H kt , and in order to eliminate the noise brought by the AMR graph, the knowledge-related noun feature representation H kt Perform semantic purification to obtain knowledge-enhanced text feature representation H et' .

[0156] The step B5 specifically comprises the following steps:

[0157] Step B51: Create a noun mask matrix to retain the text feature representation of the noun:

[0158]

[0159] Step B52: Based on the text feature representation H obtained in step B2 t Based on the noun mask matrix obtained in step B51, the noun feature representation H is obtained. nt :

[0160]

[0161] Step B53: The feature representation H of the AMR image obtained in step B22 AMR The AMR graph global feature representation H is obtained by further encoding through the graph neural network. GAMR :

[0162] H GAMR =GNN(H AMR )

[0163] Step B54: Based on the noun feature representation H obtained in step B52 nt The AMR image global feature representation H obtained in step B53 GAMR Perform interactive attention mechanism to obtain knowledge-related noun feature representation H kt :

[0164]

[0165] in, represents the global feature representation of the i-th node in the AMR graph, represents the feature representation of the j-th noun, and f() is a function that measures the similarity between two words.

[0166] Step B55: Based on the knowledge-related noun feature representation H obtained in step B54 kt In order to reduce the noise between multi-source information, the semantic purification module is applied to remove the noise; the knowledge-related noun feature representation H obtained in step B54 is kt and noun feature representation H nt After two fully connected layers, the semantic parameter matrix is ​​obtained:

[0167]

[0168] Among them, MLP1() and MLP2() are two different fully connected layers, f(·) represents the Leaky_ReLU activation function, and W1 and W2 are semantic parameter matrices.

[0169] Step B56: Based on the semantic parameter matrices W1 and W2 obtained in step B55, the final knowledge-enhanced noun feature representation H is obtained. kn :

[0170] H kn =W2(W1×H kt +b1)+b2

[0171] Among them, b1 and b2 are trainable parameters.

[0172] Step B57: The noun feature representation H is enhanced according to the knowledge obtained in step B56 kn And the text feature representation H obtained in step B23 t , use the multi-head attention mechanism to obtain knowledge-related text feature representation H ktn :

[0173] H ktn =MultiAtt(Q,K,V)

[0174] =Concat(head1,…,head2)W v

[0175]

[0176] Among them, Q is the text feature representation H t , K and V are knowledge-enhanced noun feature representations H kn , and W v is a trainable parameter.

[0177] Step B58: Represent the text features H based on the knowledge obtained in B57 ktn , using a gating mechanism to dynamically fuse it with the text feature representation H t Get the knowledge-enhanced text feature representation H et' :

[0178]

[0179] H et' =g×H t +(1-g)×H ktn

[0180] Among them, g is the gating coefficient automatically generated according to the features, and W3 and b3 are trainable parameters.

[0181] Step B6: Represent H based on the image feature obtained in step B2 v and image label feature representation H vtag , using the interactive attention mechanism to fuse the knowledge-related image feature representation H ev In order to reduce the semantic conflict caused by the fusion of image features with image label features, the knowledge-related image features are represented by H ev Heterogeneity elimination is performed to obtain the knowledge-enhanced image feature representation H ev' .

[0182] The step B6 specifically comprises the following steps:

[0183] Step B61: Based on the image label feature representation H obtained in step B2 vtag , using noun features to represent H nt The guided attention mechanism dynamically learns the correlation between nouns and each image label, and obtains the noun-guided image label feature representation H vtag' :

[0184] H vtag‘ =MultiAtt(Q,K,V)

[0185] =Concat(head1,…,head2)W v

[0186]

[0187] Among them, Q is the noun feature representation H nt , K and V are image label features representing H vtag , and W v is a trainable parameter.

[0188] Step B62: Based on the noun-guided image label feature representation H obtained in step B61 vtag' The image feature representation H obtained in step B23 v Perform interactive attention mechanism to obtain knowledge-related image feature representation H ev :

[0189]

[0190] in, represents the feature representation of the jth image object, and f() is a function that measures the similarity between two words.

[0191] Step B63: Obtain knowledge-related image feature representation H based on step B62 ev In order to reduce the semantic conflict between image labels and image features, the heterogeneity elimination module is applied to remove the semantic conflict. First, the knowledge-related image feature representation H obtained in step B62 is ev Map to high-dimensional space and obtain high-dimensional feature representation H high :

[0192] H high =f(W4H ev +b4)

[0193] Among them, f(·) represents the Leaky_ReLU activation function, and W4 and b4 are trainable parameters.

[0194] Step B64: The high-dimensional features obtained in B63 are fed to the activation function layer, and the high-dimensional features are dynamically fused through the gating mechanism to represent H high Knowledge-related image feature representation H ev Obtain knowledge-enhanced image feature representation H ev' :

[0195] θ=σ(H high )

[0196] H ev' =W5(θ×H high +(1-θ)×H ev )+b5

[0197] Among them, σ(·) represents the sigmoid activation function used to generate the mean vector, and W5 and b5 are trainable parameters.

[0198] Step B7: Represent the multi-granularity text-visual fusion feature obtained in step B4 The knowledge-enhanced text feature representation H obtained in step B5 et' And the knowledge-enhanced image feature representation H obtained in step B6 ev' The semantically enhanced multimodal feature representation H is obtained by fusion f , extract the aspect words of the text data in each sample and predict the sentiment polarity of the aspect words.

[0199] The step B7 specifically includes the following steps:

[0200] Step B71: Represent the features obtained in step B4 The knowledge-enhanced text feature representation H obtained in step B5 et' And the knowledge-enhanced image feature representation H obtained in step B6 ev' The semantically enhanced multimodal feature representation H is obtained by fusion f :

[0201]

[0202] Step B72: The semantically enhanced multimodal feature representation H obtained in step B71 f Input into the BART decoder to get the output at the tth moment

[0203]

[0204] Among them, Y <t represents the output of the Decoder at the first t moments, It represents the output of Decoder at the tth moment;

[0205] Step B73: Based on step B72, the output of the decoder at the tth moment is obtained, and the embedding representation C of the sentiment polarity (positive, negative, neutral) is used d And step B2 obtains the text feature representation and uses softmax for classification to obtain aspect words and corresponding sentiment polarity:

[0206]

[0207] Step B8: Compare the predicted value and the true value obtained in step B7 to calculate the loss, and the semantically enhanced multimodal feature representation H obtained in step B7 f Perform image-text comparison learning loss, use the back-propagation algorithm to calculate the gradients of each parameter in the deep network, and use the stochastic gradient descent algorithm to update the parameters.

[0208] The step B8 specifically includes the following steps:

[0209] Step B81: Compare the predicted value obtained in step B7 with the true value to calculate the L2 regularized cross entropy loss:

[0210]

[0211] Among them, S is the number of training samples in a batchB, and C is the number of classes; is the true value of the aspect word; Θ represents all trainable parameters; λ is the coefficient of the L2 regularization term.

[0212] Step B82: The semantically enhanced multimodal feature representation H obtained in step B71 f Obtain contrastive multimodal representations for image-text contrastive learning using stacked attention networks:

[0213]

[0214]

[0215] in, is the output of the lth layer of point-by-point convolutional network, PWTconv(·) is the point-by-point convolutional network, and the output of the last layer in the stacked point-by-point convolutional network is

[0216] Step B83: Comparing the multimodal representations obtained in step B82 Divide into images and text and And do a picture-text comparison to learn the loss:

[0217]

[0218] In each sample is a positive sample, and is a negative sample, and τ is the temperature coefficient.

[0219] Step B84: Add the losses obtained in step B81 and step B83, use the back propagation algorithm to calculate the gradient of each parameter in the deep network, and use the stochastic gradient descent algorithm to update the parameters:

[0220]

[0221] Among them, λ1 and λ2 are controllable parameters.

[0222] Step B9: When the loss value generated by the deep learning network model is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning model G is terminated.

[0223] This embodiment also provides a multimodal sentiment analysis system based on multi-source knowledge and multi-granularity graphic and text features, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0224] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0225] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0226] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0227] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0228] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.< / eos> < / bos> < / eos> < / bos>

Claims

1. A sentiment analysis method based on multi-source knowledge and multi-granularity image and text features, characterized in that: The following steps are involved: Step A: Collect text and image data, identify and mark aspect words and their sentiment polarity in the text, and form a training set TS containing original text, related images, aspect items and their sentiment polarity, where the aspect item and sentiment polarity of each sample are used as true values ​​to train the model to identify and understand the emotional expression in text and images; Step B: construct a deep learning network model G based on external multi-source knowledge and multi-granularity image and text features, wherein the deep learning network model G uses AMR images and image labels to obtain knowledge-enhanced text feature representation and image feature representation, and uses the syntactic dependency of the text, the component tree structure of the text, and the fine-grained image-text relationship matrix, combined with a multi-layer graph attention network to obtain a multi-granularity text-visual fusion feature representation; Use the training set TS to train the deep learning network model G based on external multi-source knowledge and multi-granularity image and text features; Step C: Input the text data and image data into the trained deep learning network model G in sequence, extract the aspect words in the text data, and predict the sentiment polarity corresponding to the aspect words.

2. The sentiment analysis method based on multi-source knowledge and multi-granularity graphic features according to claim 1 is characterized in that: The step B specifically comprises the following steps: Step B1: Perform syntactic dependency analysis on the text data of each sample in the training set TS to obtain the syntactic dependency relationship DA between different nodes; obtain the text division of the sentence with the help of component tree analysis to obtain the component tree structure CA of the text; Step B2: For each sample in the training set TS, obtain the AMR image G corresponding to the text data in the sample AMR And the image label V corresponding to the image data in the sample tag ; Encode the text data and image data to obtain the text feature representation H t And image feature representation H v ; For the acquired AMR image G AMR The feature representation H of the AMR image is obtained by processing AMR , and obtain the image label V tag Processing is performed to obtain the image label feature representation H vtag ; Step B3: Represent the text feature obtained in step B2 as H t And image feature representation H v Perform similarity calculation to obtain a fine-grained image-text relationship matrix SA; and fuse it with the syntactic dependency relationship DA and component tree structure CA obtained in step B1 to obtain a heterogeneous graph matrix EA; Step B4: Based on the heterogeneous graph matrix EA of step B3, a multi-layer graph attention network is used to obtain the interaction of multi-granular text features and image features, and finally a multi-granular text-visual fusion feature representation is obtained. Step B5: Based on the text feature representation H obtained in step B2 t and AMR graph feature representation H AMR , get the noun feature representation H of the text nt And use the interactive attention mechanism to fuse the noun feature representation and the AMR graph feature representation to obtain the knowledge-related noun feature representation H kt , and in order to eliminate the noise brought by the AMR graph, the knowledge-related noun feature representation H kt Perform semantic purification to obtain knowledge-enhanced text feature representation H et '; Step B6: Represent H based on the image feature obtained in step B2 v and image label feature representation H vtag , using the interactive attention mechanism to fuse the knowledge-related image feature representation H ev In order to reduce the semantic conflict caused by the fusion of image features with image label features, the knowledge-related image features are represented by H ev Heterogeneity elimination is performed to obtain the knowledge-enhanced image feature representation H ev '; Step B7: Represent the multi-granularity text-visual fusion feature obtained in step B4 The knowledge-enhanced text feature representation H obtained in step B5 et ' and the knowledge-enhanced image feature representation H obtained in step B6 ev 'Fuse to obtain semantically enhanced multimodal feature representation H f , extract the aspect words of the text data in each sample and predict the sentiment polarity of the aspect words; Step B8: Compare the predicted value and the true value obtained in step B7 to calculate the loss, and the semantically enhanced multimodal feature representation H obtained in step B7 f Perform image-text comparison learning loss, use the back-propagation algorithm to calculate the gradients of each parameter in the deep network, and use the stochastic gradient descent algorithm to update the parameters; Step B9: When the loss value generated by the deep learning network model is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning model G is terminated.

3. The sentiment analysis method based on multi-source knowledge and multi-granularity graphic features according to claim 2 is characterized in that: The specific implementation method of step B1 is: performing syntactic dependency analysis on the text data of each sample in the training set TS to obtain the syntactic dependency relationship DA between different nodes; obtaining the text division of the sentence by means of component tree analysis to obtain the component tree structure CA of the text; Among them, DA ij Represents the relationship between the i-th node and the j-th node of the text in the syntactic dependency tree, CA ij It represents the relationship between the i-th node and the j-th node of the text in the component tree structure. Dep.Tree represents the syntactic dependency relationship; Con.Tree represents the component tree structure; The step B2 specifically comprises the following steps: Step B21: For the text data T of each sample in the training set TS, use the parser APRING to obtain the AMR graph G of the text data in each sample AMR ; In order to obtain the coarse-grained information of the image, the pre-trained model ResNet_101 is used to extract the image label set V of the image data V in each sample tag , if the extracted image labels are irrelevant to the image or aspect, it will lead to error propagation problems, so 10 image labels are extracted for the image: G AMR =APRING(T) Step B22: G obtained in step B21 AMR and V tag Add to the training set, that is, the training set is represented as TS = {T, V, G AMR ,V tag }, where T represents text data, V represents image data, and G AMR Indicates the AMR image corresponding to the text, V tag Indicates the image label corresponding to the image; use FasterR_CNN to encode the image data to obtain the image object feature E V , and retain the 36 objects with the highest confidence: Step B23: Obtain image object feature E from step B22 V And use the pre-trained model BART to obtain the text embedding representation E S , input into BART encoder for encoding, and obtain multimodal representation H tv : And=" "+E V +" "+" <bos> "+E S +"" <eos> "< / eos> < / bos> H tv =[H v ,H t ]=BART-Encoder(E) Step B24: AMR image G obtained in step B22 AMR , use BART encoder to encode it to get AMR image feature representation H AMR : H AMR =BART-Encoder(G AMR ) Step B25: The image label set V obtained in step B22 tag , use BART's encoder to encode it to get the image label feature representation H vtag :

4. The sentiment analysis method based on multi-source knowledge and multi-granularity graphic features according to claim 3 is characterized in that: The step B3 specifically comprises the following steps: Step B31: Represent the text feature obtained in step B2 as H t And image feature representation H v Calculate the similarity to obtain the fine-grained image-text relationship matrix SA, and set a threshold τ. If the similarity is greater than the threshold, then there is a relationship; otherwise, there is no relationship. in, represents the feature representation of the i-th word in the text, represents the feature representation of the jth object in the image, is the text-guided attention score, is the image-guided attention score, τ is the threshold, and f() is the dot product operation; and By adding them together, we can get the similarity value between the i-th word in the text and the j-th object in the image. ij Represents the relationship between the i-th word in the text and the j-th object in the image; Step B32: The syntactic dependency tree relationship DA and component tree structure CA obtained in step B1 and the fine-grained image-text relationship matrix SA obtained in step B31 are fused to obtain the heterogeneous graph matrix EA as the relationship adjacency matrix of the graph attention network.

5. The sentiment analysis method based on multi-source knowledge and multi-granularity graphic features according to claim 4 is characterized in that: The step B4 specifically comprises the following steps: Step B41: Based on the multimodal feature representation H obtained in step B2 tv And the heterogeneous graph matrix EA obtained in step B3, using the multi-layer graph attention network, obtain the feature representation of the i-th node in the l-th layer in, is the adjacent node of the ith node in the heterogeneous graph matrix EA, is the final output of the i-th node in the l-th layer of the graph attention network, is the result of the masked self-attention mechanism on the i-th node of the l-th layer. is the normalized result of the similarity values ​​between the ith node and the jth node in the lth layer, LN() is the normalization layer, || represents vector concatenation, Z is the number of attention heads, σ is the activation function, is the trainable parameter of the lth layer of the zth attention head, and f() is a function that measures the similarity between two words; Step B42: Repeat step B41 and use the previous layer as the input of the next layer, that is, Attention network layer for the second graph, resulting in By analogy, by stacking multiple layers of graph attention network layers, the granularity of the text will gradually increase from fine-grained to coarse-grained, and finally a multi-granular text-visual fusion feature representation will be obtained.

6. The sentiment analysis method based on multi-source knowledge and multi-granularity graphic features according to claim 5 is characterized in that: The step B5 specifically comprises the following steps: Step B51: Create a noun mask matrix to retain the text feature representation of the noun: Step B52: Based on the text feature representation H obtained in step B2 t Based on the noun mask matrix obtained in step B51, the noun feature representation H is obtained. nt : Step B53: The feature representation H of the AMR image obtained in step B22 AMR The AMR graph global feature representation H is obtained by further encoding through the graph neural network. GAMR : H GAMR =GNN(H AMR ) Step B54: Based on the noun feature representation H obtained in step B52 nt The global feature representation H of the AMR image obtained in step B53 GAMR Perform interactive attention mechanism to obtain knowledge-related noun feature representation H kt : in, represents the global feature representation of the i-th node in the AMR graph, represents the feature representation of the jth noun, and f() is a function that measures the similarity between two words; Step B55: Based on the knowledge-related noun feature representation H obtained in step B54 kt In order to reduce the noise between multi-source information, the semantic purification module is applied to remove the noise; the knowledge-related noun feature representation H obtained in step B54 is kt and noun feature representation H nt After two fully connected layers, the semantic parameter matrix is ​​obtained: Among them, MLP1() and MLP2() are two different fully connected layers, f(·) represents the Leaky_ReLU activation function, and W1 and W2 are semantic parameter matrices; Step B56: Based on the semantic parameter matrices W1 and W2 obtained in step B55, the final knowledge-enhanced noun feature representation H is obtained. kn : <h2 style=";text-align:left;direction:ltr">H<h2 style=";text-align:left;direction:ltr"> kn <h2 style=";text-align:left;direction:ltr"> =W2(W1×H<h2 style=";text-align:left;direction:ltr"> kt <h2 style=";text-align:left;direction:ltr"> +b1)+b2 Among them, b1 and b2 are trainable parameters; Step B57: The noun feature representation H is enhanced according to the knowledge obtained in step B56 kn And the text feature representation H obtained in step B23 t , use the multi-head attention mechanism to obtain knowledge-related text feature representation H ktn : Among them, Q is the text feature representation H t , K and V are knowledge-enhanced noun feature representations H kn , and W v is a trainable parameter; Step B58: Represent the text features H based on the knowledge obtained in B57 ktn , using a gating mechanism to dynamically fuse it with the text feature representation H t Get the knowledge-enhanced text feature representation H et ': H et' =g×H t +(1-g)×H ktn Among them, g is the gating coefficient automatically generated according to the features, and W3 and b3 are trainable parameters.

7. The sentiment analysis method based on multi-source knowledge and multi-granularity graphic features according to claim 6 is characterized in that: The step B6 specifically comprises the following steps: Step B61: Based on the image label feature representation H obtained in step B2 vtag , using noun features to represent H nt The guided attention mechanism dynamically learns the correlation between nouns and each image label, and obtains the noun-guided image label feature representation H vtag ': Among them, Q is the noun feature representation H nt , K and V are image label features representing H vtag , and W v is a trainable parameter; Step B62: Based on the noun-guided image label feature representation H obtained in step B61 vtag ' and the image feature representation H obtained in step B23 v Perform interactive attention mechanism to obtain knowledge-related image feature representation H ev : in, represents the feature representation of the jth image object, and f() is a function that measures the similarity between two words; Step B63: Obtain knowledge-related image feature representation H based on step B62 ev In order to reduce the semantic conflict between image labels and image features, the heterogeneity elimination module is applied to remove the semantic conflict. First, the knowledge-related image feature representation H obtained in step B62 is ev Map to high-dimensional space and obtain high-dimensional feature representation H high : H high =f(W4H ev +b4) Where f(·) represents the Leaky_ReLU activation function, W4 and b4 are trainable parameters; Step B64: The high-dimensional features obtained in B63 are fed to the activation function layer, and the high-dimensional features are dynamically fused through the gating mechanism to represent H high Knowledge-related image feature representation H ev Obtain knowledge-enhanced image feature representation H ev ': θ=σ(H high ) H ev' =W5(θ×H high +(1-θ)×H ev )+b5 Among them, σ(·) represents the sigmoid activation function used to generate the mean vector, and W5 and b5 are trainable parameters.

8. The sentiment analysis method based on multi-source knowledge and multi-granularity graphic features according to claim 7 is characterized in that: The step B7 specifically includes the following steps: Step B71: Represent the features obtained in step B4 The knowledge-enhanced text feature representation H obtained in step B5 et ' and the knowledge-enhanced image feature representation H obtained in step B6 ev 'Fuse to obtain semantically enhanced multimodal feature representation H f : Step B72: The semantically enhanced multimodal feature representation H obtained in step B71 f Input into the BART decoder to get the output at the tth moment Among them, Y <t represents the output of the Decoder at the first t moments, It represents the output of Decoder at the tth moment; Step B73: Based on step B72, the output of the decoder at the tth moment is obtained, and the embedding representation C of the sentiment polarity is used d And step B2 obtains the text feature representation and uses softmax for classification to obtain aspect words and corresponding sentiment polarity:

9. The sentiment analysis method based on multi-source knowledge and multi-granularity graphic features according to claim 8 is characterized in that: The step B8 specifically includes the following steps: Step B81: Compare the predicted value obtained in step B7 with the true value to calculate the L2 regularized cross entropy loss: Among them, S is the number of training samples in a batchB, and C is the number of classes; is the true value of the aspect word; Θ represents all trainable parameters; λ is the coefficient of the L2 regularization term; Step B82: The semantically enhanced multimodal feature representation H obtained in step B71 f Obtain contrastive multimodal representations for image-text contrastive learning using stacked attention networks: in, is the output of the lth layer of point-by-point convolutional network, PWTconv(·) is the point-by-point convolutional network, and the output of the last layer in the stacked point-by-point convolutional network is Step B83: Comparing the multimodal representations obtained in step B82 Divide into images and text and And do a picture-text comparison to learn the loss: In each sample is a positive sample, and is a negative sample, τ is the temperature coefficient; Step B84: Add the losses obtained in step B81 and step B83, use the back propagation algorithm to calculate the gradient of each parameter in the deep network, and use the stochastic gradient descent algorithm to update the parameters: Among them, λ1 and λ2 are controllable parameters.

10. A multimodal sentiment analysis system based on multi-source knowledge and multi-granularity graphic features, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the method steps as described in any one of claims 1 to 9 can be implemented.

Images