Sentiment analysis method and system based on multi-source knowledge and multi-granularity image-text features

Abstract

The application relates to a sentiment analysis method based on multi-source knowledge and multi-granularity image-text features, which comprises the following steps: step A: collecting text and picture data, identifying and marking aspect words and their sentiment polarities in the text to form a training set; step B: constructing a deep learning network model based on external multi-source knowledge and multi-granularity image-text features, obtaining knowledge-enhanced text feature representation and image feature representation by using AMR graphs and image labels, and obtaining multi-granularity text-visual fusion feature representation by using the syntactic dependency relationship of the text, the component tree structure of the text and the fine-grained image-text relationship matrix in combination with a multi-layer graph attention network; training the deep learning network model by using the training set; and step C: inputting the text data and the picture data into the trained deep learning network model in sequence, extracting aspect words in the text data, and predicting the sentiment polarities corresponding to the aspect words. The method and system are beneficial to improving the accuracy of sentiment polarity analysis.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, specifically relating to a sentiment analysis method and system based on multi-source knowledge and multi-granularity image and text features. Background Technology

[0002] Sentiment analysis, also known as opinion mining, is a major branch of natural language processing. It involves processing and analyzing text with emotional connotations to determine their emotional polarity. Unlike text-level sentiment analysis, aspect-level sentiment analysis shifts the focus of emotion expression from the entire document or sentence to a single entity or a specific aspect of an entity. For example, when a customer buys clothes, they need to understand multiple aspects such as price, quality, and after-sales service. Sentiment analysis needs to be performed on each of these aspects to meet the customer's needs.

[0003] Traditional sentiment analysis primarily uses text data for sentiment classification and recognition, while multimodal sentiment analysis combines data from multiple modalities (such as text, images, and audio) for sentiment analysis. With the widespread use of social media and multimedia content, more and more information appears in multimodal forms, such as combinations of text and images in social media posts, and videos in audio comments. By fusing multiple data sources, information from different perspectives can be complementary, enhancing sentiment analysis of specific aspects. For example, textual information expresses relatively ambiguous emotions, while image information has a clearer emotional tendency, allowing for a more accurate determination of the user's expressed emotions, and vice versa. Therefore, the extraction and fusion methods of sentiment features from different modalities are crucial for the final tasks of sentiment word extraction and sentiment polarity prediction.

[0004] In recent years, researchers have employed DNN methods such as CNNs, RNNs, and LSTMs to obtain feature representations of text. With the development of attention mechanisms, the combination of DNN methods and attention mechanisms has shown excellent performance in capturing semantic information; however, it has neglected syntactic relations. Therefore, graph convolutional networks have begun to be applied to aspect-level sentiment analysis to help models further identify sentiment information. As multimodal data gradually becomes the main form of data, capturing aspect-related information from different modalities (such as text and images) for sentiment analysis has gradually become a major research problem in this field. To better explore the relationship between text and image modalities, researchers have interacted with image information in unimodal text sentiment analysis methods, achieving some results. However, the above work still has certain limitations. For example, simply interacting with text and image information can introduce visual noise that is irrelevant to aspect terms. Therefore, to reduce the introduction of visual noise, the fine-grained information of the image should be fully considered and aligned with the text information. Secondly, most existing multimodal aspect-level sentiment analysis uses graph attention networks to model the syntactic dependencies and semantic relationships of text. Although this method has achieved significant results, it only considers the fine-grained relationships of the text, i.e., word-level relationships, and focuses more on the local features of the text, without fully considering the global features of the text.

[0005] Current multimodal sentiment analysis methods involve modeling textual and image information separately and then fusing them to obtain the final feature representation for sentiment analysis. Alternatively, the two modalities can be concatenated before obtaining feature representations and then fed into a pre-trained model. Both of these methods have certain problems. If image and text information are fused too late, the feature information from the two modalities cannot interact deeply. If they are fused too early, the information from each modality may not be sensitive enough to its own modality. Both approaches can lead to the propagation of incorrect sentiment information, affecting the accuracy of the final sentiment prediction.

[0006] In summary, graph attention networks have achieved certain successes in fusing text and image information, but they still have shortcomings in controlling image noise, the global features of text information, and the fusion of text and image information. They have not made full use of the global features of text and the fine-grained information of images. Summary of the Invention

[0007] The purpose of this invention is to provide a sentiment analysis method and system based on multi-source knowledge and multi-granularity image and text features, which is beneficial to improving the accuracy of sentiment polarity analysis.

[0008] To achieve the above objectives, the technical solution adopted by this invention is: a sentiment analysis method based on multi-source knowledge and multi-granularity image and text features, comprising the following steps:

[0009] Step A: Collect text and image data, identify and label aspect terms and their sentiment polarities in the text, and form a training set TS containing the original text, related images, aspect terms and their sentiment polarities. The aspect terms and sentiment polarities of each sample are used as ground truth values ​​to train the model to identify and understand the sentiment expressions in text and images.

[0010] Step B: Construct a deep learning network model G based on external multi-source knowledge and multi-granularity image-text features. The deep learning network model G uses AMR graphs and image labels to obtain knowledge-enhanced text feature representations and image feature representations, and uses the syntactic dependencies of the text, the component tree structure of the text, and the fine-grained image-text relationship matrix, combined with a multi-layer graph attention network to obtain multi-granularity text-visual fusion feature representations; train the deep learning network model G based on external multi-source knowledge and multi-granularity image-text features using the training set TS.

[0011] Step C: Input the text data and image data into the trained deep learning network model G in sequence, extract the aspect words in the text data, and predict the sentiment polarity corresponding to the aspect words.

[0012] Furthermore, step B specifically includes the following steps:

[0013] Step B1: Perform syntactic dependency parsing on the text data of each sample in the training set TS to obtain the syntactic dependency relationship DA between different nodes; use the constituent tree parsing to obtain the text partition of the sentence and obtain the constituent tree structure CA of the text.

[0014] Step B2: For each sample in the training set TS, obtain the AMR image G corresponding to the text data in the sample. AMR And the image labels V corresponding to the image data in the sample. tag Encode the text and image data to obtain the text feature representation H. t and image feature representation H v ; for the obtained AMR image G AMR The feature representation H of the AMR image is obtained through processing. AMR and the obtained image label V tag The image label feature representation H is obtained through processing. vtag ;

[0015] Step B3: Represent the text features H obtained in step B2 t and image feature representation H v Similarity calculations are performed to obtain the fine-grained image-text relationship matrix SA; and it is fused with the syntactic dependency relationship DA and the constituent tree structure CA obtained in step B1 to obtain the heterogeneous graph matrix EA;

[0016] Step B4: Based on the heterogeneous graph matrix EA from Step B3, a multi-layer graph attention network is used to obtain the interaction between multi-granularity text features and image features, ultimately obtaining a multi-granularity text-visual fusion feature representation.

[0017] Step B5: Based on the text feature representation H obtained in step B2 t and AMR plot feature representation H AMR Obtain the noun feature representation H of the text. nt And an interactive attention mechanism is used to fuse noun feature representations and AMR graph feature representations to obtain knowledge-related noun feature representations H. kt Furthermore, to eliminate noise from AMR graphs, the knowledge-related noun feature representation H... kt Semantic cleansing yields knowledge-enhanced text feature representations H. et' ;

[0018] Step B6: Based on the image feature representation H obtained in step B2 v Image label feature representation H vtag The interactive attention mechanism is used to fuse and obtain knowledge-related image feature representations H. ev Furthermore, to reduce semantic conflicts arising from the fusion of image features and image label features, the knowledge-related image feature representation H is optimized. ev Heterogeneity resolution yields knowledge-enhanced image feature representations H. ev' ;

[0019] Step B7: Represent the multi-granularity text-visual fusion features obtained in Step B4 The knowledge-enhanced text feature representation H obtained in step B5 et' The knowledge-enhanced image feature representation H obtained in step B6 ev' The fusion process yields a semantically enhanced multimodal feature representation H. f Extract aspect words from the text data of each sample and predict the sentiment polarity of those aspect words;

[0020] Step B8: Compare the predicted values ​​and true values ​​obtained in Step B7 to calculate the loss, and perform semantically enhanced multimodal feature representation H obtained in Step B7. f We employ a text-image comparison learning loss algorithm, use backpropagation to calculate the gradients of each parameter in the deep network, and then use stochastic gradient descent to update the parameters.

[0021] Step B9: When the loss value generated by the deep learning network model is less than the set threshold or the maximum number of iterations is reached, terminate the training of the deep learning model G.

[0022] Further, the specific implementation method of step B1 is as follows: perform syntactic dependency parsing on the text data of each sample in the training set TS to obtain the syntactic dependency relationship DA between different nodes; use the component tree parsing to obtain the text segmentation of the sentence and obtain the component tree structure CA of the text.

[0023]

[0024] Among them, DA ij This represents the relationship between the i-th node and the j-th node of the text in the syntactic dependency tree, CA. ij Dep.Tree represents the relationship between the i-th node and the j-th node in the constituent tree structure of the text; Dep.Tree represents the syntactic dependency relationship; Con.Tree represents the constituent tree structure.

[0025] Step B2 specifically includes the following steps:

[0026] Step B21: For the text data T of each sample in the training set TS, use the parser APRING to obtain the AMR graph G of the text data in each sample. AMR To obtain coarse-grained information about the images, a pre-trained model ResNet_101 is used to extract the image label set V from the image data V of each sample. tag If the extracted image labels are unrelated to the image or aspect, it will lead to error propagation problems. Therefore, 10 image labels are extracted for each image:

[0027] G AMR =APRING(T)

[0028]

[0029] Step B22: Take the G obtained in step B21 AMR and V tag Add to the training set, i.e., the training set is represented as TS = {T, V, G}. AMR V tag}, where T represents text data, V represents image data, and G represents text data. AMR V represents the AMR diagram corresponding to the text. tag The image label represents the image corresponding to the image; Faster R-CNN is used to encode the image data to obtain the image object features E. V And retain the 36 objects with the highest confidence:

[0030]

[0031] Step B23: Obtain image object features E from step B22. V And using the pre-trained BART model to obtain the text embedding representation E SThe input is encoded by the BART encoder to obtain the multimodal representation H. tv :

[0032] E = " +E V +" "+" <bos> "+E S +"" <eos>"

[0033] H tv =[H v H t = BART-Encoder(E)

[0034] Step B24: Process the AMR image G obtained in step B22. AMR The AMR map feature representation H is obtained by encoding it using a BART encoder. AMR :

[0035] H AMR =BART-Encoder(G AMR )

[0036] Step B25: Process the image label set V obtained in step B22. tag The image label feature representation H is obtained by encoding it using a BART encoder. vtag :

[0037]

[0038] Furthermore, step B3 specifically includes the following steps:

[0039] Step B31: Represent the text features H obtained in step B2 t and image feature representation H v Similarity is calculated to obtain a fine-grained image-text relationship matrix SA, and a threshold τ is set. If the similarity is greater than the threshold, the images are related; otherwise, they are not related.

[0040]

[0041] in, This represents the feature representation of the i-th word in the text. This represents the feature representation of the j-th object in the image. Attention score guided by text. Let τ be the image-guided attention score, τ be the threshold, and f() be the dot product operation; for and Adding them together gives the similarity value (SA) between the i-th word of the text and the j-th object of the image. ij This represents the relationship between the i-th word of the text and the j-th object of the image;

[0042] Step B32: Fuse the syntactic dependency tree relation DA and constituent tree structure CA obtained in step B1 with the fine-grained image-text relation matrix SA obtained in step B31 to obtain the heterogeneous graph matrix EA, which serves as the relation adjacency matrix of the graph attention network.

[0043] Furthermore, step B4 specifically includes the following steps:

[0044] Step B41: Based on the multimodal feature representation H obtained in step B2 tv Using the heterogeneous graph matrix EA obtained in step B3, a multi-layer graph attention network is used to obtain the feature representation of the i-th node in the l-th layer.

[0045]

[0046]

[0047] in, It is the adjacent node of the i-th node in the heterogeneous graph matrix EA. It is the final output of the i-th node in the l-th layer of the graph attention network. It is the result of the i-th node in layer l after passing through the masked self-attention mechanism. This is the normalized similarity value between the i-th node and the j-th node in the l-th layer. LN() is the normalization layer, || denotes vector concatenation, Z is the number of attention heads, and σ is the activation function. is the trainable parameter of the l-th layer of the z-th attention head, and f() is a function that measures the similarity between two words;

[0048] Step B42: Repeat step B41, using the previous layer as the input for the next layer, i.e. For the input of the second graph attention network layer, thus obtaining By stacking multiple layers of graph attention network layers, the granularity of the text gradually increases from fine to coarse, ultimately resulting in a multi-granularity text-visual fusion feature representation.

[0049] Furthermore, step B5 specifically includes the following steps:

[0050] Step B51: Construct a noun mask matrix, preserving the textual feature representation of the nouns:

[0051]

[0052] Step B52: Based on the text feature representation H obtained in step B2 t Based on the noun mask matrix obtained in step B51, the noun feature representation H is obtained. nt :

[0053]

[0054] Step B53: Represent the characteristic features of the AMR image obtained in step B22. AMR Further encoding using a graph neural network yields the global feature representation H of the AMR graph. GAMR :

[0055] H GAMR =GNN(H AMR )

[0056] Step B54: Based on the noun feature representation H obtained in step B52 nt The global feature representation H of the AMR map obtained in step B53 GAMR We implement an interactive attention mechanism to obtain knowledge-related noun feature representations H. kt :

[0057]

[0058]

[0059] in, Let represent the global feature representation of the i-th node in the AMR graph. Let f() represent the feature representation of the j-th noun, and f() be a function that measures the similarity between two words;

[0060] Step B55: Based on the knowledge-related noun feature representation H obtained in step B54 kt To reduce noise among multi-source information, a semantic cleanup module is applied to remove noise; the knowledge-related noun feature representation H obtained in step B54 is then processed. kt and noun features represent H nt The semantic parameter matrix is ​​obtained after two fully connected layers:

[0061]

[0062] Where MLP1() and MLP2() are two different fully connected layers, f(·) represents the Leaky_ReLU activation function, and W1 and W2 are semantic parameter matrices;

[0063] Step B56: Based on the semantic parameter matrices W1 and W2 obtained in step B55, obtain the final knowledge-enhanced noun feature representation H. kn :

[0064] H kn =W2(W1×H kt +b1)+b2

[0065] Here, b1 and b2 are trainable parameters;

[0066] Step B57: Enhance the noun feature representation H based on the knowledge obtained in Step B56. kn The text feature representation H obtained in step B23 t Using a multi-head attention mechanism to obtain knowledge-related text feature representations H ktn :

[0067] H ktn =MultiAtt(Q,K,V)

[0068] =Concat(head1,…,head2)W v

[0069]

[0070] Where Q represents the text feature representation H t K and V are the noun feature representations for knowledge enhancement H. kn , and W v These are trainable parameters;

[0071] Step B58: Based on the knowledge-related text feature representation H obtained in B57 ktn Using a gating mechanism, it dynamically fuses its text feature representation H. t Obtain knowledge-enhanced text feature representation H et' :

[0072]

[0073] H et' =g×H t +(1-g)×H ktn

[0074] Where g is a gating coefficient automatically generated based on features, and W3 and b3 are trainable parameters.

[0075] Furthermore, step B6 specifically includes the following steps:

[0076] Step B61: Based on the image label feature representation H obtained in step B2 vtag Using noun features to represent H nt The guided attention mechanism dynamically learns the correlation between nouns and each image label, resulting in a noun-guided image label feature representation H. vtag' :

[0077] Hv tag‘ =MultiAtt(Q,K,V)

[0078] =Concat(head1,…,head2)W v

[0079]

[0080] Where Q represents the noun feature representation H nt K and V represent the image label features H. vtag , and W v These are trainable parameters;

[0081] Step B62: Based on the noun-guided image label feature representation H obtained in step B61 vtag' The image feature representation H obtained in step B23 v We implement an interactive attention mechanism to obtain knowledge-related image feature representations H. ev :

[0082]

[0083] in, Let f() represent the feature representation of the j-th image object, and f() be a function that measures the similarity between two words.

[0084] Step B63: Obtain the knowledge-related image feature representation H based on step B62. ev To reduce semantic conflicts between image labels and image features, a heterogeneity resolution module is applied to remove semantic conflicts; firstly, the knowledge-related image feature representation H obtained in step B62 is... ev Mapping to a high-dimensional space yields a high-dimensional feature representation H. high :

[0085] H high =f(W4H ev +b4)

[0086] Where f(·) represents the Leaky_ReLU activation function, and W4 and b4 are trainable parameters;

[0087] Step B64: Feed the high-dimensional features obtained in B63 into the activation function layer, and dynamically fuse the high-dimensional feature representation H through a gating mechanism. high Image feature representation H related to knowledge ev Obtaining knowledge-enhanced image feature representation H ev' :

[0088] θ=σ(H high )

[0089] H ev' =W5(θ×H) high +(1-θ)×H ev )+b5

[0090] Where σ(·) represents the sigmoid activation function used to generate the mean vector, and W5 and b5 are trainable parameters.

[0091] Furthermore, step B7 specifically includes the following steps:

[0092] Step B71: Represent the features obtained in step B4 The knowledge-enhanced text feature representation H obtained in step B5 et' The knowledge-enhanced image feature representation H obtained in step B6 ev' The fusion process yields a semantically enhanced multimodal feature representation H. f :

[0093]

[0094] Step B72: Develop the semantically enhanced multimodal feature representation H obtained in step B71. f The input is fed into the BART decoder to obtain the output at time t.

[0095]

[0096] Among them, Y <t This represents the output of the Decoder at the first t time steps. This represents the output of the Decoder at time t;

[0097] Step B73: Based on step B72, obtain the decoder's output at time t, and utilize the emotional polarity embedding representation C. d And in step B2, the text feature representations obtained are classified using softmax to obtain aspect words and their corresponding sentiment polarities:

[0098]

[0099] Furthermore, step B8 specifically includes the following steps:

[0100] Step B81: Compare the predicted values ​​and actual values ​​obtained in Step B7 to calculate the L2 regularized cross-entropy loss.

[0101]

[0102] Where S is the number of training samples in a batch B, and C is the number of classes; Θ represents the true value of the aspect term; Θ represents all trainable parameters; λ is the coefficient of the L2 regularization term;

[0103] Step B82: Develop the semantically enhanced multimodal feature representation H obtained in step B71. f Use a stacked attention network to obtain a contrastive multimodal representation for image-text contrast learning:

[0104]

[0105]

[0106] in, Let PWTcpnv(·) be the output of the l-th pointwise convolutional network, and let PWTcpnv(·) be the output of the pointwise convolutional network. The output of the last layer in the stacked pointwise convolutional networks is...

[0107] Step B83: Compare the contrastive multimodal representations obtained in step B82. Divided into images and text and And perform a graphical and textual comparison of learning loss:

[0108]

[0109] In each sample As a positive sample, and For negative samples, τ is the temperature coefficient;

[0110] Step B84: Add the losses obtained in steps B81 and B83, calculate the gradients of each parameter in the deep network using the backpropagation algorithm, and update the parameters using the stochastic gradient descent algorithm.

[0111]

[0112] Among them, λ1 and λ2 are controllable parameters.

[0113] The present invention also provides a multimodal sentiment analysis system based on multi-source knowledge and multi-granularity image and text features, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, it can implement the above-mentioned method steps.

[0114] Compared with existing technologies, this invention has the following advantages: First, addressing the modality alignment problem between different modalities, this invention does not introduce information from the entire image. Instead, it uses fine-grained images and fine-grained text for similarity calculation and combines component trees to obtain different granularities of text. A multi-layer graph attention network is then used to obtain multi-granular text-visual fusion feature representations, thereby achieving alignment between image and text modalities and reducing the introduction of noise and redundant information irrelevant to aspect terms. Second, this invention introduces image labels and AMR graphs as external knowledge for both images and text. The model can fully explore the semantic information of images and text using multi-source knowledge, obtaining knowledge-enhanced feature representations. Finally, to further reduce the semantic gap between image and text modalities, this invention introduces image-text contrastive learning. It fully utilizes multi-granular text-visual fusion feature representations, knowledge-enhanced text feature representations, and knowledge-enhanced image feature representations to narrow the semantic information distance between different modalities within the same sample. This helps the model learn the correlation and consistency between different modalities, thereby improving the accuracy of aspect term extraction and aspect term sentiment polarity prediction. Attached Figure Description

[0115] Figure 1 This is a flowchart illustrating the method implementation of an embodiment of the present invention;

[0116] Figure 2 This is an architecture diagram of a deep learning network model G based on external multi-source knowledge and multi-granularity image and text features in an embodiment of the present invention. Detailed Implementation

[0117] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0118] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0119] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0120] like Figure 1 As shown, this embodiment provides a sentiment analysis method based on multi-source knowledge and multi-granularity image and text features, including the following steps:

[0121] Step A: Collect text and image data, identify and label aspect terms and their sentiment polarities in the text, and form a training set TS containing the original text, related images, aspect terms and their sentiment polarities. The aspect terms and sentiment polarities of each sample are used as ground truth values ​​to train the model to identify and understand the sentiment expressions in text and images.

[0122] Step B: Construct a deep learning network model G based on external multi-source knowledge and multi-granularity image-text features. The deep learning network model G uses AMR graphs and image labels to obtain knowledge-enhanced text feature representations and image feature representations, and uses the syntactic dependencies of the text, the component tree structure of the text, and the fine-grained image-text relationship matrix, combined with a multi-layer graph attention network to obtain multi-granularity text-visual fusion feature representations; train the deep learning network model G based on external multi-source knowledge and multi-granularity image-text features using the training set TS.

[0123] Step C: Input the text data and image data into the trained deep learning network model G in sequence, extract the aspect words in the text data, and predict the sentiment polarity corresponding to the aspect words.

[0124] Figure 2 This is the architecture diagram of the deep learning network model G based on external multi-source knowledge and multi-granularity image and text features in this embodiment. The specific implementation steps of step B in this embodiment are as follows.

[0125] Step B1: Perform syntactic dependency parsing on the text data of each sample in the training set TS to obtain the syntactic dependency relationship DA between different nodes; use the constituent tree parsing to obtain the text partition of the sentence and obtain the constituent tree structure CA of the text.

[0126] The specific implementation method of step B1 is as follows: perform syntactic dependency parsing on the text data of each sample in the training set TS to obtain the syntactic dependency relationship DA between different nodes; use the component tree parsing to obtain the text segmentation of the sentence and obtain the component tree structure CA of the text.

[0127]

[0128] Among them, DA ij This represents the relationship between the i-th node and the j-th node in the syntactic dependency tree, CA. ij This represents the relationship between the i-th node and the j-th node in the constituent tree structure. Dep.Tree represents syntactic dependencies; Con.Tree represents the constituent tree structure.

[0129] Step B2: For each sample in the training set TS, obtain the AMR (Abstract Semantic Representation) graph G corresponding to the text data in the sample. AMR And the image labels V corresponding to the image data in the sample. tag Encode the text and image data to obtain the text feature representation H. t and image feature representation H v ; for the obtained AMR image G AMR The feature representation H of the AMR image is obtained through processing. AMR and the obtained image label V tag The image label feature representation H is obtained through processing. vtag .

[0130] Step B2 specifically includes the following steps:

[0131] Step B21: For the text data T of each sample in the training set TS, use the parser APRING to obtain the AMR graph G of the text data in each sample. AMR To obtain coarse-grained information about the images, a pre-trained model ResNet_101 is used to extract the image label set V from the image data V of each sample. tag If the extracted image labels are unrelated to the image or aspect, it will lead to error propagation problems. Therefore, 10 image labels are extracted for each image:

[0132] G AMR =APRING(T)

[0133]

[0134] Step B22: Take the G obtained in step B21 AMR and V tag Add to the training set, i.e., the training set is represented as TS = {T, V, G}. AMR V tag }, where T represents text data, V represents image data, and G represents text data. AMR V represents the AMR diagram corresponding to the text. tag The image label represents the image corresponding to the image; Faster R-CNN is used to encode the image data to obtain the image object features E. V And retain the 36 objects with the highest confidence:

[0135]

[0136] Step B23: Obtain image object features E from step B22. V And using the pre-trained BART model to obtain the text embedding representation E S The input is encoded by the BART encoder to obtain the multimodal representation H. tv :

[0137] E = " +E V +" "+" <bos> "+E S +"" <eos>"

[0138] H tv =[H v H t = BART-Encoder(E)

[0139] Step B24: Process the AMR image G obtained in step B22. AMR The AMR map feature representation H is obtained by encoding it using a BART encoder. AMR :

[0140] H AMR =BART-Encoder(G AMR )

[0141] Step B25: Process the image label set V obtained in step B22. tag The image label feature representation H is obtained by encoding it using a BART encoder. vtag :

[0142]

[0143] Step B3: Represent the text features H obtained in step B2 t and image feature representation H v Similarity calculations are performed to obtain the fine-grained image-text relation matrix SA; this matrix is ​​then fused with the syntactic dependency relation DA and the constituent tree structure CA obtained in step B1 to obtain the heterogeneous graph matrix EA.

[0144] Step B3 specifically includes the following steps:

[0145] Step B31: Represent the text features H obtained in step B2 t and image feature representation H v Similarity is calculated to obtain a fine-grained image-text relationship matrix SA, and a threshold τ is set. If the similarity is greater than the threshold, the images are related; otherwise, they are not related.

[0146]

[0147] in, This represents the feature representation of the i-th word in the text. This represents the feature representation of the j-th object in the image. Attention score guided by text. Let τ be the image-guided attention score, τ be the threshold, and f() be the dot product operation; for and Adding them together gives the similarity value (SA) between the i-th word of the text and the j-th object of the image. ij This represents the relationship between the i-th word of the text and the j-th object of the image.

[0148] Step B32: Fuse the syntactic dependency tree relation DA and constituent tree structure CA obtained in step B1 with the fine-grained image-text relation matrix SA obtained in step B31 to obtain the heterogeneous graph matrix EA, which serves as the relation adjacency matrix of the graph attention network.

[0149] Step B4: Based on the heterogeneous graph matrix EA from Step B3, a multi-layer graph attention network is used to obtain the interaction between multi-granularity text features and image features, ultimately obtaining a multi-granularity text-visual fusion feature representation.

[0150] Step B4 specifically includes the following steps:

[0151] Step B41: Based on the multimodal feature representation H obtained in step B2 tv Using the heterogeneous graph matrix EA obtained in step B3, a multi-layer graph attention network is used to obtain the feature representation of the i-th node in the l-th layer.

[0152]

[0153] in, It is the adjacent node of the i-th node in the heterogeneous graph matrix EA. It is the final output of the i-th node in the l-th layer of the graph attention network. It is the result of the i-th node in layer l after passing through the masked self-attention mechanism. This is the normalized similarity value between the i-th node and the j-th node in the l-th layer. LN() is the normalization layer, || denotes vector concatenation, Z is the number of attention heads, and σ is the activation function. is the trainable parameter of the l-th layer of the z-th attention head, and f() is a function that measures the similarity between two words.

[0154] Step B42: Repeat step B41, using the previous layer as the input for the next layer, i.e. For the input of the second graph attention network layer, thus obtaining By stacking multiple layers of graph attention network layers, the granularity of the text gradually increases from fine to coarse, ultimately resulting in a multi-granularity text-visual fusion feature representation.

[0155] Step B5: Based on the text feature representation H obtained in step B2 t and AMR plot feature representation H AMR Obtain the noun feature representation H of the text. nt And an interactive attention mechanism is used to fuse noun feature representations and AMR graph feature representations to obtain knowledge-related noun feature representations H. kt Furthermore, to eliminate noise from AMR graphs, the knowledge-related noun feature representation H... kt Semantic cleansing yields knowledge-enhanced text feature representations H. et' .

[0156] Step B5 specifically includes the following steps:

[0157] Step B51: Construct a noun mask matrix, preserving the textual feature representation of the nouns:

[0158]

[0159] Step B52: Based on the text feature representation H obtained in step B2 t Based on the noun mask matrix obtained in step B51, the noun feature representation H is obtained. nt :

[0160]

[0161] Step B53: Represent the characteristic features of the AMR image obtained in step B22. AMR Further encoding using a graph neural network yields the global feature representation H of the AMR graph. GAMR :

[0162] H GAMR =GNN(H AMR )

[0163] Step B54: Based on the noun feature representation H obtained in step B52 nt The global feature representation H of the AMR map obtained in step B53 GAMR We implement an interactive attention mechanism to obtain knowledge-related noun feature representations H. kt :

[0164]

[0165] in, Let represent the global feature representation of the i-th node in the AMR graph. Let f() represent the feature representation of the j-th noun, and f() be a function that measures the similarity between two words.

[0166] Step B55: Based on the knowledge-related noun feature representation H obtained in step B54 kt To reduce noise among multi-source information, a semantic cleanup module is applied to remove noise; the knowledge-related noun feature representation H obtained in step B54 is then processed. kt and noun features represent H nt The semantic parameter matrix is ​​obtained after two fully connected layers:

[0167]

[0168] Here, MLP1() and MLP2() are two different fully connected layers, f(·) represents the Leaky_ReLU activation function, and W1 and W2 are semantic parameter matrices.

[0169] Step B56: Based on the semantic parameter matrices W1 and W2 obtained in step B55, obtain the final knowledge-enhanced noun feature representation H. kn :

[0170] H kn =W2(W1×H kt +b1)+b2

[0171] Here, b1 and b2 are trainable parameters.

[0172] Step B57: Enhance the noun feature representation H based on the knowledge obtained in Step B56. kn The text feature representation H obtained in step B23 t Using a multi-head attention mechanism to obtain knowledge-related text feature representations H ktn :

[0173] H ktn =MultiAtt(Q,K,V)

[0174] =Concat(head1,…,head2)W v

[0175]

[0176] Where Q represents the text feature representation H t K and V are the noun feature representations for knowledge enhancement H. kn , and W v These are trainable parameters.

[0177] Step B58: Based on the knowledge-related text feature representation H obtained in B57 ktn Using a gating mechanism, it dynamically fuses its text feature representation H. t Obtain knowledge-enhanced text feature representation H et' :

[0178]

[0179] H et' =g×H t +(1-g)×H ktn

[0180] Where g is a gating coefficient automatically generated based on features, and W3 and b3 are trainable parameters.

[0181] Step B6: Based on the image feature representation H obtained in step B2 v Image label feature representation H vtag The interactive attention mechanism is used to fuse and obtain knowledge-related image feature representations H. ev Furthermore, to reduce semantic conflicts arising from the fusion of image features and image label features, the knowledge-related image feature representation H is optimized. ev Heterogeneity resolution yields knowledge-enhanced image feature representations H. ev' .

[0182] Step B6 specifically includes the following steps:

[0183] Step B61: Based on the image label feature representation H obtained in step B2 vtag Using noun features to represent H nt The guided attention mechanism dynamically learns the correlation between nouns and each image label, resulting in a noun-guided image label feature representation H. vtag' :

[0184] H vtag‘ =MultiAtt(Q,K,V)

[0185] =Concat(head1,…,head2)W v

[0186]

[0187] Where Q represents the noun feature representation H nt K and V represent the image label features H. vtag , and W v These are trainable parameters.

[0188] Step B62: Based on the noun-guided image label feature representation H obtained in step B61 vtag' The image feature representation H obtained in step B23 v We implement an interactive attention mechanism to obtain knowledge-related image feature representations H. ev :

[0189]

[0190] in, Let f() represent the feature representation of the j-th image object, and let f() be a function that measures the similarity between two words.

[0191] Step B63: Obtain the knowledge-related image feature representation H based on step B62. ev To reduce semantic conflicts between image labels and image features, a heterogeneity resolution module is applied to remove semantic conflicts; firstly, the knowledge-related image feature representation H obtained in step B62 is... ev Mapping to a high-dimensional space yields a high-dimensional feature representation H. high :

[0192] H high =f(W4H ev +b4)

[0193] Where f(·) represents the Leaky_ReLU activation function, and W4 and b4 are trainable parameters.

[0194] Step B64: Feed the high-dimensional features obtained in B63 into the activation function layer, and dynamically fuse the high-dimensional feature representation H through a gating mechanism. high Image feature representation H related to knowledge ev Obtaining knowledge-enhanced image feature representation H ev' :

[0195] θ=σ(H high )

[0196] H ev' =W5(θ×H) high +(1-θ)×H ev )+b5

[0197] Where σ(·) represents the sigmoid activation function used to generate the mean vector, and W5 and b5 are trainable parameters.

[0198] Step B7: Represent the multi-granularity text-visual fusion features obtained in Step B4 The knowledge-enhanced text feature representation H obtained in step B5 et' The knowledge-enhanced image feature representation H obtained in step B6 ev' The fusion process yields a semantically enhanced multimodal feature representation H. f We extract aspect words from the text data of each sample and predict the sentiment polarity of those aspect words.

[0199] Step B7 specifically includes the following steps:

[0200] Step B71: Represent the features obtained in step B4 The knowledge-enhanced text feature representation H obtained in step B5 et' The knowledge-enhanced image feature representation H obtained in step B6 ev' The fusion process yields a semantically enhanced multimodal feature representation H. f :

[0201]

[0202] Step B72: Develop the semantically enhanced multimodal feature representation H obtained in step B71. f The input is fed into the BART decoder to obtain the output at time t.

[0203]

[0204] Among them, Y <t This represents the output of the Decoder at the first t time steps. This represents the output of the Decoder at time t;

[0205] Step B73: Based on the output of the decoder at time t obtained in step B72, the emotional polarity (positive, negative, neutral) embedding representation C is used. d And in step B2, the text feature representations obtained are classified using softmax to obtain aspect words and their corresponding sentiment polarities:

[0206]

[0207] Step B8: Compare the predicted values ​​and true values ​​obtained in Step B7 to calculate the loss, and perform semantically enhanced multimodal feature representation H obtained in Step B7. f We employ a graph-text comparison learning loss algorithm, use backpropagation to calculate the gradients of each parameter in the deep network, and then use stochastic gradient descent to update the parameters.

[0208] Step B8 specifically includes the following steps:

[0209] Step B81: Compare the predicted values ​​and actual values ​​obtained in Step B7 to calculate the L2 regularized cross-entropy loss.

[0210]

[0211] Where S is the number of training samples in a batch B, and C is the number of classes; Θ represents the true value of the aspect term; Θ represents all trainable parameters; λ is the coefficient of the L2 regularization term.

[0212] Step B82: Develop the semantically enhanced multimodal feature representation H obtained in step B71. f Use a stacked attention network to obtain a contrastive multimodal representation for image-text contrast learning:

[0213]

[0214]

[0215] in, PWTconv(·) is the output of the l-th pointwise convolutional network, and PWTconv(·) is the output of the last layer in the stacked pointwise convolutional networks.

[0216] Step B83: Compare the contrastive multimodal representations obtained in step B82. Divided into images and text and And perform a graphical and textual comparison of learning loss:

[0217]

[0218] In each sample As a positive sample, and For negative samples, τ is the temperature coefficient.

[0219] Step B84: Add the losses obtained in steps B81 and B83, calculate the gradients of each parameter in the deep network using the backpropagation algorithm, and update the parameters using the stochastic gradient descent algorithm.

[0220]

[0221] Among them, λ1 and λ2 are controllable parameters.

[0222] Step B9: When the loss value generated by the deep learning network model is less than the set threshold or the maximum number of iterations is reached, terminate the training of the deep learning model G.

[0223] This embodiment also provides a multimodal sentiment analysis system based on multi-source knowledge and multi-granularity image and text features, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, it can implement the above-mentioned method steps.

[0224] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0225] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0226] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0227] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0228] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.< / eos> < / bos> < / eos> < / bos>

Claims

1. A sentiment analysis method based on multi-source knowledge and multi-granularity image and text features, characterized in that, Includes the following steps: Step A: Collect text and image data, identify and label aspect terms and their sentiment polarities in the text, forming a training set containing the original text, related images, aspect terms, and their sentiment polarities. The aspect terms and sentiment polarity of each sample are used as ground truth values ​​to train the model to recognize and understand sentiment expressions in text and images. Step B: Construct a deep learning network model G based on external multi-source knowledge and multi-granularity image and text features. The deep learning network model G uses AMR graphs and image labels to obtain knowledge-enhanced text feature representations and image feature representations. It also uses the syntactic dependencies of the text, the component tree structure of the text, and the fine-grained image-text relationship matrix, combined with a multi-layer graph attention network, to obtain multi-granularity text-visual fusion feature representations. Using the training set Train a deep learning network model G based on external multi-source knowledge and multi-granularity image and text features; In step B, the text features are represented. and image feature representation Similarity calculations are performed to obtain a fine-grained image-text relationship matrix. ; And fine-grained image-text relationship matrix Syntactic Dependencies and component tree structure The heterogeneous graph matrix is ​​obtained by fusion. ; Based on heterogeneous graph matrix By using a multi-layer graph attention network, the interaction between multi-granular text features and image features is obtained, ultimately yielding a multi-granular text-visual fusion feature representation. ; Step C: Input the text data and image data into the trained deep learning network model G in sequence, extract the aspect words in the text data, and predict the sentiment polarity corresponding to the aspect words.

2. The sentiment analysis method based on multi-source knowledge and multi-granularity image and text features according to claim 1, characterized in that, Step B specifically includes the following steps: Step B1: For the training set Syntactic dependency parsing is performed on the text data of each sample to obtain the syntactic dependencies between different nodes. By using component tree parsing, the text segmentation of the sentence is obtained, resulting in the component tree structure of the text. ; Step B2: For the training set For each sample, obtain the AMR image corresponding to the text data in the sample. and the image labels corresponding to the image data in the sample. Text and image data are encoded to obtain text feature representations. ; the obtained AMR image The feature representation of the AMR image is obtained through processing. and the obtained image labels The image label feature representation is obtained through processing. ; Step B3: Represent the text features obtained in Step B2 and image feature representation Similarity calculations are performed to obtain a fine-grained image-text relationship matrix. ; and a fine-grained image-text relationship matrix Syntactic dependencies obtained in step B1 and component tree structure The heterogeneous graph matrix is ​​obtained by fusion. ; Step B4: Heterogeneous graph matrix based on step B3 By using a multi-layer graph attention network, the interaction between multi-granular text features and image features is obtained, ultimately yielding a multi-granular text-visual fusion feature representation. ; Step B5: Text feature representation based on step B2 and AMR plot feature representation Obtain the noun feature representation of the text. Furthermore, an interactive attention mechanism is used to fuse noun feature representations and AMR graph feature representations to obtain knowledge-related noun feature representations. Furthermore, in order to eliminate noise from AMR graphs, the feature representation of knowledge-related nouns is improved. Semantic cleansing yields knowledge-enhanced text feature representations. ; Step B6: Image feature representation based on step B2 Image label feature representation An interactive attention mechanism is used to fuse and obtain knowledge-related image feature representations. Furthermore, to reduce semantic conflicts arising from the fusion of image features and image label features, knowledge-related image feature representations are optimized. Heterogeneity reduction yields knowledge-enhanced image feature representations. ; Step B7: Represent the multi-granularity text-visual fusion features obtained in Step B4 The knowledge-enhanced text feature representation obtained in step B5 The knowledge-enhanced image feature representation obtained in step B6 The fusion process yields semantically enhanced multimodal feature representations. Extract aspect words from the text data of each sample and predict the sentiment polarity of those aspect words; Step B8: Compare the predicted values ​​and true values ​​obtained in Step B7 to calculate the loss, and perform semantically enhanced multimodal feature representation on the data obtained in Step B7. We employ a text-image comparison learning loss algorithm, use backpropagation to calculate the gradients of each parameter in the deep network, and then use stochastic gradient descent to update the parameters. Step B9: Terminate the deep learning model when the loss value generated by the deep learning network model is less than the set threshold or the maximum number of iterations is reached. Training.

3. The sentiment analysis method based on multi-source knowledge and multi-granularity image and text features according to claim 2, characterized in that, The specific implementation method of step B1 is as follows: for the training set Syntactic dependency parsing is performed on the text data of each sample to obtain the syntactic dependencies between different nodes. By using component tree parsing, the text segmentation of the sentence is obtained, resulting in the component tree structure of the text. ; in, This represents the relationship between the i-th node and the j-th node of the text in the syntactic dependency tree. This represents the relationship between the i-th node and the j-th node of the text in the component tree structure. Indicate syntactic dependencies; Represents a component tree structure; Step B2 specifically includes the following steps: Step B21: For the training set For each sample's text data T, the AMR graph of the text data in each sample is obtained using the parser APRING. To obtain coarse-grained information about the images, a pre-trained model, ResNet_101, is used to extract the image label set for each sample's image data V. If the extracted image labels are unrelated to the image or aspect, it will lead to error propagation problems. Therefore, 10 image labels are extracted for each image: Step B22: Obtain the results from step B21 and Added to the training set, i.e., the training set is represented as = Where T represents text data and V represents image data. This represents the AMR diagram corresponding to the text. The image labels represent the corresponding images; Faster R-CNN is used to encode the image data to obtain image object features. And retain the 36 objects with the highest confidence: Step B23: Obtain image object features from step B22. And using the pre-trained BART model to obtain the embedding representation of the text. The input is encoded by the BART encoder to obtain a multimodal representation. : Step B24: AMR image obtained in step B22 The AMR map feature representation is obtained by encoding it using a BART encoder. : Step B25: Process the image label set obtained in step B22 The image label feature representation is obtained by encoding it using a BART encoder. :

4. The sentiment analysis method based on multi-source knowledge and multi-granularity image and text features according to claim 3, characterized in that, Step B3 specifically includes the following steps: Step B31: Represent the text features obtained in step B2 and image feature representation Similarity is calculated to obtain a fine-grained image-text relationship matrix SA, and a threshold is set. If the similarity is greater than the threshold, then they are related; otherwise, they are not related. in, This represents the feature representation of the i-th word in the text. This represents the feature representation of the j-th object in the image. Attention score guided by text. The image-guided attention score, For the threshold, It is a function that measures the similarity between two words; for and Adding them together gives the similarity value between the i-th word of the text and the j-th object of the image. This represents the relationship between the i-th word of the text and the j-th object of the image; Step B32: Fuse the syntactic dependency tree relation DA and constituent tree structure CA obtained in step B1 with the fine-grained image-text relation matrix SA obtained in step B31 to obtain the heterogeneous graph matrix EA, which serves as the relation adjacency matrix of the graph attention network.

5. The sentiment analysis method based on multi-source knowledge and multi-granularity image and text features according to claim 4, characterized in that, Step B4 specifically includes the following steps: Step B41: Multimodal feature representation based on step B2 Using the heterogeneous graph matrix EA obtained in step B3, and a multi-layer graph attention network, the first... Feature representation of the i-th node in the layer : in, This is the graph attention network. The set of adjacent nodes of the i-th node in the heterogeneous graph matrix EA. This is the graph attention network. The final output of the i-th node of the layer. It is the first The result of the i-th node of the layer after the masked self-attention mechanism. It is the first The normalized result of the similarity value between the i-th node and the j-th node of the layer. It is a normalization layer, || represents vector concatenation, and Z is the number of attention heads. It is an activation function. It is the zth attention head's first The trainable parameters of the layer, It is a function that measures the similarity between two words; Step B42: Repeat step B41, using the previous layer as the input for the next layer, i.e. For the input of the second graph attention network layer, thus obtaining Similarly, by stacking multiple layers of graph attention network layers, the granularity of the text gradually increases from fine-grained to coarse-grained, ultimately resulting in a multi-granularity text-visual fusion feature representation. .

6. The sentiment analysis method based on multi-source knowledge and multi-granularity image and text features according to claim 5, characterized in that, Step B5 specifically includes the following steps: Step B51: Construct a noun mask matrix, preserving the textual feature representation of the nouns: Step B52: Text feature representation based on step B2 Based on the noun mask matrix obtained in step B51, the noun feature representation is obtained. : Step B53: Feature representation of the AMR image obtained in step B22 The global feature representation of the AMR graph is obtained through further encoding using a graph neural network. : Step B54: Noun feature representation based on step B52 Global feature representation of the AMR map obtained in step B53 Implement an interactive attention mechanism to obtain knowledge-related noun feature representations. : in, Let represent the global feature representation of the i-th node in the AMR graph. This represents the characteristic of the j-th noun. It is a function that measures the similarity between two words; Step B55: Representation of knowledge-related noun features based on the information obtained in Step B54. To reduce noise among multi-source information, a semantic cleanup module is applied to remove noise; the knowledge-related noun feature representation obtained in step B54 is then processed. and noun feature representation The semantic parameter matrix is ​​obtained after two fully connected layers: in, and They are two different fully connected layers. This represents the Leaky_ReLU activation function. and This is a semantic parameter matrix; Step B56: Based on the semantic parameter matrix obtained in step B55 and To obtain the final knowledge-enhanced noun feature representation. : in, and These are trainable parameters; Step B57: Enhanced noun feature representation based on the knowledge obtained in Step B56 and the text feature representation obtained in step B23 Using a multi-head attention mechanism to obtain knowledge-related text feature representations : ,K )= Where Q represents the text feature representation. K and V are noun feature representations for knowledge enhancement. , , and These are trainable parameters; Step B58: Based on the knowledge-related text feature representation obtained in B57 Using a gating mechanism to dynamically fuse its text feature representation Obtain knowledge-enhanced text feature representation : Where g is a gating coefficient automatically generated based on the features. and These are trainable parameters.

7. The sentiment analysis method based on multi-source knowledge and multi-granularity image and text features according to claim 6, characterized in that, Step B6 specifically includes the following steps: Step B61: Image label feature representation based on step B2 Using noun features to represent The guided attention mechanism dynamically learns the correlation between nouns and each image label, resulting in a noun-guided image label feature representation. : ,K )= Where Q represents the noun feature representation. K and V are the image label feature representations. , , and These are trainable parameters; Step B62: Image label feature representation based on the nouns obtained in step B61 Image feature representation obtained in step B23 Implement an interactive attention mechanism to obtain knowledge-related image feature representations. : in, This represents the feature representation of the j-th image object. It is a function that measures the similarity between two words; Step B63: Obtain knowledge-related image feature representations based on step B62. To reduce semantic conflicts between image labels and image features, a heterogeneity resolution module is applied to remove semantic conflicts; firstly, the knowledge-related image feature representation obtained in step B62 is... Mapping to a high-dimensional space yields a high-dimensional feature representation. : in, This represents the Leaky_ReLU activation function. and These are trainable parameters; Step B64: Feed the high-dimensional features obtained in B63 into the activation function layer, and dynamically fuse the high-dimensional feature representations through a gating mechanism. Image feature representation related to knowledge Obtaining knowledge-enhanced image feature representations : in, This indicates that the sigmoid activation function is used to generate the mean vector. and These are trainable parameters.

8. The sentiment analysis method based on multi-source knowledge and multi-granularity image and text features according to claim 7, characterized in that, Step B7 specifically includes the following steps: Step B71: Represent the features obtained in step B4 The knowledge-enhanced text feature representation obtained in step B5 The knowledge-enhanced image feature representation obtained in step B6 The fusion process yields semantically enhanced multimodal feature representations. : Step B72: Represent the semantically enhanced multimodal features obtained in Step B71 The input is fed into the BART decoder to obtain the output at time t. : in, This represents the output of the Decoder at the first t time steps. This represents the output of the Decoder at time t; Step B73: Based on step B72, obtain the decoder's output at time t, and utilize the emotional polarity embedding representation. And in step B2, the text feature representations obtained are classified using softmax to obtain aspect words and their corresponding sentiment polarities:

9. The sentiment analysis method based on multi-source knowledge and multi-granularity image and text features according to claim 8, characterized in that, Step B8 specifically includes the following steps: Step B81: Compare the predicted values ​​and actual values ​​obtained in Step B7 to calculate the L2 regularized cross-entropy loss. Where B is the number of training samples in a batch, and c is the number of classes; Θ represents the true value of the aspect term; Θ represents all trainable parameters; λ is the coefficient of the L2 regularization term; Step B82: Represent the semantically enhanced multimodal features obtained in step B71 Use a stacked attention network to obtain a contrastive multimodal representation for image-text contrast learning: in, For the first The output of a layer-by-layer pointwise convolutional network, For a pointwise convolutional network, the output of the last layer in a stacked pointwise convolutional network is... ; Step B83: Compare the contrastive multimodal representations obtained in step B82. Divided into images and text and And perform a graphical and textual comparison of learning loss: In each sample As a positive sample, and For negative samples, Temperature coefficient; Step B84: Add the losses obtained in steps B81 and B83, calculate the gradients of each parameter in the deep network using the backpropagation algorithm, and update the parameters using the stochastic gradient descent algorithm. in, , It is a controllable parameter.

10. A multimodal sentiment analysis system based on multi-source knowledge and multi-granularity image and text features, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the steps of the method as described in any one of claims 1-9.

Images