A multi-modal review sentiment analysis method and system based on heterogeneous graph convolution

Abstract

The application relates to a multi-modal review sentiment analysis method based on heterogeneous graph convolution, which comprises the following steps: step A: collecting user reviews and related images, extracting aspect words of products or services involved in the user reviews, and labeling the sentiment polarity of the user reviews to the specific aspect of the products or services, so as to construct a training set DB ; step B: using the training set DB , training a deep learning network model based on a knowledge graph and a heterogeneous graph convolution network DLM for analyzing the sentiment polarity of user reviews and related images to specific aspects of products or services; and step C: inputting the user reviews and related images and the aspect words of the products or services involved into the trained deep learning network model to obtain the sentiment polarity of the user reviews and related images to the specific aspect of the products or services. The method and system are beneficial to improving the accuracy of sentiment classification.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of natural language processing, and particularly relates to a multi-modal review sentiment analysis method and system based on heterogeneous graph convolution. BACKGROUND

[0002] Sentiment Analysis (SA), also known as opinion mining, is a basic task in the field of natural language processing (NLP). The basic goal of this task is to mine the sentiment information in the given text and analyze its sentiment polarity. However, in the face of growing comment text data, manual methods are simply unable to complete the task of collecting and analyzing the vast amount of information on the Internet, so research institutions have increasingly valued sentiment analysis technology. This task is divided into documentation-level sentiment analysis, sentence-level sentiment analysis, and aspect-level sentiment analysis according to different classification granularity. Early research on sentiment analysis mainly focused on documentation-level and sentence-level, that is, assuming that only one entity (entitv) expresses sentiment in a document or sentence. Although the documentation-level and sentence-level tasks are widely studied in the field of sentiment analysis, due to the limitations of these two tasks themselves, traditional documentation-level or sentence-level sentiment analysis models can only analyze the entire document or sentence to identify its sentiment polarity. In actual applications, this cannot meet the needs. With the rapid development of the Internet, a large number of comment texts have emerged on network social media platforms and network shopping platforms, and when a comment sentence has multiple aspects and each aspect has different sentiment polarity, these documentation-level or sentence-level sentiment analysis models obviously cannot correctly interpret the sentiment information therein. Therefore, the fine-grained sentiment analysis method for specific entity aspects has become the main research problem of the current sentiment analysis task. Aspect-level sentiment analysis aims to judge the sentiment polarity corresponding to the specific aspect of the target in the comment text, and this task involves core natural language processing problems such as lexical semantics, reference resolution, and opinion extraction, and has strong theoretical research significance and application value.

[0003] Moreover, in the era of rapidly developing network, people tend to express their opinions and emotions in the form of combination of text and image or video. These multi-modal language data, with their more rich and attractive forms, have occupied the social media sites with overwhelming advantages, and also provided sufficient data resources for the research of multi-modal language computing. Therefore, in recent years, multi-modal sentiment analysis has gradually become a key task in the field of sentiment analysis, and sentiment analysis in the multi-modal context will make the machine closer to the real human emotional processing. For the aspect-level sentiment analysis task, image information is usually as indicative as text information. On the one hand, in multi-modal data, both text and image are highly related to aspect sentiment. In addition, different aspects may be related to different parts of each modal data. In other words, customers may write different words or attach different images for different aspects. On the other hand, text and image information can complement each other and enhance the analysis of the sentiment of a specific aspect. In summary, there are various correlations in multi-modal data for aspect-level sentiment analysis.

[0004] In recent years, with the rise of deep learning techniques, the techniques have also been widely applied to aspect-level sentiment analysis tasks. The most commonly used neural networks are convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Among them, CNN performs well in capturing semantic information from text, while RNN, especially long short-term memory (LSTM) and gated recurrent unit (GRU), can better extract sentiment features of specific aspects for sentiment classification according to context information. However, these neural network models all ignore the syntactic dependency between the given aspect and its context words, which represents the dependency between words in the review sentence and is particularly important for correctly determining the sentiment polarity of the aspect. Recently, some scholars have used graph neural networks (GNN) and its variants to consider syntactic information when learning aspect-level representations for sentiment classification. Zhang et al. combined graph convolutional network (GCN) with attention mechanism to obtain semantic relevance between context information and aspects. Hang et al. used Bi-LSTM to encode sentences and then used graph attention network (GAT) to extract dependencies between context words. However, most existing aspect-level sentiment analysis models ignore the implicit sentiment information in the text data, and there are some data with implicit sentiment information in the existing data sets. If these data are not processed specifically, the model may not be able to correctly identify when there are irony or implicit expressions, thereby hindering the improvement of model performance. In addition, due to the small size of the data set, the model is difficult to learn the pattern of such implicit sentiment expression, and most of the work to solve the implicit sentiment problem mainly introduces external knowledge to assist the model to identify the implicit sentiment information in the text. However, their knowledge selection algorithms are mostly based on rules or combined with attention mechanisms, without considering the context semantics comprehensively.

[0005] With the popularity of multi-form user-generated content (e.g., text, image, speech or video), sentiment analysis has gone beyond traditional text-based analysis. Multimodal sentiment analysis is an emerging research field that integrates text and non-text information into user sentiment analysis. Text-image pairs are the most common form of multi-modal data. With the development of deep learning technology, some neural network-based models have been proposed for multi-modal sentiment analysis and have made significant progress. Yu et al. pre-trained text CNN and image CNN to extract feature representations from text and image respectively, and combined these multi-modal features to train a logistic regression model. In order to fully capture the visual semantic information, Xu et al. extracted scene and object features from images and used these visual semantic features to absorb text words to simulate the influence of images on text, which also proved that text and image can complement each other in sentiment analysis tasks and improve the performance of the model. Therefore, Xu and Chen et al.

[38] proposed a co-memory attention mechanism to simulate the interaction between text and image in an interactive way. Their model considers the influence of one modality on the other (i.e., text to image and image to text), and achieves better performance than other related methods. However, there is relatively little work on the intersection of aspect-level and multi-modal sentiment analysis, and the above model structure is relatively simple, and there is no further problem of fully utilizing the relationship between image and text. SUMMARY

[0006] The purpose of the present application is to provide a multi-modal review sentiment analysis method and system based on heterogeneous graph convolution, which is beneficial to improve the accuracy of sentiment classification.

[0007] To achieve the above purpose, the technical scheme adopted by the present application is: a multi-modal review sentiment analysis method based on heterogeneous graph convolution, comprising the following steps:

[0008] Step A: Collect user reviews and related images, extract aspect words related to products or services in user reviews, and label the sentiment polarity of user reviews for specific aspects of the product or service, to build a training set DB:

[0009] Step B: Using the training set DB, training a deep learning network model DLM based on knowledge graph and heterogeneous graph convolution network, for analyzing the sentiment polarity of user reviews and related images for specific aspects of products or services;

[0010] Step C: Input the user reviews and related images and the aspect words related to the products or services into the trained deep learning network model to obtain the sentiment polarity of the user reviews and related images for specific aspects of the product or service.

[0011] Further, the step B specifically comprises the following steps:

[0012] Step B1: Encode each training sample in the training set DB to obtain the semantic representation vector X of the comment. r The semantic representation vector X of the aspect a Syntactic Dependency Adjacency Matrix A r and image region representation vector X im ;

[0013] Step B2: Select a set (Set) of knowledge triples related to the comment context from the knowledge graph based on the dynamic knowledge selection mechanism. skt Then, its knowledge terms are encoded to obtain a set Set. skt Knowledge word representation vector X kg ;

[0014] Step B3: By analyzing the semantic representation vector X of the aspect a and image region representation vector X im The aspect-dependent image region representation vector X is obtained using the interactive attention mechanism. air ; Obtain a set of tuples of tags t related to the comment context using an image tag selection mechanism. sit ;

[0015] Step B4: For X a Average pooling is used to obtain the aspect average representation vector. For X r Position encoding is performed to obtain the position-enhanced comment representation vector X. pw By connecting X pw and Obtain the representation vector C sd，0 Based on the text-knowledge-image heterogeneous graph construction strategy, the adjacency matrix A is obtained by generating a text-knowledge-image heterogeneous graph TKIHG. hg Then, the nodes are encoded using a cross-modal attention mechanism to obtain the node representation vector C of the heterogeneous graph TKIHG. hg，0 ;

[0016] Step B5: Transfer the representation vector C sd，0 The node representation vector C of the heterogeneous graph TKIHG hg，0 The inputs are fed into two different L-layer graph convolutional networks, denoted as the Comment Text Graph Convolutional Network (RGCN) and the Text-Knowledge-Image Heterogeneous Graph Convolutional Network (HGCN), respectively. These networks learn and extract heterogeneous information such as syntactic dependency relations and contextual semantics, as well as image labels and external knowledge, to obtain the text graph convolutional representation vector C. sd，L and heterogeneous graph convolution representation vector C hg，L ;

[0017] Step B6: text graph convolutional representation vector C of the text sd，L Aspect masking operation is performed to obtain the text graph convolutional masking representation vector C of the review mask，L , and the semantic representation vector X of the review r The aspect information enhanced by the aggregated syntactic information is further used to enhance the context representation using the interaction attention mechanism, to obtain the aspect enhanced representation vector X of the review r ea ;

[0018] Step B7: heterogeneous graph convolutional representation vector C hg，L is respectively connected with the aspect enhanced representation vector X of the review r ea and the aspect related image region representation vector X air The cross-modal attention mechanism is used to further utilize the heterogeneous information such as image labels and external knowledge to enhance the learning of sentiment features in the text and image modalities, to obtain the heterogeneous enhanced text representation vector X hm and the image representation vector X hair , and finally connect X hm and X hair to obtain the final representation vector X fin ;

[0019] Step B8: input the final representation vector X fin into the final prediction layer, calculate the gradient of each parameter in the deep learning network model according to the target loss function loss using the back propagation method, and update each parameter using the stochastic gradient descent method;

[0020] Step B9: when the loss value generated by the deep learning network model has an iterative change less than a given threshold value, or reaches a maximum number of iterations, the training process of the deep learning network model is terminated.

[0021] Further, the step B1 specifically comprises the following steps:

[0022] Step B11: traverse the training set DB, perform word segmentation processing and remove stop words on the user reviews and aspects therein, and adjust the images related to the reviews to an image with a pixel of 3×224×224, and each training sample in DB is represented as d=(r, im, a, p);

[0023] Wherein, r is a user review, im is an image related thereto after adjustment, a is an aspect word or phrase of a product or service involved in the user review r extracted from the user review, and p∈(positive, negative, neutral) is the sentiment polarity of the review on the aspect;

[0024] The review r is represented as:

[0025]

[0026] in, Let i be the i-th word in comment r, where i = 1, 2, ..., n, and n is the number of words in comment r.

[0027] Aspect 'a' is represented as:

[0028]

[0029] in, Let i be the i-th word in aspect a, where i = 1, 2, ..., m, and m is the number of words in aspect a;

[0030] Step B12: Get comments on step B11 The semantic representation vector X of comment r is obtained by encoding using a pre-trained BERT model and then reducing its dimensionality using a fully connected layer. r , is represented as:

[0031]

[0032] in, To comment on the i-th word The corresponding semantic representation vector, where d represents the output dimension;

[0033] Step B13: Obtaining aspects from step B11 The semantic representation vector X of aspect a is obtained by encoding using a pre-trained BERT model and then reducing its dimensionality using a fully connected layer. a , is represented as:

[0034]

[0035] in, The i-th word indicates aspect The corresponding word vectors, where d represents the output dimension;

[0036] Step B14: Encode and smooth the adjusted image using the pre-trained ResNet-152 model. After dimensionality reduction through a fully connected layer, obtain the image region representation vector. Where z represents the size of the output feature map, and d represents the output dimension;

[0037] Step B15: Perform syntactic dependency parsing on comment r to obtain the syntactic dependency tree SDT;

[0038]

[0039] in, Words in comments Harmony There is a syntactic dependency relationship between them;

[0040] Step B16: encode the syntax dependency tree SDT obtained in step B15 into an n-order adjacency matrix A r , A r is expressed as:

[0041]

[0042] wherein, is 1, indicating that there is a syntax dependency relationship between the words and the word , is 0, indicating that there is no syntax dependency relationship between the words and the word .

[0043] Further, the step B2 specifically comprises the following steps:

[0044] Step B21: select the context words connected to the aspect words from the syntax dependency tree SDT obtained in step B15 to form a seed node set Set sn , expressed as:

[0045]

[0046] wherein, is the aspect word or the context word connected to it, m≤m′≤n;

[0047] Step B22: for each node in the seed node set, select its sentiment polarity word and five semantically similar words from the knowledge graph to form five candidate knowledge triples CKT for the seed node, expressed as:

[0048] CKT = <w sn , w sp , w ss >

[0049] wherein, w sn is the seed node word, w sp is the sentiment polarity word, and w ss is the semantically similar word;

[0050] Step B23: construct each candidate knowledge triple CKT into a candidate knowledge sentence r cks , input it into the pre-trained model BERT, and take the average value to obtain its average semantic representation vector X cks ; r cks is expressed as:

[0051] r cks = "The sentiment polarity of the word 'w sn ' is 'w spThe word that is most semantically similar to 'w ss '

[0052] Step B24: Calculate the semantic representation vector X r of the review after taking the average value of X rm , rm and X cks , cks then calculate the cosine similarity between X rm and X cks ,

[0053] Similarity_Score(r, r cks ) = CosineSimilarity(X rm , X cks )

[0054] wherein X rm ,

[0055] Step B25: Calculate the similarity scores of the candidate knowledge sentences formed by all candidate knowledge triples CKT according to step B24, and select the top k candidate knowledge triples with the highest scores as the seed node's knowledge triple set, as the seed node's external knowledge most relevant to the context.

[0056] Step B26: Repeat the above steps for all seed nodes in the seed node set Set sn to obtain a set Set skt of knowledge triple sets containing all seed nodes, represented as:

[0057]

[0058] wherein, is the knowledge triple set of the i-th seed node ;

[0059] Step B27: Encode all sentiment polarity words and semantically similar words in the knowledge triple set Set skt through knowledge graph embedding to obtain their node representation vectors as wherein, is the knowledge representation vector corresponding to the i-th sentiment polarity word or semantically similar word, which is obtained by searching in the pre-trained knowledge word vector matrix , wherein d represents the dimension of the knowledge word vector, and |V| is the number of words in the knowledge word embedding V.

[0060] Further, the step B3 specifically comprises the following steps:

[0061] Step B31: semantic representation vector X of aspect a and image region representation vector X im Use the interaction attention mechanism to obtain the aspect-related image region representation vector X air , X air The calculation process is as follows:

[0062]

[0063]

[0064]

[0065] wherein, (·) T denotes the transpose operation;

[0066] Step B32: input the image im into the pre-trained model ResNet-152 to obtain 10 relevant image labels tag = {tag1, …, tag i , …, tag 10}, and then input each seed node word in the seed node set Set sn obtained in step 21 and the 10 image labels into the pre-trained model BERT to obtain their semantic representation vectors and wherein, is the semantic representation vector of the seed node , and is the semantic representation vector of the i-th image label tag i

[0067] Step B33: calculate the cosine similarity between the semantic representation vector of the image label tag i and the semantic representation vector of the seed node to obtain the similarity score between the seed node and the image label tag i , and the calculation process is as follows:

[0068]

[0069] Step B34: calculate the similarity scores of the seed node and the 10 labels respectively according to step B33, and select the top t image labels with the highest scores to form a label t-tuple TT together with the seed node , which is represented as:

[0070]

[0071] ​Step B35: For the seed node set Set sn Repeat the above steps for all seed nodes to obtain a set Set containing the label tuples of all seed nodes. sit , is represented as:

[0072] Set sit ={TT1, ..., TT} i , ..., TT m′}

[0073] Among them, TT i For the i-th seed node The tag t tuple.

[0074] Furthermore, step B4 specifically includes the following steps:

[0075] Step B41: Semantic representation vector X of the aspect a Perform average pooling operation to obtain The calculation formula is as follows:

[0076]

[0077] in,

[0078] Step B42: Semantic representation vector X of the comment r Position encoding is performed to obtain the position-enhanced representation vector X of the comment. pw X pw Represented as:

[0079]

[0080]

[0081] in, To strengthen the representation vector corresponding to the i-th word in comment r, "·" indicates multiplication of a real number with a vector, pw i The position weight corresponding to the i-th word in comment r is calculated as follows:

[0082]

[0083] Where θ and θ+m-1 represent the beginning and end positions of aspect a in the comment r, respectively;

[0084] Step B43: Calculate the aspect average representation vector obtained in step B41. X obtained in step B42 pw By performing a connection, we obtain the representation vector C. sd，0 , Represented as:

[0085]

[0086] where, is the representation vector input into the graph convolution network corresponding to the i-th word in the review r, i = 1, 2, …, n, and “;” represents the vector concatenation operation;

[0087] Step B44: Based on the set Set skt obtained in step 26, the set of knowledge triples of each seed node is obtained, and the sentiment polarity w sp and the semantic similar word w ss in each knowledge triple are sequentially taken as the knowledge expansion node and added an edge to the related seed node in the syntactic dependency tree SDT obtained in step B15; Similarly, based on the set Set sit obtained in step 35, each image label is sequentially taken as the image label expansion node and added an edge to the related seed node in the syntactic dependency tree SDT; In order to avoid redundancy, if the new knowledge expansion node or image label expansion node already exists in the graph, only an edge between it and the related seed node will be added; Finally, a text-knowledge-image heterogeneous graph TKIHG is obtained.

[0088]

[0089] where, represents that there is a syntactic dependency relationship between the words w and w in the review, represents the sentiment polarity of the word w in the review is w sp , represents the semantic similar word of the word w in the review is w ss , represents the image label related to the word w in the review is tag j ;

[0090] Step B45: The text-knowledge-image heterogeneous graph TKIHG obtained in step B44 is encoded into a u-order adjacency matrix A hg , A hg is represented as:

[0091]

[0092] where, is 1, indicating that there is a connection relationship between the two words, u = n + m' x (t + k), where u = 0 means that there is no connection between the two words, the first n nodes in the figure are the words in the review text, the n+1th to the n+m'xtth nodes are the words of the related image label nodes, and the last n+m'xt+1th to the u nodes are the words of the related knowledge expansion nodes;

[0093] Step B46: The image label node words in the text-knowledge-image heterogeneous graph TKIHG are sequentially spliced into the review text r to form a new sequence and input into the pre-trained model BERT to reduce dimension by using a full connection layer to obtain the context semantic representation vector X of the text mode rt , which is expressed as:

[0094]

[0095] wherein

[0096] Step B47: The context semantic representation vector X of the text mode rt is mapped to the same feature space as X kg by using a self-attention mechanism to obtain the transformed context semantic representation vector X kg′ , and the calculation process is as follows:

[0097]

[0098] wherein W1, W2 and are learnable weight matrices;

[0099] Step B48: The context semantic representation vector X of the text mode rt is mapped to the same feature space as X kg′ by using a self-attention mechanism to obtain the transformed context semantic representation vector X rt′ , and the calculation process is as follows:

[0100]

[0101] wherein is the mapped context semantic representation vector of the text mode, W4, W5 and are learnable weight matrices;

[0102] Step B49: The transformed context semantic representation vector X rt′ and the semantic guided external knowledge representation vector X kg′ are connected to obtain the node representation vector C of the text-knowledge-image heterogeneous graph TKIHG hg，0 , which is expressed as:

[0103]

[0104] wherein

[0105] Further, the step B5 specifically comprises the following steps:

[0106] Step B51: for the review text graph convolutional network RGCN, the representation vector C sd，0 input the first layer graph convolutional network, use the adjacency matrix A r update the representation vector of each word node, output C sd，1 , and take it as the input of the next layer graph convolutional network;

[0107] wherein, C sd，1 is expressed as:

[0108]

[0109] wherein, is the output of the i-th node in the first layer graph convolutional network, The calculation formula is as follows:

[0110]

[0111]

[0112] wherein, is a learnable weight matrix, is a bias vector; ReLU is an activation function; the i-th node in the graph convolutional network corresponds to the i-th word in the review r d i indicates the degree of the i-th node, d i +1 is to prevent the degree of the i-th node from being 0 to cause calculation error;

[0113] Step B52: for the text-knowledge-image heterogeneous graph convolutional network HGCN, the node representation vector C hg，0 input the first layer graph convolutional network, use the adjacency matrix A hg update the representation vector of each node, output C hg，1 , and take it as the input of the next layer graph convolutional network;

[0114] wherein, C hg，1 is expressed as:

[0115]

[0116] wherein, It is the output of the i-th node in the first layer of the graph convolutional network. The calculation formula is as follows:

[0117]

[0118]

[0119] in It is a learnable weight matrix. It is the bias vector; ReLU is the activation function; in a graph convolutional network, the edges between nodes represent the connections between them, d i d represents the degree of the i-th node. i +1 is to prevent calculation errors when the degree of the i-th node is 0;

[0120] Step B53: Place C separately sd，1 and C hg，1 Input into the next layer of graph convolutional networks of RGCN and HGCN, and repeat steps B51 and B52;

[0121] Specifically, for the comment text graph convolutional network RGCN, the output of the l-th layer graph convolutional network is... As input to the (l+1)th layer of the graph convolutional network, the text graph convolutional representation vector is obtained after iteration. For the text-knowledge-image heterogeneous graph convolutional network HGCN, the output of the l-th layer graph convolutional network... As input to the (l+1)th layer of the graph convolutional network, the heterogeneous graph convolutional representation vector is obtained after iteration. L is the number of layers in the graph convolutional network, where 1 ≤ l ≤ L.

[0122] Furthermore, step B6 specifically includes the following steps:

[0123] Step B61: Convolve the text graph representation vector C obtained in step B53. sd，L Aspect masking is performed to mask the text graph convolution outputs that do not belong to aspect words, resulting in the text graph convolution aspect representation vector C of comment r. mask，L The calculation process is as follows:

[0124]

[0125] Where θ represents the starting position of the aspect in the comment sentence, and θ+m-1 represents the ending position of the aspect in the comment sentence. Let represent the graph convolution aspect representation vector corresponding to the i-th word in the comment, where 0 represents the zero vector of dimension d;

[0126] Step B62: Convert the semantic representation vector X of comment r obtained in step B12. rand the aspect representation vector C of the review r obtained in step B61 mask，L The input interaction attention network enhances the context representation with aspect information aggregated by the interaction attention mechanism using syntactic information, to obtain an aspect-enhanced representation vector X of the review r ea The calculation formula is as follows:

[0127]

[0128]

[0129]

[0130] Wherein, (·) T represents the transpose operation, β i is the attention weight of the i-th word in the review r.

[0131] Further, the step B7 specifically comprises the following steps:

[0132] Step B71: the heterogeneous graph convolution representation vector C hg，L and the aspect-enhanced representation vector X of the review r ea Using a cross-modal attention mechanism, further utilize heterogeneous information such as image labels and external knowledge to enhance the learning of the text modality, to obtain a heterogeneous-enhanced text representation vector X hm The calculation process is as follows:

[0133]

[0134] Wherein, W9, W 10 and are learnable weight matrices;

[0135] Step B72: the heterogeneous graph convolution representation vector C hg，L and the aspect-related image region representation vector X air Using a cross-modal attention mechanism, further utilize heterogeneous information such as image labels and external knowledge to enhance the learning of the image modality, to obtain a heterogeneous-enhanced image representation vector X hair The calculation process is as follows:

[0136]

[0137] Wherein, W 12 , W 13 and are learnable weight matrices;

[0138] Step B73: the heterogeneous-enhanced text representation vector X hmand image representation vector X hair a connection operation is performed to obtain a final representation of the final representation vector X fin The calculation process is as follows:

[0139] X fin = [X hm ; X hair ].

[0140] The application also provides a multi-modal review sentiment analysis system using the above method, comprising:

[0141] A data collection module is configured to extract user reviews and related images, aspect words in the reviews, position information of the aspect words, and mark the sentiment polarity of the aspect, and construct a training set.

[0142] A preprocessing module is configured to preprocess training samples in the training set, including word segmentation processing, removing stop words, adjusting image size, syntax dependency parsing, selecting a relevant knowledge triple set and an image label set, and generating a text-knowledge-image heterogeneous graph.

[0143] An encoding module is configured to find word vectors of knowledge words of the knowledge triple set in a pre-trained knowledge graph word vector matrix to obtain knowledge word representation vectors of the knowledge triple set.

[0144] A network training module is configured to input the processed user reviews, related images, aspects, text-knowledge-image heterogeneous graphs, and knowledge word representation vectors of the knowledge triple set into a deep learning network to obtain a final representation vector of the multi-modal review and train the deep learning network using the representation vector, use the probability that the representation vector belongs to a certain category and the labels in the training set as a loss, and train the entire deep learning network to minimize the loss to obtain a deep learning network model based on a knowledge graph and a heterogeneous graph convolution network.

[0145] A sentiment analysis module uses NLP tools to extract aspects in the input user reviews, and then uses the trained deep learning network model based on the knowledge graph and the heterogeneous graph convolution network to analyze and process the input reviews, images, and aspects, and outputs sentiment evaluation polarity of specific aspects in the user reviews and related images.

[0146] Compared with the prior art, the present application has the following beneficial effects: a multi-modal review sentiment analysis method and system based on heterogeneous graph convolution are provided, the method and system first encode review sentences, product aspects and images respectively by using a pre-training model, then obtain relevant knowledge nodes of the corresponding review sentences by using a knowledge graph and a dynamic knowledge selection mechanism; next, aspect-related image labels are obtained by using an image selection mechanism, then text-knowledge-image heterogeneous graphs are constructed by using knowledge information and image label information; then, the position of the review sentence representation is weighted by using position information, and the syntactic dependency relationship and heterogeneous information in the multi-modal review are learned by using two GCNs, finally, the sentiment features of the text and image modalities are further enhanced by using cross-modal attention mechanisms to utilize heterogeneous information such as image labels and external knowledge, so that the accuracy of the model in predicting sentiment classification can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0147] Figure 1 is a method implementation flowchart of an embodiment of the present application;

[0148] Figure 2 is an architecture diagram of a deep learning network model in an embodiment of the present application;

[0149] Figure 3 is a system structure schematic diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0150] The present application will be further described below in conjunction with the drawings and embodiments.

[0151] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0152] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0153] As shown in Figure 1 The present embodiment provides a multi-modal review sentiment analysis method based on heterogeneous graph convolution, comprising the following steps:

[0154] Step A: Collect user reviews and related images, extract aspect words of products or services involved in user reviews, and label the sentiment polarity of user reviews for specific aspects of the product or service, to build a training set DB.

[0155] Step B: Using the training set DB, train the deep learning network model DLM based on the knowledge graph and the heterogeneous graph convolution network, for analyzing the sentiment polarity of user reviews and related images for specific aspects of products or services. The architecture of the deep learning network model is shown in Figure 2 .

[0156] In this embodiment, the step B specifically comprises the following steps:

[0157] Step B1: Encode each training sample in the training set DB to obtain the semantic representation vector X r of the review, a the semantic representation vector X r of the aspect, im and the image region representation vector X .

[0158] In this embodiment, the step B1 specifically comprises the following steps:

[0159] Step B11: Traverse the training set DB, perform word segmentation processing and remove stop words for user reviews and aspects therein, and adjust the images related to the reviews to an image with 3×224×224 pixels, and each training sample in the DB is represented as d=(r, im, a, p).

[0160] Wherein, r is the user review, im is the adjusted image related thereto, a is the aspect word or phrase of the product or service involved in the user review r extracted from the user review, and p∈(positive, negative, neutral) is the sentiment polarity of the review for the aspect.

[0161] The review r is represented as:

[0162]

[0163] Wherein, is the i-th word in the review r, i=1, 2, …, n, and n is the number of words in the review r.

[0164] The aspect a is represented as:

[0165]

[0166] Wherein, is the i-th word in the aspect a, i=1, 2, …, m, and m is the number of words in the aspect a.

[0167] Step B12: Get comments on step B11 The semantic representation vector X of comment r is obtained by encoding using a pre-trained BERT model and then reducing its dimensionality using a fully connected layer. r , is represented as:

[0168]

[0169] in, To comment on the i-th word The corresponding semantic representation vector, where d represents the output dimension.

[0170] Step B13: Obtaining aspects from step B11 The semantic representation vector X of aspect a is obtained by encoding using a pre-trained BERT model and then reducing its dimensionality using a fully connected layer. a , is represented as:

[0171]

[0172] in, The i-th word indicates aspect The corresponding word vector, d represents the output dimension.

[0173] Step B14: Encode and smooth the adjusted image using the pre-trained ResNet-152 model. After dimensionality reduction through a fully connected layer, obtain the image region representation vector. Where z represents the size of the output feature map and d represents the output dimension.

[0174] Step B15: Perform syntactic dependency parsing on comment r to obtain the syntactic dependency tree SDT.

[0175]

[0176] in, Words in comments Harmony There is a syntactic dependency relationship between them.

[0177] Step B16: Encode the syntactic dependency tree SDT obtained in step B15 into an n-order adjacency matrix A. r A r Represented as:

[0178]

[0179] in, A value of 1 indicates a word in the comment. Harmony There is a syntactic dependency relationship between them. A value of 0 indicates a word Harmony There is no syntactic dependency relationship between them.

[0180] Step B2: selecting a set of knowledge triples related to the context of the review from the knowledge graph according to the dynamic knowledge selection mechanism skt , and then encoding the knowledge words thereof to obtain the knowledge word representation vector X skt of the set Set kg .

[0181] In this embodiment, the step B2 specifically comprises the following steps:

[0182] Step B21: selecting the context words connected to the aspect words from the syntactic dependency tree SDT obtained in step B15 to form a seed node set Set sn with the aspect a having a length of m' and denoted as:

[0183]

[0184] wherein, is the aspect word or the context word connected thereto, and m≤m'≤n.

[0185] Step B22: for each node in the seed node set, selecting the sentiment polarity word and 5 words semantically similar thereto from the knowledge graph to form 5 candidate knowledge triples CKT of the seed node, respectively, and denoted as:

[0186] CKT = <w sn , w sp , w ss )

[0187] wherein, w sn is the seed node word, w sp is the sentiment polarity word, and w ss is the semantically similar word.

[0188] Step B23: constructing each candidate knowledge triple CKT into a candidate knowledge sentence r cks , inputting the pre-trained model BERT and obtaining the average semantic representation vector X cks thereof after averaging, and denoted as:

[0189] r cks = "the sentiment polarity of the word 'w sn ' is 'w sp ', and the word most similar in semantics is 'w ss '"

[0190] Step B24: obtaining the average semantic representation vector X r of the review after averaging the semantic representation vectors X rm of the review.rm Then calculate the cosine similarity between X rm and X cks , get the similarity score of review text r and candidate knowledge sentence r cks , the calculation process is as follows:

[0191] Similarity_Score(r,r cks )=CosineSimilarity(X rm , X cks )

[0192] Wherein, X rm ,

[0193] Step B25: Calculate the similarity score of the candidate knowledge sentence formed by all candidate knowledge triples CKT according to step B24, and select the top k candidate knowledge triples with the highest score as the seed node's knowledge triple set, as the seed node's external knowledge most relevant to the context.

[0194] Step B26: Repeat the above steps for all seed nodes in the seed node set Set sn , to get a set Set skt containing the knowledge triple sets of all seed nodes, denoted as:

[0195]

[0196] Wherein, is the knowledge triple set of the i-th seed node .

[0197] Step B27: Encode all sentiment polarity words and semantic similar words in the knowledge triple set Set skt through knowledge graph embedding, get its node representation vector as Wherein, is the knowledge representation vector corresponding to the i-th sentiment polarity word or semantic similar word, which is obtained by searching in the pre-trained knowledge word vector matrix , where d represents the dimension of the knowledge word vector, and |V| is the number of words in the knowledge word embedding V.

[0198] Step B3: Obtain the aspect-related image region representation vector X air by using the interaction attention mechanism on the aspect semantic representation vector X a and the image region representation vector X im ; obtain the label t-tuple set Set sit related to the context of the review through the image label selection mechanism.

[0199] In the present embodiment, the step B3 specifically comprises the following steps:

[0200] Step B31: obtaining aspect semantic representation vector X a and image region representation vector X im using an interaction attention mechanism to obtain aspect-related image region representation vector X air , X air The calculation process is as follows:

[0201]

[0202]

[0203]

[0204] wherein, (·) T denotes the transposition operation.

[0205] Step B32: inputting the image im into a pre-trained model ResNet-152 to obtain 10 relevant image labels tag = {tag1, …, tag i , …, tag 10}, and then inputting each seed node word in the seed node set Set sn obtained in step 21 and the 10 image labels into a pre-trained model BERT to obtain their semantic representation vectors and wherein, is the semantic representation vector of the seed node , and is the semantic representation vector of the i-th image label tag i .

[0206] Step B33: calculating the cosine similarity between the semantic representation vector of the image label tag i and the semantic representation vector of the seed node to obtain the similarity score between the seed node and the image label tag i , and the calculation process is as follows:

[0207]

[0208] Step B34: calculating the similarity scores of the seed node and the 10 labels respectively according to step B33, and selecting the top t image labels with the highest scores to form a label t-tuple TT together with the seed node , and denoted as:

[0209]

[0210] Step B35: Repeat the above steps for all seed nodes in the seed node set Set sn to obtain a set Set sit containing the label t-tuple of all seed nodes, denoted as:

[0211] Set sit = {TT1, …, TT i , …, TT m′}

[0212] wherein TT i is the label t-tuple of the i-th seed node.

[0213] Step B4: Average pooling is performed on X a to obtain an aspect average representation vector Position encoding is performed on X r to obtain a position-enhanced review representation vector X pw , and the representation vector C sd，0 is obtained by concatenating X pw and ; the generated text-knowledge-image heterogeneous graph TKIHG is obtained according to the text-knowledge-image heterogeneous construction strategy, and its adjacency matrix A hg is obtained, and then the cross-modal attention mechanism is used to encode the nodes of the heterogeneous graph TKIHG to obtain the node representation vector C hg，0 of the heterogeneous graph TKIHG.

[0214] In this embodiment, the step B4 specifically comprises the following steps:

[0215] Step B41: Average pooling is performed on the semantic representation vector X a of the aspect to obtain The calculation formula is as follows:

[0216]

[0217] wherein,

[0218] Step B42: Position encoding is performed on the semantic representation vector X r of the review to obtain a position-enhanced representation vector X pw of the review, and X pw is denoted as:

[0219]

[0220] ​

[0221] wherein, is the position-enhanced representation vector of the i-th word in the review r, “·” denotes the multiplication operation between a real number and a vector, pw i is the position weight of the i-th word in the review r, which is calculated as follows:

[0222]

[0223] wherein, θ and θ+m-1 represent the start and end positions of aspect a in review r, respectively.

[0224] Step B43: connecting the aspect average representation vector obtained in step B41 with X pw obtained in step B42, to obtain a representation vector C sd，0 , is represented as:

[0225]

[0226] wherein, is the representation vector input into the graph convolution network corresponding to the i-th word in the review r, i = 1, 2, …, n, and “;” denotes the vector connection operation.

[0227] Step B44: based on the set Set skt obtained in step 26, the knowledge triple set of each seed node is sequentially connected, and the sentiment polarity w sp and the semantic similar word w ss in each knowledge triple are respectively taken as knowledge expansion nodes and added with an edge to the relevant seed node in the syntactic dependency tree SDT obtained in step B15; similarly, based on the label t-tuple of each seed node in the set Set sit obtained in step 35, each image label is sequentially taken as an image label expansion node and added with an edge to the relevant seed node in the syntactic dependency tree SDT; in order to avoid redundancy, if the new knowledge expansion node or image label expansion node already exists in the graph, only an edge between it and the relevant seed node will be added; finally, a text-knowledge-image heterogeneous graph TKIHG is obtained.

[0228]

[0229] wherein, denotes the words and in the review have a syntactic dependency relationship, denotes the sentiment polarity word w sp of the word in the review, Words in the review The semantic similar words of w ss , The image tags related to the words in the review are tag j .

[0230] Step B45: The text-knowledge-image heterogeneous graph TKIHG obtained in step B44 is encoded into a u-order adjacency matrix A hg , A hg is expressed as:

[0231]

[0232] wherein, 1 indicates that there is a connection relationship between the two words, 0 indicates that there is no connection relationship between the two words, u=n+m′×(t+k); the first n nodes in the graph are the words in the review text, the n+1th to the n+m′×tth nodes are the words of the related image tag nodes, and the last n+m′×t+1th to the uth nodes are the words of the related knowledge expansion nodes.

[0233] Step B46: The image tag node words in the text-knowledge-image heterogeneous graph TKIHG are sequentially spliced into the review text r to form a new sequence and input into the pre-trained model BERT to use the full connection layer for dimension reduction to obtain the context semantic representation vector X rt of the text mode, which is expressed as:

[0234]

[0235] wherein

[0236] Step B47: The context semantic representation vector X rt of the text mode and the representation vector X kg of the knowledge expansion node obtained in step 27 are used to obtain the semantic guided external knowledge representation vector X kg′ using the cross-modal attention mechanism, and the calculation process is as follows:

[0237]

[0238] wherein, W1, W2 and are learnable weight matrices.

[0239] Step B48: The context semantic representation vector X rt of the text mode is mapped to the same feature space as X kg′ using the self-attention mechanism to obtain the transformed context semantic representation vector Xrt′ The calculation process is as follows:

[0240]

[0241] wherein, is the context semantic representation vector of the mapped text modality, W4, W5 and are learnable weight matrices.

[0242] Step B49: the transformed context semantic representation vector X rt′ is concatenated with the semantic guided external knowledge representation vector X kg′ to obtain the node representation vector C hg，0 of the text-knowledge-image heterogeneous graph TKIHG, which is represented as follows:

[0243]

[0244] wherein

[0245] Step B5: the representation vector C sd，0 and the node representation vector C hg，0 of the heterogeneous graph TKIHG are respectively input into two different L-layer graph convolutional networks, namely the review text graph convolutional network RGCN and the text-knowledge-image heterogeneous graph convolutional network HGCN, to respectively learn and extract the syntactic dependency relationship and the context semantics, image label and external knowledge heterogeneous information, and obtain the text graph convolutional representation vector C sd，L and the heterogeneous graph convolutional representation vector C hg，L .

[0246] In the embodiment, the step B5 specifically comprises the following steps:

[0247] Step B51: for the review text graph convolutional network RGCN, the representation vector C sd，0 obtained in step B43 is input into the first layer graph convolutional network, and the representation vector of each word node is updated by using the adjacency matrix A r , and C sd，1 is output and used as the input of the next layer graph convolutional network.

[0248] wherein, C sd，1 is represented as:

[0249]

[0250] wherein, is the output of the i-th node in the first layer graph convolutional network, and the calculation formula of C

[0251]

[0252]

[0253] wherein, is a learnable weight matrix, is a bias vector; ReLU is an activation function; the i-th node in the graph convolution network corresponds to the i-th word in the review r d i represents the degree of the i-th node, d i +1 is to prevent the degree of the i-th node from being 0, which leads to calculation error.

[0254] Step B52: For the text-knowledge-image heterogeneous graph convolution network HGCN, input the node representation vector C hg，0 of the heterogeneous graph TKIHG obtained in step B49 into the first layer of the graph convolution network, and use the adjacency matrix A hg to update the representation vector of each node, and output C hg，1 , which is used as the input of the next layer of the graph convolution network.

[0255] wherein, C hg，1 is represented as:

[0256]

[0257] wherein, is the output of the i-th node in the first layer of the graph convolution network, The calculation formula is as follows:

[0258]

[0259]

[0260] wherein is a learnable weight matrix, is a bias vector; ReLU is an activation function; the edge between nodes in the graph convolution network represents the connection relationship between nodes, d i represents the degree of the i-th node, d i +1 is to prevent the degree of the i-th node from being 0, which leads to calculation error.

[0261] Step B53: input C sd，1 and C hg，1 into the next layer of the graph convolution network of RGCN and HGCN respectively, and repeat steps B51 and B52.

[0262] wherein, for the review text graph convolution network RGCN, the output C of the l-th layer of the graph convolution network is used as the input of the l+1-th layer of the graph convolution network, and the text graph convolution representation vector C is obtained after iteration. For the text-knowledge-image heterogeneous graph convolutional network HGCN, the output of the lth layer graph convolutional network is taken as the input of the (l+1)th layer graph convolutional network, and the heterogeneous graph convolutional representation vector is obtained after iteration L is the number of layers of the graph convolutional network, and 1≤l≤L.

[0263] Step B6: performing aspect shielding operation on the text graph convolutional representation vector C sd，L to obtain the text graph convolutional shielding representation vector C mask，L of the aspect of the comment, and further combining the semantic representation vector X r of the comment to obtain the aspect enhanced representation vector X ea of the comment r using the interaction attention mechanism to further enhance the context representation with the aspect information aggregated by the syntax.

[0264] In the embodiment, the step B6 specifically comprises the following steps:

[0265] Step B61: performing aspect shielding operation on the text graph convolutional representation vector C sd，L obtained in step B53 to shield the text graph convolutional output that does not belong to the aspect word, and obtaining the text graph convolutional aspect representation vector C mask，L of the comment r, and the calculation process is as follows:

[0266]

[0267] wherein, θ represents the starting position of the aspect in the comment sentence, θ+m-1 represents the ending position of the aspect in the comment sentence, represents the graph convolutional aspect representation vector corresponding to the i-th word in the comment, and 0 represents a zero vector with a dimension of d.

[0268] Step B62: inputting the semantic representation vector X r of the comment r obtained in step B12 and the graph convolutional aspect representation vector C mask，L of the comment r obtained in step B61 into the interaction attention network, and obtaining the aspect enhanced representation vector X ea of the comment r by enhancing the context representation with the aspect information aggregated by the syntax through the interaction attention mechanism, and the calculation formula is as follows:

[0269]

[0270]

[0271]

[0272] wherein, (·) T represents the transpose operation, and β i ​is the attention weight of the i-th word in the review r.

[0273] Step B7: the isomorphic graph convolutional representation vector C hg，L respectively with the aspect enhanced representation vector X ea of the image region related to the aspect air Using the cross-modal attention mechanism, further utilize the isomorphic information such as image labels and external knowledge to enhance the learning of the sentiment features of the text and image modalities, to obtain the isomorphic enhanced text representation vector X hm and the image representation vector X hair , finally connect X hm and X hair to obtain the final representation vector X fin .

[0274] In this embodiment, the step B7 specifically comprises the following steps:

[0275] Step B71: the isomorphic graph convolutional representation vector C hg，L obtained in step B53 and the aspect enhanced representation vector X ea of the review r are used to further utilize the isomorphic information such as image labels and external knowledge to enhance the learning of the text modality, to obtain the isomorphic enhanced text representation vector X hm , the calculation process is as follows:

[0276]

[0277] wherein, W9, W 10 and are learnable weight matrices.

[0278] Step B72: the isomorphic graph convolutional representation vector C hg，L and the image region representation vector X air related to the aspect are used to further utilize the isomorphic information such as image labels and external knowledge to enhance the learning of the sentiment features of the image modality, to obtain the isomorphic enhanced image representation vector X hair , the calculation process is as follows:

[0279]

[0280] wherein, W 12 , W 13 and are learnable weight matrices.

[0281] Step B73: the isomorphic enhanced text representation vector X hm and the image representation vector X hairThe connection operation is performed to obtain a final representation vector X fin The calculation process is as follows:

[0282] X fin = [X hm ; X hair ].

[0283] Step C: inputting the user comment and the related image and the aspect word of the product or service involved into the trained deep learning network model to obtain the sentiment polarity of the user comment and the related image for a specific aspect in the product or service.

[0284] Step B8: inputting the final representation vector X fin into the final prediction layer, calculating the gradient of each parameter in the deep learning network model according to the target loss function loss by using the back propagation method, and updating each parameter by using the stochastic gradient descent method.

[0285] Step B9: when the iterative change of the loss value generated by the deep learning network model is less than a given threshold value or the maximum number of iterations is reached, the training process of the deep learning network model is terminated.

[0286] As shown in Figure 3 , the embodiment also provides a multi-modal comment sentiment analysis system using the above method, which comprises a data collection module, a preprocessing module, an encoding module, a network training module and a sentiment analysis module.

[0287] The data collection module is used to extract user comments and related images, aspect words in the comments, position information of the aspect words, and label the sentiment polarity of the aspect, and construct a training set.

[0288] The preprocessing module is used to preprocess the training samples in the training set, including word segmentation processing, removing stop words, adjusting image size, syntax dependency relation analysis, selecting a related knowledge triple set and an image label set, and generating a text-knowledge-image heterogeneous graph.

[0289] The encoding module is used to find the word vector of the knowledge word of the knowledge triple set in the pre-trained knowledge graph word vector matrix to obtain the knowledge word representation vector of the knowledge triple set.

[0290] The network training module is configured to input the processed user comment, the related image, the aspect, the knowledge word representation vector of the text-knowledge-image heterogeneous graph and the knowledge triple set into a deep learning network, obtain a final representation vector of the multi-modal comment, and train the deep learning network by using the final representation vector, use the probability that the representation vector belongs to a certain category and the label in the training set as a loss, train the entire deep learning network by taking minimizing the loss as an objective, and obtain the deep learning network model based on the knowledge graph and the heterogeneous graph convolution network.

[0291] The sentiment analysis module extracts the aspect in the input user comment by using an NLP tool, and then analyzes and processes the input comment, image and aspect by using the trained deep learning network model based on the knowledge graph and the heterogeneous graph convolution network, and outputs the sentiment evaluation polarity of the specific aspect involved in the user comment and the related image.

[0292] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0293] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems) and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0294] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0295] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0296] The above descriptions are only the preferred embodiment of the present application, not intended to limit the present application to other forms described. Any person skilled in the art can make changes or modifications to the above-described technical contents as equivalent embodiments. However, any simple modification, equivalent change and modification of the above embodiments without departing from the technical solution of the present application, according to the technical essence of the present application, still belongs to the protection scope of the technical solution of the present application.

Claims

1. A multimodal sentiment analysis method for comments based on heterogeneous graph convolution, characterized in that, Includes the following steps: Step A: Collect user reviews and related images, extract the product or service-related terms in the user reviews, and label the sentiment polarity of the user reviews for specific aspects of the product or service in order to build the training set DB; Step B: Using the training set DB, train a deep learning network model DLM based on knowledge graph and heterogeneous graph convolutional network to analyze the sentiment polarity of user reviews and related images on specific aspects of a product or service; Step C: Input user reviews and related images along with the relevant product or service terms into the trained deep learning network model to obtain the sentiment polarity of user reviews and related images for specific aspects of the product or service; Step B specifically includes the following steps: Step B1: Encode each training sample in the training set DB to obtain the semantic representation vector of the comment. semantic representation vectors of aspects Syntactic Dependency Adjacency Matrix and image region representation vector ; Step B2: Select a set of knowledge triples related to the comment context from the knowledge graph based on the dynamic knowledge selection mechanism. Then, its knowledge terms are encoded to obtain a set. Knowledge word representation vector ; Step B3: Through the semantic representation vector of the aspect and image region representation vector Use the interactive attention mechanism to obtain aspect-dependent image region representation vectors. ; Obtain a set of tag tuples related to the comment context through an image tag selection mechanism. ; Step B4: For Average pooling is used to obtain the aspect average representation vector. ,right Position encoding is performed to obtain a position-enhanced comment representation vector. , through connection and The representation vector is obtained. Based on the text-knowledge-image heterogeneous graph construction strategy, the adjacency matrix of the generated text-knowledge-image heterogeneous graph TKIHG is obtained. Then, the nodes are encoded using a cross-modal attention mechanism to obtain the node representation vectors of the heterogeneous graph TKIHG. ; Step B5: Convert the representation vector Node representation vectors of heterogeneous graph TKIHG Input into two different In the layered graph convolutional network, denoted as the comment text graph convolutional network RGCN and the text-knowledge-image heterogeneous graph convolutional network HGCN, respectively, they learn and extract heterogeneous information such as syntactic dependency relations and contextual semantics, and image labels and external knowledge, to obtain the text graph convolutional representation vector. and heterogeneous graph convolution representation vector ; Step B6: Convolve the representation vector of the text graph Perform aspect masking to obtain the text graph convolution masking representation vector of the comment. Then, combined with the semantic representation vector of the comment Using an interactive attention mechanism, the contextual representation is further enhanced with aspect information that aggregates syntactic information to obtain comments. Enhance the representation vector aspect ; Step B7: Convolve the heterogeneous graph representation vector Respectively with comments Enhance the representation vector aspect Image region representation vectors related to aspects By employing a cross-modal attention mechanism, heterogeneous information such as image labels and external knowledge is further utilized to enhance the learning of sentiment features in both text and image modalities, resulting in heterogeneously enhanced text representation vectors. and image representation vector Finally, connect and The final representation vector is obtained. ; Step B8: Convert the final representation vector Input the final prediction layer, calculate the gradient of each parameter in the deep learning network model using the backpropagation method according to the target loss function loss, and update each parameter using the stochastic gradient descent method; Step B9: When the iterative change of the loss value generated by the deep learning network model is less than the given threshold, or when the maximum number of iterations is reached, the training process of the deep learning network model is terminated.

2. The multimodal comment sentiment analysis method based on heterogeneous graph convolution according to claim 1, characterized in that, Step B1 specifically includes the following steps: Step B11: Traverse the training set DB, perform word segmentation on user comments and aspects, remove stop words, and resize images related to comments to one pixel. The image, where each training sample in DB is represented as ; in, For user reviews, It is an adjusted image related to it. For user comments Extracting words or phrases related to the products or services mentioned in user reviews. To comment on the emotional polarity in this regard; Comment Represented as: in, For comments The i-th word in , For comments The number of words; Aspect 'a' is represented as: in, For aspect The i-th word in , For aspect The number of words; Step B12: Get comments on step B11 The comments are obtained by encoding using a pre-trained BERT model and then reducing the dimensionality using fully connected layers. semantic representation vector , is represented as: in, To comment on the i-th word The corresponding semantic representation vector, where d represents the output dimension; Step B13: Obtaining aspects from step B11 The semantic representation vector of aspect a is obtained by encoding using a pre-trained BERT model and then reducing its dimensionality using a fully connected layer. , is represented as: in, The i-th word indicates aspect The corresponding word vectors, where d represents the output dimension; Step B14: Encode and smooth the adjusted image using the pre-trained ResNet-152 model. After dimensionality reduction through a fully connected layer, obtain the image region representation vector. ;in d represents the size of the output feature map; Step B15: Comments Perform syntactic dependency parsing to obtain the syntactic dependency tree (SDT); in, Words in comments Harmony There is a syntactic dependency relationship between them; Step B16: Encode the syntactic dependency tree (SDT) obtained in step B15 into an n-order adjacency matrix. , Represented as: in, A value of 1 indicates a word in the comment. Harmony There is a syntactic dependency relationship between them. A value of 0 indicates a word Harmony There is no syntactic dependency relationship between them.

3. The multimodal comment sentiment analysis method based on heterogeneous graph convolution according to claim 2, characterized in that, Step B2 specifically includes the following steps: Step B21: Select context words connected to aspect words by edges from the syntactic dependency tree (SDT) obtained in step B15, and combine them with aspect a to form a sequence of length [length missing]. seed node set , is represented as: in, It is an aspect term or a context term that it is connected to by an edge. ; Step B22: For each node in the seed node set, select its sentiment polarity word and 5 semantically similar words from the knowledge graph to form 5 candidate knowledge triples (CKT) for that seed node, represented as: in, It is a seed node word. It is a word with emotional polarity. They are semantically similar words; Step B23: Construct each candidate knowledge triplet CKT into a candidate knowledge sentence. The average semantic representation vector is obtained by inputting it into the pre-trained BERT model and averaging the results. Represented as: Step B24: Convert the semantic representation vector of the comment The average semantic representation vector is obtained by taking the average value. Then calculate and The cosine similarity between them is used to obtain the comment text. and candidate knowledge sentences The similarity score is calculated as follows: in, ; Step B25: Calculate the similarity scores of the candidate knowledge sentences constructed by all candidate knowledge triples CKT according to step B24, and select the original candidate knowledge triples of the top k candidate knowledge sentences with the highest scores to form the knowledge triple set of the seed node, which serves as the external knowledge of the seed node that is most relevant to the context. Step B26: For the seed node set Repeat the above steps for all seed nodes to obtain a set of knowledge triples containing all seed nodes. , is represented as: in, For the i-th seed node The knowledge triplet set; Step B27: Embed the set of knowledge triples through knowledge graph embedding All sentiment polarity words and semantically similar words are encoded to obtain their node representation vectors. ;in, The knowledge representation vector corresponding to the i-th sentiment polarity word or semantically similar word is obtained through a pre-trained knowledge word vector matrix. The value is found in the search, where d represents the dimension of the knowledge word vector. It is the number of words embedded in knowledge words V.

4. The multimodal comment sentiment analysis method based on heterogeneous graph convolution according to claim 3, characterized in that, Step B3 specifically includes the following steps: Step B31: Semantic representation vector of aspects and image region representation vector Use the interactive attention mechanism to obtain aspect-dependent image region representation vectors. , The calculation process is as follows: in, , Indicates the transpose operation; Step B32: Transfer the image Inputting the pre-trained ResNet-152 model yields 10 relevant image labels. Then combine it with the seed node set obtained in step 21 Each seed node word is input into the pre-trained BERT model to obtain its semantic representation vector. and ;in, It is a seed node semantic representation vector, It is the i-th image label The semantic representation vector; Step B33: Calculate image labels semantic representation vector and seed node The seed node is obtained by calculating the cosine similarity of the semantic representation vectors. and image tags The similarity score between them is calculated as follows: Step B34: Calculate the seed nodes according to step B33. The similarity scores of the 10 labels are used to select the top t image labels with the highest scores and the seed node. Together they form a tuple TT with the label t; represented as: Step B35: For the seed node set Repeat the above steps for all seed nodes to obtain a set of label tuples containing all seed nodes. , is represented as: in, For the i-th seed node The tag t tuple.

5. A multimodal comment sentiment analysis method based on heterogeneous graph convolution as described in claim 4, characterized in that, Step B4 specifically includes the following steps: Step B41: Semantic representation vector of the aspect Perform average pooling operation to obtain The calculation formula is as follows: in, ; Step B42: Semantic representation vector of the comment Position encoding is performed to obtain a position-enhanced representation vector of the comment. , Represented as: in, For comments The position-enhanced representation vector corresponding to the i-th word in the text, " indicates a multiplication operation between a real number and a vector. For comments The position weight corresponding to the i-th word is calculated as follows: in, and These represent the beginning and end positions of aspect a in the comment r, respectively; Step B43: Calculate the aspect average representation vector obtained in step B41. The result obtained in step B42 Connect the elements to obtain the representation vector. , , is represented as: in, In response to comments The representation vector corresponding to the i-th word is input into the graph convolutional network. , ";" indicates a vector concatenation operation; Step B44: Based on the set obtained in step 26 For each seed node's set of knowledge triples, sequentially assign the sentiment polarity of each knowledge triple to... and semantically similar words Each node is used as a knowledge extension node and connected to the relevant seed node in the syntactic dependency tree (SDT) obtained in step B15 by adding an edge; similarly, based on the set obtained in step 35... For each seed node in the graph, the label tuple t is used to sequentially extend each image label as an image label extension node and add an edge connecting it to the relevant seed node in the syntactic dependency tree SDT. To avoid redundancy, if a new knowledge extension node or image label extension node already exists in the graph, only an edge is added between it and the relevant seed node. Finally, a text-knowledge-image heterogeneous graph TKIHG is obtained. in, Words in comments Harmony There is a syntactic dependency relationship between them. Words in comments The emotional polarity words are , Words in comments Semantic similar words are , Indicates words in comments The relevant image tags are ; Step B45: Encode the text-knowledge-image heterogeneous graph TKIHG obtained in step B44 into a single... adjacency matrix , Represented as: in, A value of 1 indicates a connection between the two words. A value of 0 indicates that there is no connection between the two words. The first n nodes in the graph are words in the comment text, and the nth node is the word in the comment text. One to the first The first node is a word associated with the related image label node, and the last node is... One to the first Each node contains words related to knowledge expansion nodes; Step B46: Concatenate the image tag node words from the Text-Knowledge-Image Heterogeneous Graph (TKIHG) into the comment text in sequence. The new sequence is then generated and input into the pre-trained BERT model. Dimensionality reduction is performed using fully connected layers to obtain the contextual semantic representation vector of the text modality. , is represented as: in ; Step B47: Contextual semantic representation vector of the text modality The representation vector of the knowledge extension node is obtained in step 27. Using a cross-modal attention mechanism, a semantically guided external knowledge representation vector is obtained. The calculation process is as follows: in, , , and It is a learnable weight matrix; Step B48: Contextual semantic representation vector of the text modality Map it to a value using a self-attention mechanism. Under the same feature space, the transformed context semantic representation vector is obtained. The calculation process is as follows: in, It is the context semantic representation vector of the mapped text modality. , and It is a learnable weight matrix; Step B49: Transform the context semantic representation vector and semantically guided external knowledge representation vectors By performing concatenation, we obtain the node representation vectors of the text-knowledge-image heterogeneous graph TKIHG. It is expressed as follows: in .

6. The multimodal comment sentiment analysis method based on heterogeneous graph convolution according to claim 5, characterized in that, Step B5 specifically includes the following steps: Step B51: For the RGCN (Graph Convolutional Network for Comment Text), use the representation vector obtained in step B43. Input the first layer of the graph convolutional network, and use the adjacency matrix Update the representation vector of each word node and output the result. And it serves as the input to the next layer of the graph convolutional network; in, Represented as: in, It is the output of the i-th node in the first layer of the graph convolutional network. The calculation formula is as follows: in, It is a learnable weight matrix. It is the bias vector; It is an activation function; the i-th node in the graph convolutional network is related to the i-th word in comment r. Correspondingly, This represents the degree of the i-th node. This is to prevent calculation errors caused by the degree of the i-th node being 0; Step B52: For the Text-Knowledge-Image Heterogeneous Graph Convolutional Network (HGCN), convert the node representation vectors of the heterogeneous graph TKIHG obtained in step B49... Input the first layer of the graph convolutional network, and use the adjacency matrix Update the representation vector of each node and output. And it serves as the input to the next layer of the graph convolutional network; in, Represented as: in, It is the output of the i-th node in the first layer of the graph convolutional network. The calculation formula is as follows: in It is a learnable weight matrix. It is the bias vector; It is an activation function; in a graph convolutional network, edges between nodes represent connections between them. This represents the degree of the i-th node. This is to prevent calculation errors caused by the degree of the i-th node being 0; Step B53: Separately... and Input into the next layer of graph convolutional networks of RGCN and HGCN, and repeat steps B51 and B52; Among them, for the comment text graph convolutional network RGCN, the first... Output of layer graph convolutional network As the first The input to the layered graph convolutional network, after iteration, yields the text graph convolutional representation vector. For the text-knowledge-image heterogeneous graph convolutional network HGCN, the first... Output of layer graph convolutional network As the first The input to the layered graph convolutional network, after iteration, yields a heterogeneous graph convolutional representation vector. ; The number of layers in a graph convolutional network. .

7. A multimodal comment sentiment analysis method based on heterogeneous graph convolution as described in claim 6, characterized in that, Step B6 specifically includes the following steps: Step B61: Convolve the text graph representation vector obtained in step B53. Perform aspect masking to mask the text graph convolution output that does not belong to aspect terms, and obtain the comments. Text graph convolution aspect representation vector The calculation process is as follows: in, Indicate the starting position of the aspect in the comment sentence. Indicate the position of the aspect at the end of the comment sentence. Let represent the graph convolution aspect representation vector corresponding to the i-th word in the comment, where 0 represents the zero vector of dimension d; Step B62: The comments obtained in step B12 semantic representation vector Comments obtained from step B61 The graph convolution aspect represents the vector The input is an interactive attention network, which enhances the contextual representation with aspect information by aggregating syntactic information through the interactive attention mechanism, resulting in comments. Enhance the representation vector aspect The calculation formula is as follows: in, This indicates the transpose operation. It's a comment The attention weight of the i-th word.

8. A multimodal comment sentiment analysis method based on heterogeneous graph convolution as described in claim 7, characterized in that, Step B7 specifically includes the following steps: Step B71: Convolution representation vector of heterogeneous graph obtained in step B53 and comments Enhance the representation vector aspect By employing a cross-modal attention mechanism, heterogeneous information such as image labels and external knowledge is further utilized to enhance the learned text modality, resulting in heterogeneously enhanced text representation vectors. The calculation process is as follows: in, , , and Learnable weight matrix; Step B72: Convolve the representation vector of the heterogeneous graph Image region representation vectors related to aspects By employing a cross-modal attention mechanism, heterogeneous information such as image labels and external knowledge is further utilized to enhance the emotional features of the learned image modality, resulting in heterogeneously enhanced image representation vectors. The calculation process is as follows: in, , , and Learnable weight matrix; Step B73: Enhance the heterogeneous text representation vector and image representation vector Perform a join operation to obtain the final representation vector. The calculation process is as follows:

9. A multimodal comment sentiment analysis system employing the method described in any one of claims 1-8, characterized in that, include: The data collection module is used to extract user comments and related images, aspect words in comments, the location information of aspect words, and to label the sentiment polarity of aspects to build a training set; The preprocessing module is used to preprocess the training samples in the training set, including word segmentation, stop word removal, image resizing, syntactic dependency parsing, selection of relevant knowledge triple sets and image label sets, and generation of text-knowledge-image heterogeneous graphs. The encoding module is used to find the word vectors of knowledge words in the knowledge triple set in the pre-trained knowledge graph word vector matrix, and obtain the knowledge word representation vector of the knowledge triple set; The network training module is used to input the processed user comments, related images, aspects, text-knowledge-image heterogeneous graphs, and knowledge triplet sets of knowledge word representation vectors into the deep learning network to obtain the final representation vector of multimodal comments. The deep learning network is trained using this representation vector as the probability of the representation vector belonging to a certain category and the annotations in the training set as the loss. The entire deep learning network is trained with the goal of minimizing the loss, resulting in a deep learning network model based on knowledge graphs and heterogeneous graph convolutional networks. as well as The sentiment analysis module uses NLP tools to extract aspects from the input user comments. Then, it uses a trained deep learning network model based on knowledge graphs and heterogeneous graph convolutional networks to analyze and process the input comments, images, and aspects, and outputs the sentiment polarity of the user comments and related images related to specific aspects.

Images