A text sentiment analysis method based on DualGCN

Abstract

The present invention relates to a text sentiment analysis method based on DualGCN, belonging to the field of text analysis. The method is: first pre-process the text, use pre-trained Glove word embedding, use BERT as an encoder to mine the hidden state vector of the sentence, use the phrase-level grammatical structure provided by the spanning tree and the information of the grammatical structure of the dependency tree clause to replace the previous grammatical structure information only obtained from the dependency tree, consider the influence between aspects, integrate the dependency tree structure and the composition tree structure, solve the long-distance dependency problem of aspect-level sentiment analysis, and increase the receptive field of each layer in GCN. Finally, through the information exchange between the grammatical module and the semantic module, the classification result is output by softmax. This method uses the information of the grammatical structure, alleviates the over-smoothing phenomenon that is easy to occur in GCN, and can effectively improve the accuracy of classification.

Description

Technical Field

[0001] The present invention belongs to the field of text analysis and relates to a text sentiment analysis method based on DualGCN. Background Art

[0002] Aspect-level sentiment analysis analyzes entities by identifying specific entities in sentences. It is a fine-grained sentiment analysis that can more accurately determine the sentiment polarity toward specific entities in sentences. Traditional sentiment analysis models use sentiment dictionaries for sentiment analysis, which requires a lot of prior knowledge, relies too much on the construction of sentiment dictionaries, and consumes a lot of manpower. They focus on extracting sentiment vocabulary features and bag-of-words features to train sentiment classifiers. The performance depends largely on the effectiveness of feature engineering work, and the generalization ability is poor. The deep neural network method can automatically generate sentence features, which greatly improves the generalization and generalization of sentiment analysis models. However, long texts often contain multiple aspects and multiple emotions. There are still great defects in solving the long-distance dependency problem of such long texts. At the same time, it is limited to certain grammatical rules and ignores the influence of context words and semantics on sentiment polarity. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a text sentiment analysis method based on DualGCN.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A text sentiment analysis model based on dual graph convolutional neural network DualGCN combined with component tree and multi-hop information includes the following steps:

[0006] S1: Use the pre-trained Glove word embedding to obtain the corresponding vector and word vector for each word, and use BERT (a pre-trained language representation model) as the encoder to obtain the hidden state vector of the sentence.

[0007] S2: Use the dependency parser to get the sentence dependency tree (Dep.Tree), and construct its corresponding adjacency matrix DA according to the dependency tree, so as to provide all potential syntactic structures to capture rich structural information. In the SynGCN module, the composition tree (Con.Tree) is introduced at the same time. Each layer of the composition tree is composed of several phrases. According to these phrases, its adjacency matrix CA is designed. The two kinds of syntactic information are fused, and the matrices DA and CA are fused to obtain the matrix A. syn , contains both phrase-level syntactic information and clause-level syntactic information, and can utilize syntactic information more accurately.

[0008] S3: Construct the grammatical module (SynGCN) and semantic module (SemGCN) based on the convolutional neural network model, respectively mine the grammatical information and semantic information of the sentence. In order to alleviate the transition smoothing phenomenon commonly seen in the GCN model, multi-hop information is introduced in the two sub-convolutional neural networks, replacing the original information obtained only from neighboring points, expanding from neighboring nodes to non-neighboring nodes, and alleviating the over-smoothing phenomenon. The hidden vectors obtained in S1) are input into SynGCN and SemGCN respectively; the feature matrix A composed of Con.Tree and Dep.Tree information in S2) is integrated syn Input into the SynGCN module to obtain the first grammatical graph feature corresponding to the sentence; in the SemGCN module, the adjacency matrix A is constructed through the self-attention mechanism sem , extract semantic features of the sentence to be analyzed, and transform the matrix A sem Input into the corresponding module to obtain the first semantic graph feature of the sentence.

[0009] S4: The obtained grammatical graph features and semantic graph features are input into the dual affine module in DualGCN, and the features of the first grammatical graph features and the first semantic graph features are exchanged respectively, the semantic and grammatical information is integrated, and the second grammatical graph features and the second semantic features are obtained.

[0010] S5: The second grammatical graph feature and the second semantic graph feature are respectively input into the pooling layer of DualGCN, and the extracted node features are averagely pooled and concatenated to obtain the first concatenated feature.

[0011] S6: Use the fully connected layer of DualGCN to perform probability analysis on the first concatenated feature to obtain the sentiment polarity results corresponding to words in different aspects. Finally, the softmax function is used for classification.

[0012] S31: When constructing the adjacency matrix in the SemGCN module, in order to enhance the semantic representation, a regularizer Orthogonal Regularizer is proposed to encourage the orthogonality between all word score vectors. The orthogonal regularizer is expressed as I is the identity matrix, the subscript F represents the Frobenius norm, A sem represents the attention score matrix obtained by the self-attention mechanism in the semantic graph convolutional neural network based on the self-attention mechanism to calculate the attention of the sample sentence. Therefore, A sem A semTEach off-diagonal element of is minimized to keep the orthogonality of the matrix Asem. The present invention expects that the two types of feature representations learned from the SynGCN and SemGCN modules represent different information contained in the grammatical dependency tree and the semantic relevance. Therefore, the present invention adopts differential regularization between the two adjacency matrices of the SynGCN and SemGCN modules. Differential regularization is R D =1 / ||A sem -A syn || F , A syn represents the dependency arc probability matrix.

[0013] S32: define the loss function of the model; during the update process, the second grammatical graph feature is used as the hidden state feature input, and the hidden state feature and the first grammatical feature feature of the last update corresponding to the sample sentence are input into the convolutional neural network to obtain the updated first grammatical feature graph; at the same time, the second semantic graph feature is used as the hidden state input, and the semantic convolutional neural network based on the attention mechanism is executed to obtain the updated first semantic graph feature. The loss function is expressed as l T = lc + λ1R0 + λ2R D +λ3||θ||2,l T represents the loss function, lc represents the cross entropy loss, λ1, λ2 and λ3 represent the regularization weight coefficients, R o Represents the orthogonal regularization loss value, R D represents the difference regularization loss value, and θ represents all trainable parameters in the dual graph convolutional neural network model.

[0014] The beneficial effects of the present invention are as follows: the present invention takes into account the over-smoothing phenomenon that is prone to occur in convolutional neural networks, and thus uses multi-hop information instead of the original method of obtaining information only from neighbor points, expanding from neighbor nodes to non-neighbor nodes, thereby alleviating the over-smoothing phenomenon. At the same time, in order to better understand the sentiment polarity in different contexts and increase the attention to context words, the composition tree relationship of the sentence is integrated, replacing the original perspective of only the dependency tree, and combining the syntactic information at the phrase level of the sentence composition tree and the syntactic information at the clause level in the dependency tree, thereby improving the accuracy of sentiment polarity judgment.

[0015] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0017] Figure 1 Schematic diagram of the tree structure;

[0018] Figure 2 This is a schematic diagram of the dependency tree structure;

[0019] Figure 3 This is a diagram of the method model provided by the present invention. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0021] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0022] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0023] The present invention discloses a text sentiment analysis model based on DualGCN combined with component trees and multi-hop information. The detailed steps are as follows:

[0024] S1: Collect text datasets to obtain sentences with sentiment analysis and their aspect words.

[0025] S2: For the text dataset to be analyzed, the sentence to be analyzed can be represented as s = {w1, w2, ..., w n}, where w n For each word in the sentence, the nth word and the aspect words in the sentence are obtained, and a sentence-aspect pair (s, a) is constructed, where a = {a1, a2, ..., a m}. That is, a is an aspect word in sentence s, a m For the mth aspect word in sentence s, use the pre-trained GloVe to convert each word in the sentence to be analyzed into the corresponding word vector and part-of-speech information. It can be expressed as x = {x1, x2, ..., x n}, x n is the word vector corresponding to the nth word. The word vector and part-of-speech information are input into the Bert encoder and dependency parser respectively to obtain the index set of subwords in the BERT sequence. According to the constituent tree that expresses the syntactic structure of the sentence and the dependency tree that expresses the dependency relationship between words in the sentence, the structure of the constituent tree is as follows: Figure 1 As shown in , only the leaf nodes are associated with the words in the input sentence, and the other intermediate nodes are all marked phrase components; the structure of the dependency tree is as follows Figure 2 As shown in the figure, it is used to express the dependency relationship between words in a sentence, analyze and identify grammatical components such as "subject, predicate, object", "attributive, adverbial, complement" in a sentence, and each node is a word. The composition tree and dependency tree are used to construct the graph. According to each layer of the composition tree, there are several phrases in the sentence {ph 1 u}, construct its adjacency matrix CA according to the phrase, and construct its adjacency matrix DA according to the dependency tree. Then, the CA and DA feature matrices are fused according to the grammar encoder. The grammar encoder is composed of several designed hierarchical graph attention blocks stacked together. Under the guidance of the grammatical information of the composition tree and the dependency tree, the corresponding graph is constructed. After fusion, A syn As the input of the SynGCN module. At the same time, the hidden state vector output by BERT is input into the SemGCN module. Based on the self-attention mechanism, the attention score of each pair of elements can be calculated in parallel, and the calculated attention score matrix A sem As the adjacency matrix of the SemGCN module, its attention score matrix in Q and K are the graph representations of the previous layer of SemGCN, WQ and WK represent learnable parameter weight matrices, d is the dimension of the input node features, g is the number of attention channels, and dk is the scaling factor. When constructing the adjacency matrix in the SemGCN module, in order to enhance the semantic representation, a regularizer OrthogonalRegularizer is proposed to encourage orthogonality between all word score vectors.

[0026] S3: transform the matrix A sem and A syn Input into the GCN with multi-hop information, which can alleviate the over-smoothing phenomenon in DualGCN by expanding from neighbor nodes to non-neighbor nodes and increasing the receptive field of each layer. Figure 3 Schematic diagram of adding neighbor nodes. The present invention expects that the two types of feature representations learned from the SynGCN and SemGCN modules represent different information contained in the grammatical dependency tree and semantic relevance. Therefore, the present invention adopts differential regularization between the two adjacency matrices of the SynGCN and SemGCN modules.

[0027] S4: Based on the dual affine module in DualGCN, grammatical features and semantic features are communicated. SynGCN uses the hidden state vector of BERT as the initial node representation in the syntactic graph and obtains the grammatical graph representation through the SynGCN module. h n syn Represented as the nth hidden node representation, and similarly, the SemGCN module obtains the semantic graph representation The dual affine module exchange process is: Among them, W1W2 are controllable parameters.

[0028] S5: Perform concatenation and pooling operations on the extracted aspect word node features. The specific pooling and concatenation processing is as follows: Where f is the average pooling function on the application-side node representation, γ represents the concatenated features, and [,] represents the concatenation operation.

[0029] S6: Input the concatenated features into the linear layer, and then use the softmax function to obtain the emotion probability distribution P. The specific operation is represented by p (a) =softmax(w p γ+b P ), where w p and b p are learnable weights and biases.

[0030] Figure 3 This is a diagram of the method model provided by the present invention.

[0031] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A text sentiment analysis method based on DualGCN, Features: The method comprises the following steps: S1: Use the pre-trained GloVe word embedding to obtain the vector and word vector corresponding to each word, and use the pre-trained language representation model BERT as the encoder to obtain the hidden state vector of the sentence; S2: Use the dependency parser to obtain the sentence dependency tree, and construct its corresponding adjacency matrix DA based on the dependency tree to provide all potential syntactic structures to capture rich structural information in the SynGCN module. At the same time, introduce the composition tree. Each layer of the composition tree consists of several phrases. According to these phrases, design their adjacency matrix CA. The two types of syntactic information are fused, and the matrix DA and CA are fused to obtain the matrix A. syn , A syn It contains both phrase-level syntactic information and clause-level syntactic information, and can use syntactic information more accurately; S3: Construct the grammatical module SynGCN and the semantic module SemGCN based on the convolutional neural network model, respectively mine the grammatical information and semantic information of the sentence. In order to alleviate the transition smoothing phenomenon commonly seen in the GCN model, multi-hop information is introduced in the two sub-convolutional neural networks, instead of the original information only obtained from neighbor nodes, and expand from neighbor nodes to non-neighbor nodes to alleviate the over-smoothing phenomenon; the hidden state vector obtained by S1 is input into SynGCN and SemGCN respectively; S2 is fused into the feature matrix A composed of the tree Con.Tree and the dependency tree Dep.Tree information syn Input into the SynGCN module to obtain the first grammatical graph feature corresponding to the sentence; in the SemGCN module, the adjacency matrix A is constructed through the self-attention mechanism sem , extract semantic features from the sentence to be analyzed and obtain the corresponding first semantic graph features; S4: Input the obtained first syntax graph feature and the first semantic graph feature into the dual affine module in DualGCN, perform feature exchange on the first syntax graph feature and the first semantic graph feature respectively, fuse the semantic and syntax information, and obtain the second syntax graph feature and the second semantic feature; S5: Input the second syntax graph feature and the second semantic graph feature into the pooling layer of DualGCN respectively, perform average pooling and concatenation operations on the extracted node features, and obtain the first concatenation feature; S6: Use the fully connected layer of DualGCN to perform probability analysis on the first concatenated feature to obtain the sentiment polarity results corresponding to words in different aspects, and classify them through the softmax function; The sentence to be analyzed is represented as s={w 1 ,w 2 ,…,w n }, where w n To obtain the nth word in the sentence and the aspect words in the sentence, construct the sentence-aspect pair (s, a), where a = {a 1 ,a 2 ,…,a m }; that is, a is an aspect word in sentence s, a m is the mth aspect word in sentence s. The pre-trained GloVe is used to convert each word in the sentence to be analyzed into the corresponding word vector and part-of-speech information; it is represented as x = {x 1 ,x 2 ,…,x n }, x n is the word vector corresponding to the nth word; the word vector and part-of-speech information are input into the Bert encoder and dependency parser respectively to obtain the index set of subwords in the BERT sequence. In the composition tree expressing the syntactic structure of the sentence and the dependency tree expressing the dependency relationship between words in the sentence, for the composition tree, only the leaf nodes are associated with the words in the input sentence, and the other intermediate nodes are all marked phrase components; for the dependency tree, it is used to express the dependency relationship between words in the sentence, analyze and identify the grammatical components of "subject, predicate, object" and "attributive, adverbial, complement" in the sentence, and each node is a word; the composition tree and dependency tree are used to construct the graph, and each layer l of the composition tree consists of several phrases of the sentence {ph l u }, construct its adjacency matrix CA according to the phrase, and construct its adjacency matrix DA according to the dependency tree; then fuse the CA and DA feature matrices according to the grammar encoder. The grammar encoder is composed of several designed hierarchical graph attention blocks stacked together, and constructs the corresponding graph under the guidance of the grammatical information of the composition tree and the dependency tree. After fusion, A syn As the input of the SynGCN module; at the same time, the hidden state vector output by BERT is input into the SemGCN module, and the attention score of each pair of elements is calculated in parallel based on the self-attention mechanism, and the calculated attention score matrix A is sem As the adjacency matrix of the SemGCN module, its attention score matrix in Q and K are the graph representations of the previous layer of SemGCN, W Q and W K represents the learnable parameter weight matrix, d is the dimension of the input node feature, g is the number of attention channels, and d k is the scaling factor; when constructing the adjacency matrix in the SemGCN module, in order to enhance the semantic representation, a regularizer Orthogonal Regularizer is proposed to encourage the orthogonality between all word score vectors.

2. According to the text sentiment analysis method based on DualGCN according to claim 1, Features: The matrix A sem and A syn Input into the GCN with multi-hop information. The multi-hop information alleviates the over-smoothing phenomenon in DualGCN and increases the receptive field of each layer by expanding from neighbor nodes to non-neighbor nodes. Differentiable regularization is adopted between the two adjacency matrices of SynGCN and SemGCN modules.

3. According to the text sentiment analysis method based on DualGCN according to claim 1, Features: The SynGCN uses the hidden state vector of BERT as the initial node representation in the syntactic graph and obtains the syntactic graph representation through the SynGCN module. h n syn Represented as the nth hidden node, and similarly, the SemGCN module obtains the semantic graph representation The dual affine module exchange process is: Where W 1 W 2 is a controllable parameter.

4. According to the text sentiment analysis method based on DualGCN according to claim 1, Features: The operations of performing average pooling and concatenation processing on the extracted node features are specifically as follows: Where f is the average pooling function on the application node representation, γ represents the concatenated features, and [,] represents the concatenation operation. The concatenated features are input into the linear layer, and then the softmax function is used to obtain the emotion probability distribution P. The specific operation is expressed as P = softmax(w p γ+b p ), where w p and b p are the learnable weights and biases.

Images