A text sentiment multi-classification method based on dual-graph convolutional network with fused label graph

Abstract

The present invention relates to a text sentiment multi-classification method based on a dual-graph convolutional network fused with a label graph, and belongs to the field of computer technology. The method comprises the following steps: S1: sentence encoding layer; S2: label graph embedding; S3: syntactic graph convolutional network SynGCN; S4: semantic graph convolutional network SemGCN; S5: dual affine transformation. In view of the fact that most existing studies only consider the semantic representation of sentences and ignore the grammatical information of sentences, the present invention invents a dual-graph convolutional network, namely a syntactic graph convolutional network and a semantic graph convolutional network. The method can simultaneously extract the syntactic information and semantic information of sentences. Previous models ignored the semantic information of labels. Based on this, the present invention proposes label graph embedding. By fusing the label graph representation and the semantic graph convolutional network, the model can learn more accurate semantic representation and capture the potential relationship between labels and words in the text.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology and relates to a text sentiment multi-classification method based on a dual-graph convolutional network fused with a label graph. Background Art

[0002] Text multi-sentiment analysis is a more fine-grained sentiment classification of text with emotional colors, such as happiness, fear, anger, sadness, etc., which has very important reference and research value for corporate market research, personal consumption choices, etc. However, most of the research in the field of text sentiment analysis is based on binary or ternary classification tasks, that is, describing the text from positive, negative or neutral aspects, and lacks research on more detailed sentiment classification.

[0003] With the development of deep learning, neural networks have achieved good performance in text multi-sentiment analysis. For example, in 2013, foreign scholars R. Socher et al. proposed the SST-5 dataset for sentiment multi-classification tasks, and proposed the recursive neural tensor network (RNTN), which constructed the representation of the entire sentence based on the sentence structure. In 2015, domestic researchers C. Zhou et al. proposed C-LSTM by combining convolutional neural network (CNN) and recurrent neural network (RNN). C-LSTM uses CNN to extract high-level phrase representations, and then inputs them into LSTM to obtain sentence representations. Peters et al. proposed the pre-training method ELMo in 2018. This method uses a bidirectional LSTM language model, which consists of a forward and a backward language model. Using the pre-trained bidirectional language model, the context-dependent current word representation can be obtained from the language model according to its input. In the same year, Google proposed a new method based on BERT. Through unsupervised learning in a massive corpus, the pre-trained context-dependent word vector encoding mechanism is used for the language model, which fully considers the contextual semantic information of the word. In subsequent studies, most researchers achieved good results in sentiment classification by fine-tuning the BERT model.

[0004] (1) Existing methods mainly rely on sentence representation without considering the grammatical information of the sentence, resulting in poor model classification effect;

[0005] (2) Through research, it is found that there is semantic similarity between different sentiment labels in the text sentiment multi-classification task. Previous studies usually regard labels as symbols without semantics, ignoring the semantic relationship between labels, resulting in information loss. Summary of the invention

[0006] In view of this, the object of the present invention is to provide a text sentiment multi-classification method based on a dual-graph convolutional network fused with a label graph.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A text sentiment multi-classification method based on a dual-graph convolutional network fused with a label graph, the method comprising the following steps:

[0009] S1: Establish sentence encoding layer;

[0010] S2: Embedding label map;

[0011] S3: Establish the Syntactic Graph Convolutional Network SynGCN;

[0012] S4: Establish the semantic graph convolutional network SemGCN;

[0013] S5: Perform double affine transformation.

[0014] Optionally, the S1 is specifically:

[0015] To obtain the text representation of each sentence, the model uses BERT and BiLSTM as sentence encoders to extract hidden context representations; the pre-trained model BERT is used to generate word vectors, and dynamic word vectors are generated according to the context in which the words are located; then the word vectors obtained by BERT are input into BiLSTM to calculate the hidden state vector representation H = [h1,h2,...,h n ] The obtained hidden state vector is used as the input of the subsequent module; where h i Represents the word vector representation of the i-th word in a sentence, and n represents the number of words in the input sentence.

[0016] Optionally, S2 is specifically:

[0017] The label graph is defined as G = {V, e}; where V represents the nodes in the graph, that is, the representation of the label [c1, c2, ..., c m ], c i Then represents the vector representation of the i-th label, m represents the number of labels, and e represents the edge in the graph, that is, the semantic relationship between labels;

[0018] To obtain the representation of each tag, the text content of the tag is used as input, and the word vector average function is used to calculate the word vector representation of the tag. The calculation is shown in formula (1);

[0019]

[0020] c i Represents the vector representation of the i-th sentiment label, k i Indicates the number of samples of the i-th label H jRepresents the hidden state vector representation of the jth sentence in the i-th label sample; in order to obtain the relationship between labels, the correlation coefficient between labels is used as the adjacency matrix A of the label graph label ; Use the cosine similarity between two vectors to calculate the similarity between different labels, that is, c i 、c j Represents the vector representation of two different labels, || || represents the modulus operation of the vector; then use GCN to update the representation of the label, the formula is as follows:

[0021]

[0022] Where l represents the number of layers of the graph convolutional network, represents the representation of the i-th label of the graph volume and the l-th layer of the network, represents the representation of the jth label in the l-1th layer of the graph convolutional network, W and b are the learned weights and biases, σ is the activation function, and m is the number of labels;

[0023] In order to effectively use the label semantic information, the semantic components related to the words and labels in the sentence are calculated through label attention; the calculation formula is as follows:

[0024]

[0025]

[0026] H label =HU T (5)

[0027] h j Represents the vector representation of the jth word in the sentence; c i represents the label representation after GCN update in formula (2), T represents the transpose of the vector, α ij represents the information content of the feature vector of the jth word in the sentence for the i-th label; U = [u1,u2,...,u m ], where u i Indicates the i-th label c i The amount of semantic information carried in the sentence; H represents the contextual hidden representation obtained by BiLSTM; H label A specific label representation for each word in each sentence.

[0028] Optionally, the S3 is specifically:

[0029] The SynGCN module takes the syntactic encoding of the sentence as input; to encode the syntactic information, the syntactic parsing method PerturbedMasking is used; given an input text X = [x1, x2, ..., xn ], x i Represents the i-th word in the sentence, and each word x in the text i Mapped into a contextual representation H θ (x) i ; First, use [MASK] to replace the word x i , and replace the sentence x\{x i} Input to BERT and get word x i The vector representation H θ (x\{x i}) i ; Secondly, calculate x j ∈x\{x i} θ (x\{x i}) i The influence of x j Replace with [MASK] to get the second sentence x\{x i ,x j}, the obtained x j The new vector represents H θ (x\{x i ,x j}) i ; Based on the representation of two vectors, define the function f(x i ,x j );

[0030] f(x i ,x j )=d(H θ (x\{x i}) i ,H θ (x\{x i ,x j}) i ) (6)

[0031] Where d(x,y) is a function that measures the distance between two vectors, expressed as Euclidean distance; by i , x j Repeat the above steps to obtain an influence matrix A syn ; The hidden state vector H of BiLSTM is used as the initial node representation in the grammar graph; then the syntactic graph representation is obtained represents the syntactic representation of the i-th word in the sentence. The node calculation in the syntactic graph is shown in formula (7);

[0032]

[0033] Where l represents the number of layers of the graph convolutional network, represents the representation of the i-th word in the graph volume and the l-th layer of the network, W and b are the learned weights and biases, and σ is the activation function.

[0034] Optionally, the S4 is specifically:

[0035] In the SemGCN module, the self-attention mechanism is used to calculate the attention score of each pair of elements, and the obtained attention score matrix A sem As the adjacency matrix of the SemGCN module, the specific calculation formula is as follows:

[0036]

[0037] Softmax is a normalized exponential function, Q and K are feature vectors obtained by the attention mechanism, and W Q , W K is the weight matrix, T represents the vector transpose, and d represents the dimension of the feature vector;

[0038] Then, similar to SynGCN, the H obtained by label attention is label As the initial node representation in the semantic graph; then the semantic graph representation is obtained The node calculation in the figure is shown in formula (9);

[0039]

[0040] Where W and b are the learned weights and biases, and σ is the activation function.

[0041] Optionally, the S5 is specifically:

[0042] In order to make the feature interaction between SynGCN and SemGCN modules use a double affine transformation MutulaBiAffine, the specific process is as follows:

[0043] H syn′ =softmax(H syn W1(H sem ) T )H sem (10)

[0044] H sem′ =softmax(H sem W2(H syn ) T )H syn (11)

[0045] Where W1 and W2 are trainable parameters;

[0046] The output of the SynGCN and SemGCN modules after the double affine transformation is subjected to average pooling and concatenation operations to obtain the final feature representation. The calculation formula is as follows:

[0047]

[0048]

[0049] r=[H syn ,H sem ] (14)

[0050] Where f represents the average pooling function, are the semantic and syntactic representations of the i-th word after double affine transformation, and r represents the final feature representation obtained by concatenating the outputs of the semantic graph convolution network and the syntactic graph convolution network. Finally, the softmax function is used to generate a sentiment probability distribution, and the category with the largest probability value is taken as the classified category. The calculation is shown in formula (15):

[0051] p(x)=softmax(W P r+b p ) (15)

[0052] Where W P and b p are the learnable weights and biases.

[0053] The beneficial effects of the present invention are:

[0054] (1) In view of the fact that most existing studies only consider the semantic representation of sentences and ignore the grammatical information of sentences, the present invention invents a dual graph convolutional network, namely a syntactic graph convolutional network and a semantic graph convolutional network. This method can simultaneously extract the syntactic information and semantic information of a sentence.

[0055] (2) Previous models ignored the semantic information of labels. Based on this, the present invention proposed label graph embedding. By fusing label graph representation with semantic graph convolutional network, the model can learn more accurate semantic representation and capture the potential relationship between labels and words in text.

[0056] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0058] Figure 1 It is a framework diagram of the present invention;

[0059] Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0061] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0062] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0063] See also Figure 1 and Figure 2 The specific implementation steps of the present invention are as follows:

[0064] Step 1: Use BiLSTM and BERT to calculate the input sentence input text X = [x1, x2, ..., x n ] word vector representation H = [h1,h2,...,h n ], and then input the hidden representation of the sentence into the syntactic graph convolutional network module and the semantic graph convolutional network module respectively;

[0065] Step 2: Use the word vector average to calculate the word vector representation of the label.

[0066] Step 3: Update the label representation using GCN

[0067] Step 4: Use label attention mechanism to learn label-specific text representations.

[0068]

[0069]

[0070] H label =HU T

[0071] Step 5: Use f(x i ,x j )=d(H θ (x\{x i}) i ,H θ (x\{x i ,x j}) i ) Calculate the adjacency matrix of the syntactic graph, use the context hidden representation obtained in step 1 as the initial node in the syntactic graph, and then use the graph convolutional network to encode the sentence syntactic information

[0072] Step 6: Use the self-attention mechanism to obtain the attention matrix as the adjacency matrix. The calculation formula is: At the same time, the sentence representation obtained in step 4 is used as the initial representation of the node in the semantic graph. Then GCN is used to encode the semantic information of the sentence.

[0073] Step 7: Use double affine transformation to interact the features between semantic graph convolutional networks and semantic graph convolutional networks.

[0074] H syn′ =softmax(H syn W1(H sem ) T )H sem

[0075] H sem′ =softmax(H sem W2(H syn ) T )H syn

[0076] Step 8: Use average pooling and concatenation operations to obtain the final feature representation obtained in step 7.

[0077] Step 9: Finally, use the softmax function to generate a sentiment probability distribution p(x) = softmax(W P r+b p )The category with the largest probability value is taken as the classified category.

[0078] The experimental data sets of this invention are: GoEmotions, SST-5

[0079] The evaluation indicators are: precision, recall and comprehensive evaluation indicator F1 value.

[0080] The experimental parameters are set as follows: first, BERT and BiLSTM are used to obtain the feature representation of the sentence, and the hidden size of the feature is set to 50; in order to reduce overfitting, dropout is applied to the input word embedding of BiLSTM, and the parameter used is 0.7; the dropout of SynGCN and SemGCN is 0.1; all GCN layers in this article are 2 layers; Adam is used as the optimizer, and its learning rate is 0.002; finally, the DualGCN model of this article is trained in 50 epochs; the regularization coefficients λ1, λ2 and λ3 are set to 0.2, 0.3 and 0.0001.

[0081] The experimental environment is: a laptop with Intel(R) Core(TM) i5-9300H CPU @ 2.4GHz, 16GB memory, and Win10 64-bit operating system. The model is trained based on the PyTorch framework and NVIDIA TESLAP100 GPU.

[0082] The experimental results are shown in Tables 1 and 2.

[0083] Table 1 Evaluation results on the GoEmotions dataset

[0084]

[0085] Table 2 Evaluation results on the SST-5 dataset

[0086]

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A text sentiment multi-classification method based on a dual-graph convolutional network fused with a label graph, characterized by: The method comprises the following steps: S1: Establish sentence encoding layer; S2: Embedding label map; S3: Establish the Syntactic Graph Convolutional Network SynGCN; S4: Establish the semantic graph convolutional network SemGCN; S5: perform double affine transformation; The S1 is specifically: To obtain the text representation of each sentence, the model uses BERT and BiLSTM as sentence encoders to extract hidden context representations; the pre-trained model BERT is used to generate word vectors, and dynamic word vectors are generated according to the context in which the words are located; then the word vectors obtained by BERT are input into BiLSTM to calculate the hidden state vector representation H = [h1,h2,...,h n ] The obtained hidden state vector is used as the input of the subsequent module; where h i represents the word vector representation of the i-th word in a sentence, and n represents the number of words in the input sentence; The S2 is specifically: The label graph is defined as G = {V, e}; where V represents the nodes in the graph, that is, the representation of the label [c1, c2, ..., c m ], c i Then represents the vector representation of the i-th label, m represents the number of labels, and e represents the edge in the graph, that is, the semantic relationship between labels; To obtain the representation of each tag, the text content of the tag is used as input, and the word vector average function is used to calculate the word vector representation of the tag. The calculation is shown in formula (1); c i The vector representation of the i-th sentiment label, k i Indicates the number of samples of the i-th label H j Represents the hidden state vector representation of the jth sentence in the i-th label sample; in order to obtain the relationship between labels, the correlation coefficient between labels is used as the adjacency matrix A of the label graph label ; Use the cosine similarity between two vectors to calculate the similarity between different labels, that is, c i 、c j Represents the vector representation of two different labels, || || represents the modulus operation of the vector; then use GCN to update the representation of the label, the formula is as follows: Where l represents the number of layers of the graph convolutional network, represents the representation of the i-th label of the graph volume and the l-th layer of the network, represents the representation of the jth label in the l-1th layer of the graph convolutional network, W and b are the learned weights and biases, σ is the activation function, and m is the number of labels; In order to effectively use the label semantic information, the semantic components related to the words and labels in the sentence are calculated through label attention; the calculation formula is as follows: H label =HU T (5) h j represents the vector representation of the jth word in the sentence; c i represents the label representation after GCN update in formula (2), T represents the transpose of the vector, and α ij represents the information content of the feature vector of the jth word in the sentence for the i-th label; U = [u1,u2,...,u m ], where u i Indicates the i-th label c i The amount of semantic information carried in the sentence; H represents the contextual hidden representation obtained by BiLSTM; H label A specific label representation for each word in each sentence.

2. According to claim 1, a text sentiment multi-classification method based on a dual-graph convolutional network fused with a label graph is characterized by: The S3 is specifically: The SynGCN module takes the syntactic encoding of the sentence as input; to encode the syntactic information, the syntactic parsing method PerturbedMasking is used; given an input text X = [x1, x2, ..., x n ], x i Represents the i-th word in the sentence, and each word x in the text i Mapped into a contextual representation H θ (x) i ; First, use [MASK] to replace the word x i , and replace the sentence x\{x i } Input to BERT and get word x i The vector representation H θ (x\{x i }) i ; Secondly, calculate x j ∈x\{x i } for H θ (x\{x i }) i The influence of x j Replace with [MASK] to get the second sentence x\{x i ,x j }, the obtained x j The new vector represents H θ (x\{x i ,x j }) i ; Based on the representation of two vectors, define the function f(x i ,x j ); f(x i ,x j )=d(H θ (x\{x i }) i ,H θ (x\{x i ,x j }) i ) (6) Where d(x,y) is a function that measures the distance between two vectors, expressed as Euclidean distance; by i , x j Repeat the above steps to obtain an influence matrix A syn ; The hidden state vector H of BiLSTM is used as the initial node representation in the grammar graph; then the syntactic graph representation is obtained represents the syntactic representation of the i-th word in the sentence. The node calculation in the syntactic graph is shown in formula (7); Where l represents the number of layers of the graph convolutional network, represents the representation of the i-th word in the graph volume and the l-th layer of the network, W and b are the learned weights and biases, and σ is the activation function.

3. According to claim 2, a text sentiment multi-classification method based on a dual-graph convolutional network fused with a label graph is characterized by: The S4 is specifically: In the SemGCN module, the self-attention mechanism is used to calculate the attention score of each pair of elements, and the obtained attention score matrix A sem As the adjacency matrix of the SemGCN module, the specific calculation formula is as follows: Softmax is a normalized exponential function, Q and K are feature vectors obtained by the attention mechanism, and W Q , W K is the weight matrix, T represents the vector transpose, and d represents the dimension of the feature vector; Then, similar to SynGCN, the H obtained by label attention is label As the initial node representation in the semantic graph; then the semantic graph representation is obtained The node calculation in the figure is shown in formula (9); Where W and b are the learned weights and biases, and σ is the activation function.

4. The text sentiment multi-classification method based on a dual-graph convolutional network fused with a label graph according to claim 3 is characterized by: The S5 is specifically: In order to make the feature interaction between SynGCN and SemGCN modules use a double affine transformation MutulaBiAffine, the specific process is as follows: H syn′ =softmax(H syn W1(H sem ) T )H sem (10) H sem′ =softmax(H sem W2(H syn ) T )H syn (11) Where W1 and W2 are trainable parameters; The output of the SynGCN and SemGCN modules after the double affine transformation is subjected to average pooling and concatenation operations to obtain the final feature representation. The calculation formula is as follows: r=[H syn ,H sem ] (14) Where f represents the average pooling function, are the semantic and syntactic representations of the i-th word after double affine transformation, and r represents the final feature representation obtained by concatenating the outputs of the semantic graph convolution network and the syntactic graph convolution network. Finally, the softmax function is used to generate a sentiment probability distribution, and the category with the largest probability value is taken as the classified category. The calculation is shown in formula (15): p(x)=softmax(W P r+b p ) (15) Where W P and b p are the learnable weights and biases.

Images