Aspect-level sentiment analysis method combining syntax tree and graph network

Abstract

In order to solve the problem that in the prior art, context capacity related to target aspect words is difficult to capture according to a sequential text sequence, the invention provides an aspect-level sentiment analysis method combining a syntax tree and a graph network, and the aspect-level sentiment analysis method comprises a training process and a prediction process; in the training process, firstly, text sample data are obtained and preprocessed; then, word vector extraction is carried out on the text data through a BERT encoder; forming a mask matrix by calculating aspect perception attention, self-attention, a syntactic dependency tree and a semantic composition tree; and inputting the word vector and the mask matrix into the graph neural network, taking the final output as an emotion polarity prediction result, training model parameters by adopting a gradient descent method, and obtaining a trained BERT encoder model and the graph neural network. The method is applied to a prediction process, word vectors extracted from a text are predicted through a graph neural network which has learned the ability of analyzing a syntax tree structure, and finally the sentiment polarity of words in a target aspect is obtained.

Description

Technical Field

[0001] The present invention relates to natural language processing technology, and in particular to aspect-level sentiment analysis technology. Background Art

[0002] Text sentiment analysis is a text analysis technology, also known as opinion mining, which is a study on the calculation of people's opinions, evaluations, attitudes and emotions expressed on entities. Text sentiment analysis can be divided into paragraph-level sentiment analysis, sentence-level sentiment analysis and aspect-level sentiment analysis according to the analysis granularity. In the early stage, paragraph-level and sentence-level sentiment analysis tasks were the focus of research. They assume that a piece of text has only one sentiment, and analyze a given text and judge its overall sentiment polarity. Sentiment polarity is used to determine whether the sentiment expressed in the text is positive, negative or neutral. However, overall sentiment analysis of the text will obscure its details, and the overall sentiment cannot reflect people's fine-grained emotional expression of opinion targets. If you only focus on the overall sentiment and ignore the specific details, you may calculate incorrect results in real-world applications such as recommendation systems and question-answering systems. Therefore, in order to conduct a more complete sentiment analysis, the system needs to discover various aspects of the text review and determine the emotional information expressed by the text for each aspect. This is the aspect-level sentiment analysis technology.

[0003] The goal of aspect-level sentiment analysis is to find the target aspect words that need to be evaluated and their corresponding sentiment polarities. The target aspect words can be extracted from the actual review text or predefined.

[0004] Aspect-level sentiment analysis aims to identify the sentiment inclination towards a specific aspect or entity in a text and output the sentiment polarity of the specific aspect or entity. This technology is of great value in many application areas, such as product review analysis, social media public opinion monitoring, etc.

[0005] In aspect-level sentiment analysis, the BERT model can be used for text encoding and sentiment classification. First, the text is input into the BERT model, and the pre-trained language representation capability is used to convert the text into a high-dimensional semantic representation. Then, a classifier can be built on the semantic representation output by BERT to identify the sentiment tendency towards a specific aspect in the text. This classifier can be a simple fully connected layer neural network or a more complex model, designed according to the specific situation.

[0006] The advantage of using the BERT model is its powerful language understanding and context awareness capabilities, which enable the model to more accurately understand the meaning and sentiment of the text. By fine-tuning the pre-trained model, good performance on aspect-level sentiment analysis tasks can be achieved without the need for a large amount of labeled data. However, BERT, which is based on a sequence data structure, cannot capture sentence information composed of graph structures, and therefore lacks the ability to capture context related to the target aspect words. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide an aspect-level sentiment analysis method that improves the ability to analyze the context of a sentence, in view of the current situation in the prior art that the deep semantics of a sentence is not captured by a sentence-like grammatical structure during training.

[0008] The technical solution adopted by the present invention to solve the above technical problems is an aspect-level sentiment analysis method combining a syntax tree and a graph network, comprising the following steps:

[0009] A. Training process:

[0010] A1. Obtain text samples and their corresponding target aspect words, and preprocess the text samples;

[0011] A2. Use multiple Transformer structures to segment the preprocessed text samples through the BERT encoder to obtain word vectors and output them to the graph neural network for learning;

[0012] A3. Construct aspect-aware attention mechanism and self-attention mechanism according to the received word vector; add the aspect-aware attention mechanism coefficient and the self-attention mechanism coefficient as the initialization matrix; construct the syntactic dependency adjacency matrix according to the dependency distance of each word in the syntactic dependency tree; construct the semantic composition adjacency matrix according to the composition distance of each word in the semantic composition tree; add and fuse the initialization matrix, the syntactic dependency adjacency matrix and the semantic composition adjacency matrix to obtain a mask matrix containing graph information and output it to the graph neural network; the graph neural network extracts semantic features related to the target aspect word from the syntactic dependency structure and semantic composition structure of the text according to the received word vector and the mask matrix; output the semantic features to the fully connected layer, and the fully connected layer outputs the predicted probability distribution of the sentiment polarity of the target aspect word;

[0013] A4. Using the gradient descent method, taking the sentiment polarity label of the target aspect word as a constraint, optimize the parameters of the BERT encoder and the graph neural network to obtain a trained BERT model and graph neural network;

[0014] B. Prediction process:

[0015] B1. Preprocess the text data to be predicted and confirm the target aspect words to be predicted;

[0016] B2. Use the trained BERT model to extract features from the preprocessed text data to be predicted to obtain word vectors;

[0017] B3. Input the word vector into the trained graph neural network, and the graph neural network outputs the sentiment polarity of the target word.

[0018] The beneficial effects of the present invention are:

[0019] (1) The present invention can automatically determine the sentiment polarity of a given aspect of the input text. Users can directly use the sentiment polarity to determine relevant content, which reduces workload and errors.

[0020] (2) During the training process, there is no need to pre-process the text or perform word segmentation operations in advance. Instead, only the text needs to be obtained as training data, which reduces the workload and avoids the problem of information loss that is easily caused by manual processing of the original text.

[0021] (3) The performance of the graph neural network based on the syntax tree structure is significantly better than the previous LSTM network and the ordinary pre-trained model: First, the graph neural network is more complex than the LSTM network in terms of network structure, and the network's ability to extract semantic features related to text and tree structure is much stronger than the LSTM network, making the final sentiment polarity prediction generated by the system more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the model training process in an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the algorithm for model training in an embodiment of the present invention;

[0024] Figure 3 4 is a flow chart of the model prediction process in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The aspect-level sentiment analysis method combining syntax tree and graph network in this embodiment includes two parts: training process and prediction process, which are described in detail below with reference to the accompanying drawings:

[0026] See also Figure 1 , the training process includes:

[0027] S1. Text sample acquisition and preprocessing: extract sentences from text samples, annotate target aspect words and corresponding sentiment polarity labels in sentences, and control the maximum length of sentences to ensure uniform sentence span.

[0028] S11. Obtain text samples, such as review data on Internet shopping websites or social media such as Weibo, and sentiment polarity labels of the reviews. These data mainly include review data and target aspect words to be predicted in the sentence;

[0029] S12, extract the sentences from the data, the target aspect words in the sentences and the corresponding sentiment polarity need to be manually annotated, the extracted text and a single target aspect word and a sentiment polarity correspond one to one, and will be used as a training data set;

[0030] S13, unifying the size of the text data, first unifying the length of the sentences, for example, the maximum length of each sentence is 100.

[0031] S2, using the Chinese-adapted BERT encoder, uses multiple Transformer structures to segment the preprocessed text samples to obtain word vectors and output them to the graph neural network for learning. The BERT encoder first segments the text to extract word vectors, including:

[0032] S21. Build the BERT encoder network;

[0033] The BERT encoder network built in this embodiment is composed of two parallel Transformer structures; each Transformer structure contains a multi-head attention layer, two residual networks, two normalization layers and a feedforward layer; the two residual networks are respectively adapted to the multi-head attention layer and the feedforward layer; the two normalization layers are also placed behind the multi-head attention layer and the feedforward layer; finally, 12 such structures are stacked together.

[0034] In the above BERT encoder network, the multi-head attention layer is used to extract deep semantic feature information of the text, the residual network is used to fit the classification function to improve the classification accuracy, the normalization layer is to solve the problem of difficulty in training multi-layer neural networks, and the feedforward layer is responsible for outputting the text feature vector.

[0035] S22. Input the preprocessed text sample into two parallel Transforme structures as BERT encoders to extract features. First, the preprocessed text sample passes through the BERT word segmenter, and each word in the sentence is converted into a specified word token through the word segmenter. Then, the continuous vector obtained after token vectorization is input into the multi-head attention layer to obtain a feature vector with a wider dimension. Finally, the intermediate feature vector is calculated through a network of multiple Transformer architectures. Finally, the feature vector of n tokens, i.e., the word vector, is obtained. The dimension of each word vector in the embodiment is d k ; The output word vector set is:

[0036] H={h1,h2,...h n}

[0037] S3. Use aspect-aware attention and self-attention mechanisms to mine the attention relationship between each word in the text sequence and the target aspect word, and use graph neural networks to learn the attention relationship between semantic features and the corresponding target aspect words:

[0038] S31 constructs an aspect-aware attention mechanism and a self-attention mechanism according to the received word vector; adds the aspect-aware attention mechanism coefficient and the self-attention mechanism coefficient as an initialization matrix; constructs a syntactic dependency adjacency matrix according to the dependency distance of each word in the syntactic dependency tree, and constructs a semantic composition adjacency matrix according to the composition distance of each word in the semantic composition tree; performs additive fusion of the initialization matrix, the syntactic dependency adjacency matrix and the semantic composition adjacency matrix to obtain a mask matrix containing graph information; specifically includes:

[0039] A311. Construct aspect-aware attention mechanism and self-attention mechanism based on the i-th word vector extracted by the BERT encoder.

[0040] Aspect-aware attention coefficient of the i-th word vector The calculation formula is:

[0041]

[0042] Among them, H a represents the representation vector of the target aspect word in the encoding layer, K represents the word representation vector set of the entire encoding layer, and W a and W k are two learnable fully connected layer parameters, b is a learnable bias parameter, and T represents transpose.

[0043] The self-attention coefficient of the i-th word vector The calculation formula is:

[0044]

[0045] Where Q and K both represent the word representation vector set of the entire encoding layer, W Q and W K are two learnable fully connected layer parameters, is the scaling factor.

[0046] A312, two attention coefficients and Add as the initialization matrix A of the subsequent graph neural network i .

[0047] A313. Construct a syntax-related adjacency matrix based on the sentence's syntactic dependency tree, treating each word token in the sentence as a node, and based on the distance D between each node:

[0048] D ij =mind(v i , v j )

[0049] Among them, i and j are the serial numbers of the nodes, v i , and v j represents the i-th and j-th nodes, mind represents the function of finding the shortest path distance between two nodes, D ij is the distance between the i-th and j-th nodes;

[0050] Syntactic dependency adjacency matrix M Dep The value of the element at position (i, j) is:

[0051]

[0052] Wherein, k is the preset path threshold.

[0053] A314, using the semantic composition tree to construct the semantic composition adjacency matrix M in the same way as A313 Con .

[0054] A315, using the initialization matrix A of the i-th word vector i , syntactic dependency adjacency matrix M Dep i And Yi form the adjacency matrix M Con i Construct a mask matrix containing graph information:

[0055]

[0056] Among them, softmax represents the normalized exponential function.

[0057] The S32 graph neural network adopts the structure of the graph convolutional network GCN. Based on the received word vectors and the constructed mask matrix containing graph information, the semantic features related to the target aspect words are extracted from the syntactic dependency structure and semantic composition structure of the text; the semantic features are output to the fully connected layer, and the fully connected layer outputs the predicted probability distribution of the sentiment polarity of the target aspect words:

[0058] S321. Suppose there are p different grammatical mask matrices and n nodes in a sentence, so A mask ∈R p×n×n . Randomly initialize a fully connected network, and the weight parameter between the nodes in the lth layer of the network is W l . For the i-th word vector hi Perform graph convolution operation. We will transform h 1 Represented as the output state of the lth layer in GCN, then h 0 Represented as the word vector input of GCN. Each node in the l-th layer GCN According to the representation of its neighborhood To update:

[0059]

[0060] Among them, σ represents the nonlinear activation function, A ij is the mask matrix The value of the element at position (i, j) in W l is the linear transformation weight of layer l, b l is the l-layer bias term, and the final output of the l-layer GNN is When the i-th word vector is used as the target aspect word, the final output of GNN is the semantic feature of the i-th word vector The semantic features of the target aspect word

[0061] S322. Output of the graph attention network We choose to use the fully connected layer and the Softmax activation function for calculation to obtain the predicted probability distribution p(a) of the sentiment polarity of the target aspect word:

[0062]

[0063] Among them, W p is the linear transformation weight of the fully connected layer, b p is the bias term of the fully connected layer.

[0064] S4. Use the gradient descent method to supervise and optimize the BERT encoder and graph attention network using the sentiment polarity labels obtained in step S1, and finally obtain a trained model for actual aspect-level sentiment analysis.

[0065] The algorithm principle of the model training in the above steps S1-S4 is as follows Figure 2 shown.

[0066] The prediction process flow in this embodiment is shown in Figure 3 , which includes:

[0067] S1. Data preprocessing:

[0068] In this step, the text to be predicted (such as product reviews, social comments, etc.) is preprocessed, uniformly scaled to a sequence of length 100, and the position of the target aspect word sentiment polarity is given.

[0069] S2. The BERT encoder, which has learned the ability to extract text features during training, extracts features from the data obtained from the previous text preprocessing:

[0070] In this step, the preprocessed text is input into the trained BERT encoder, which outputs L fixed-dimensional word vectors, each with a dimension of d. k =768.

[0071] S3. Determine the target aspect word through the graph attention network that has learned the dependent grammatical structure and composition structure related to the target aspect word:

[0072] In this step, the L text feature vectors are arranged into a sequence in sequence, and the trained graph attention network and aspect-aware attention are used to extract the information vector related to the target aspect word in the text sequence, and finally the sentiment polarity of the target aspect word is obtained.

Claims

1. A method for aspect-level sentiment analysis combining syntax tree and graph network, characterized in that: The following steps are involved: A. Training process: A1. Obtain text samples and their corresponding target aspect words, and preprocess the text samples; A2. Use multiple Transformer structures to segment the preprocessed text samples through the BERT encoder to obtain word vectors and output them to the graph neural network for learning; A3. Construct an aspect-aware attention mechanism and a self-attention mechanism based on the received word vector; add the aspect-aware attention mechanism coefficient and the self-attention mechanism coefficient as an initialization matrix; construct a syntactic dependency adjacency matrix based on the dependency distance of each word in the syntactic dependency tree; A semantic composition adjacency matrix is ​​constructed according to the composition distance of each word in the semantic composition tree; the initialization matrix, the syntactic dependency adjacency matrix and the semantic composition adjacency matrix are additively fused to obtain a mask matrix containing graph information and output to the graph neural network; the graph neural network extracts semantic features related to the target aspect word from the syntactic dependency structure and semantic composition structure of the text based on the received word vector and the mask matrix; the semantic features are output to the fully connected layer, and the fully connected layer outputs the predicted probability distribution of the sentiment polarity of the target aspect word; A4. Using the gradient descent method, taking the sentiment polarity label of the target aspect word as a constraint, optimize the parameters of the BERT encoder and the graph neural network to obtain a trained BERT model and graph neural network; B. Prediction process: B1. Preprocess the text data to be predicted and confirm the target aspect words to be predicted; B2. Use the trained BERT model to extract features from the preprocessed text data to be predicted to obtain word vectors; B3. Input the word vector into the trained graph neural network, and the graph neural network outputs the sentiment polarity of the target word.

2. The method according to claim 1, characterized in that: The preprocessing of the incoming text samples in step A1 specifically includes: extracting sentences from the text samples, marking the target aspect words in the sentences and the corresponding sentiment polarity labels; and controlling the maximum length of the sentences to ensure uniform sentence span.

3. The method according to claim 1, characterized in that: Aspect-aware attention mechanism coefficient A asp The calculation is: A asp =tanh(H a W a ×(KW k ) T +b) Among them, tanh is the hyperbolic tangent function, H a represents the representation vector of the target aspect word in the encoding layer, K represents the word vector set, and W a and W k are two learnable fully connected layer parameters, b is a learnable bias parameter, T Indicates transpose.

4. The method according to claim 1, characterized in that According to the dependency distance of each word in the syntactic dependency tree, the syntactic dependency adjacency matrix is ​​constructed by the dependency distance of each word in the syntactic dependency tree. The syntactic dependency adjacency matrix is ​​constructed as follows: Consider each word token in the sentence as a node, and according to the distance D between each node: D ij =mind(v i ,v j ) Among them, i and j are the serial numbers of the nodes, v i， With v j represents the i-th and j-th nodes, mind represents the function of finding the shortest path distance between two nodes, D ij is the distance between the i-th and j-th nodes; Syntactic dependency adjacency matrix M Dep The value of the element at position (i, j) is: Wherein, k is the preset path threshold.

5. The method according to claim 1, characterized in that The mask matrix is ​​specifically: in, Represents the mask matrix of the i-th word vector, M Dep i represents the syntactic dependency adjacency matrix of the i-th word vector, M Con i The distance of the i-th word vector constructs the semantic adjacency matrix, and softmax represents the normalized exponential function.

6. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

Images