An aspect-level sentiment classification method based on knowledge distillation lightweight

Abstract

The application discloses a kind of based on knowledge distillation light aspect-level sentiment classification method, its characteristics are the method includes: a) using bidirectional long short-term memory network modeling word features, word features are expressed into BERT model, the output of training sample is obtained;B) using parser obtains the dependency tree and its adjacency matrix of sentence, and incorporate sentiment knowledge and aspect word information;C) aspect word is root, reconstructs by pruning dependency tree, uses multi-head self-attention mechanism to extract context representation to the feature fusion of reconstructed tree, and incorporate the output of BERT model to carry out supervised training;D) using response-based knowledge distillation, feature-based knowledge distillation to optimize the effect of classification model.The application compared with prior art has few parameters, logic is clear, light, while incorporating external information such as grammar relationship, sentiment knowledge, which alleviates the influence on accuracy caused by limited network depth of light model to some extent.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of aspect-level sentiment classification, and in particular to a lightweight aspect-level sentiment classification method based on knowledge distillation. BACKGROUND

[0002] Aspect-level sentiment analysis is a challenging and practical task in the field of artificial intelligence, which is a fine-grained task in sentence analysis and belongs to a natural language processing task. Aspect-based sentiment classification (ABSC) is a subtask of aspect-level sentiment analysis and is a task for classifying the sentiment of a specific aspect word in a sentence. Aspect-based sentiment classification refers to: given a sentence and an aspect word in the sentence, the goal of aspect-based sentiment classification is to combine the context of the sentence and the aspect word information, and obtain the sentiment polarity of the aspect word by feature representation and classification of the context of the aspect word.

[0003] Early research on aspect-based sentiment classification mainly adopts traditional machine learning methods and simple neural network combinations. For example, dependency rules are designed manually to obtain words related to aspect words, and then the words are input into a machine learning SVM model to infer the sentiment polarity; for another example, a simple relation network-based method uses a recurrent neural network model to hierarchically encode word vector representations, and then uses a traditional conditional random field to predict the final answer. These models based on simple feature combinations often cannot well extract the relationship between the aspect word and the context, and the actual model performance is poor.

[0004] In recent years, researchers have also designed some novel algorithms to improve the performance of the aspect word sentiment classification task. For example, a network model based on an attention mechanism is used to weight the context of the aspect word to extract important information from the context; a method using a dependency tree-based graph network model uses dependency trees to add syntactic relationship information to facilitate the search for long-range dependencies; a method using a huge pre-trained language model such as Bert has learned knowledge from a large amount of corpus. The above methods have greatly improved the accuracy of sentiment prediction, but it is difficult to deploy in resource-constrained devices.

[0005] The aspect-level sentiment classification of the prior art has a large number of model parameters, high requirements for storage capacity and computing performance, long model running time, and is difficult to deploy in resource-constrained devices. SUMMARY

[0006] The application aims at providing a kind of aspect-level sentiment classification method based on knowledge distillation light weight aspect, which is provided to solve the problems of prior art, adopt the method of constructing light weight aspect-level sentiment classification model based on first-order predicate logic, external information such as syntax relationship and sentiment knowledge is integrated into the model, the effect of classification model is optimized by using knowledge distillation based on response and knowledge distillation based on feature, the dependency relationship between aspect words and its context is better mined, the accuracy of the model is greatly improved, the method has the advantages of few parameters, clear logic and light weight, at the same time, the integration of external information such as syntax relationship and sentiment knowledge alleviates the influence of the accuracy of light weight model caused by limited network depth to a certain extent, the method is simple, practical and effective, and has good application prospect.

[0007] The specific technical solution to achieve the object of the application is: a kind of aspect-level sentiment classification method based on knowledge distillation light weight aspect, which is characterized in that the method comprises the following steps:

[0008] Step 1: use bidirectional long short-term memory network modeling word features, input the word feature representation into the Bert model to get the output of the training sample by fine-tuning, as follows:

[0009] 1.1: use the pre-trained GloVe vector to initialize the word embedding representation.

[0010] 1.2: input the word embedding representation into the bidirectional long short-term memory network, and the hidden layer vector representation obtained is the word feature representation with context information.

[0011] 1.3: input the word embedding representation of the sentence into the BERT pre-training model in the form of "[CLS]sentence[SEP]aspect-terms[SEP]".

[0012] 1.4: fine-tune the BERT model, and the output result of the training sample is recorded as P1, then the first teacher uses P1 as the training target to supervise the relationship graph attention model and the light weight aspect-level sentiment classification model.

[0013] Step 2: use the parser to get the dependency tree and its adjacency matrix of the sentence, and integrate the sentiment knowledge and aspect word information on this basis, as follows:

[0014] 2.1: use the parser to get the dependency tree of the sentence, and get the 0-1 adjacency matrix according to the connectivity of the dependency tree.

[0015] 2.2: Integrate sentiment information into the adjacency matrix using SenticNet. If the item with a value of 1 in the adjacency matrix contains words corresponding to the nodes at both ends in SenticNet, add the sentiment score of the word in SenticNet to the corresponding item.

[0016] 2.3: Integrate aspect word information into the adjacency matrix. If one of the nodes at both ends of the item with a value of 1 in the adjacency matrix contains words in the aspect word, add 1 to the corresponding item.

[0017] Step 3: Reconstruct the aspect word as the root by pruning the dependency tree, use the multi-head self-attention mechanism to fuse the features of the reconstructed tree with edge information to extract the context representation, and integrate the output of the Bert model for supervised training, as follows:

[0018] 3.1: Reconstruct the initial dependency tree into a tree with aspect words as nodes, and take the aspect word node as the root of the tree (if the aspect word is composed of multiple words, obtain the feature representation of the aspect word as a whole through a bidirectional long short-term memory network), keep the nodes and their edge information directly connected to the aspect word in the dependency tree, and use the virtual relationship n to connect the nodes that are not directly connected to the aspect word (n represents the relative distance of the node to the aspect word), to avoid missing important information due to inaccuracy of the parser.

[0019] 3.2: Iteratively update the node to fuse the aspect word context features, use the multi-head self-attention mechanism for the nodes of the reconstructed tree, use the multi-head relationship mechanism to map the edge information between the nodes into vector representation, connect the vectors aggregated through the nodes with the vectors aggregated through the edge relationships, use an activation function (the parameters of the model as the second teacher in the activation function can be shared with the lightweight aspect-level sentiment classification model) to transform them into the original dimension, and obtain the last updated node representation.

[0020] 3.3: Iteratively update each node along the edges of the reconstructed tree, and finally take out the context feature representation of the root node to obtain the classification result through softmax. The output result P1 in the BERT model and the label are used together as the training target of the model for supervision.

[0021] Step 4: Construct a lightweight aspect-level sentiment classification model based on first-order predicate logic, and use response-based knowledge distillation and feature-based knowledge distillation to optimize the effect of the classification model, as follows:

[0022] 4.1: Use first-order predicate logic to construct a classification model, and the corresponding predicate calculation module is composed of a simple neural network module. The predicate is represented as follows:

[0023] represents the word, and a represents the aspect word.

[0024] S(x) represents a dependency relationship with emotional knowledge;

[0025] D(x) represents the relative distance from a word to an aspect word;

[0026] K(k,a) represents the rule k→a; R(x,a) represents the words related to a specific aspect of the word.

[0027] Then it exists

[0028] 4.2: This indicates the hidden layer output of the sentence through the corresponding word in the bidirectional long short-term memory network.

[0029] 4.3: Obtaining dependency relations containing affective knowledge in sentences, i.e. The following expression (a) can be used to express:

[0030]

[0031] in, is the adjacency matrix in step three, containing information such as the distilled dependency knowledge and sentiment common sense; W and b share parameters with the relational graph attention network, containing information about the edges in the dependency relations distilled from the relational graph attention network; σ is a nonlinear function.

[0032] 4.4: Assign different weights to contextual information based on its relative distance from the topic words, i.e. The closer a keyword is to an aspect, the greater its weight; the farther away a keyword is from an aspect, the smaller its weight.

[0033] 4.5: Extract words related to the aspect from the context, i.e. ).

[0034] 4.6: Incorporate previously obtained sentiment common sense, relative distance weights, and other constraints into the extracted contextual words related to aspect words, i.e. ).

[0035] 4.7: Finally, the data is input into a fully connected network to obtain the sentiment probability distribution.

[0036] Compared with existing technologies, this invention has the advantages of fewer parameters, clear logic, and lightweight design. It can better mine the dependency relationship between aspect words and their context, thus improving accuracy. At the same time, it incorporates external information such as grammatical relationships and sentiment knowledge, which to some extent alleviates the impact of the limited network depth of the lightweight model on accuracy. The method is simple, practical, and effective, and has a promising application prospect. Attached Figure Description

[0037] Figure 1 This is a flowchart of the present invention;

[0038] Figure 2 Flowchart for the embodiment. DETAILED DESCRIPTION

[0039] Reference Figure 1 The present application performs the knowledge distillation-based lightweight aspect-level sentiment classification in the following steps:

[0040] 1) Use bidirectional long short-term memory network to model word features, input word feature representation into the Bert model to get the output of the training sample by fine-tuning, which includes the following steps:

[0041] 1.1: Use the pre-trained GloVe vector to initialize the word embedding representation.

[0042] 1.2: Input the word embedding representation into the bidirectional long short-term memory network, and the hidden layer vector representation obtained is the word feature representation with context information.

[0043] 1.3: Input the word embedding representation of the sentence into the BERT pre-training model in the form of “[CLS] sentence [SEP] aspect-terms [SEP]”.

[0044] 1.4: Fine-tune the BERT model, and the output result of the training sample is denoted as P1. Next, the BERT model will be used as the first teacher to supervise the relationship graph attention model and the lightweight aspect-level sentiment classification model as the training target.

[0045] 2) Use the parser to get the dependency tree and its adjacency matrix of the sentence, and on this basis, integrate sentiment knowledge and aspect term information, which includes the following steps:

[0046] 2.1: Use the parser to get the dependency tree of the sentence, and get the 0-1 adjacency matrix according to the connected relationship of the dependency tree.

[0047] 2.2: Use SenticNet to integrate sentiment information into the adjacency matrix. If the item with 1 in the adjacency matrix contains the words corresponding to the nodes at both ends in SenticNet, add the sentiment score of the word in SenticNet to the corresponding item.

[0048] 2.3: Integrate aspect term information into the adjacency matrix. If one of the nodes at both ends of the item with 1 in the adjacency matrix corresponds to the word contained in the aspect term, add 1 to the corresponding item.

[0049] 3) Take the aspect word as the root by pruning the dependency tree, use multi-head self-attention mechanism to fuse the features of the reconstructed tree with edge information to extract the context representation, and integrate it into the output of the Bert model for supervised training, which includes the following steps:

[0050] 3.1: The initial dependency tree is reconstructed into a tree with aspect words as nodes, and the aspect word node is taken as the root of the tree (if the aspect word is composed of multiple words, the feature representation of the aspect word as a whole is obtained by bidirectional long short-term memory network), the nodes directly connected with the aspect word and their edge information are retained, and the nodes not directly connected with the aspect word are connected by a virtual relationship n (n represents the relative distance of the node to the aspect word), to avoid missing important information due to the inaccuracy of the parser.

[0051] 3.2: Iteratively update the node to fuse the aspect word context features, use multi-head self-attention mechanism for the nodes of the reconstructed tree, and use multi-head relationship mechanism for the edge information between the nodes to map it into a vector representation, connect the vector aggregated by the nodes with the vector aggregated by the edge relationship, and use an activation function (the parameters of this model as the second teacher in the activation function can be shared with the lightweight aspect-level sentiment classification model) to transform it into the original dimension, and get the last updated node representation.

[0052] 3.3: Update each node along the edge of the reconstructed tree iteratively, and finally take out the context feature representation of the root node to get the classification result by softmax, and the output result P1 in the BERT model and the label are used as the training target of the model for supervision.

[0053] 4) Construct a lightweight aspect-level sentiment classification model based on first-order predicate logic, and use response-based knowledge distillation and feature-based knowledge distillation to optimize the effect of the classification model, which includes the following steps:

[0054] 4.1: Use first-order predicate logic to construct a classification model, and the corresponding predicate calculation module is composed of a simple neural network module, and the predicate representation is as follows:

[0055] represents the word, and represents the aspect word;

[0056] S() represents the dependency relationship with sentiment knowledge;

[0057] D() represents the relative distance of the word to the aspect word;

[0058] K(k,) represents the rule k→; R(,) represents the word related to the specific aspect word;

[0059] Then there exists

[0060] 4.2: The sentence is represented by the hidden layer output of the corresponding word through the bidirectional long short-term memory network.

[0061] 4.3: Obtain the dependency relation with sentiment knowledge in the sentence, that is, ) is represented by (a) as follows:

[0062]

[0063] In the formula, is the adjacency matrix in step three, containing distilled dependency knowledge, sentiment common sense and the like; W and b share parameters with the relation graph attention network, and contain information distilled from the relation graph attention network; and sigma is a nonlinear function.

[0064] 4.4: Different weights are assigned to the context information according to the relative distance of the aspect word, that is, ), the closer the aspect word, the greater the weight, and the farther the aspect word, the smaller the weight.

[0065] 4.5: Extract the words related to the aspect word in the context, that is, ).

[0066] 4.6: Integrate the sentiment common sense, relative distance weight and the like obtained before into the extracted words related to the aspect word in the context, that is, ).

[0067] 4.7: Finally, input into the full connection network to obtain the sentiment probability distribution.

[0068] The application will be further described in conjunction with specific examples and the accompanying drawings.

[0069] Example 1

[0070] Referring to Figure 2 , the sentences and specific aspect words in the data set collected in the example are preprocessed and input into a bidirectional long short-term memory network to obtain word vector feature representation, and the word vector feature representation is input into a lightweight aspect-level sentiment classification model based on first-order predicate logic. The first teacher Bert model uses response-based distillation, and the output result and the label are used as training targets to supervise the training of the lightweight aspect-level sentiment classification model; the second teacher relation graph attention model uses feature-based distillation, and shares parameters with the lightweight aspect-level sentiment classification model and transmits the dependency adjacency matrix with sentiment information, so that the thin classification model has syntax dependency information and sentiment common sense. Finally, the output of the lightweight aspect-level sentiment classification model based on knowledge distillation is input into a full connection network to obtain the sentiment polarity prediction result for the specific aspect word.

[0071] The above merely describes the preferred embodiments of the present application, and modifications can be made within the scope of the present application defined by the claims.

Claims

1. A knowledge distillation-based lightweight aspect-level sentiment classification method, characterized in that, The method specifically comprises the following steps: Step 1: using a bidirectional long short-term memory network to model word features, inputting the word feature representation into a Bert model to obtain the output of the training sample through fine-tuning; Step 2: using a parser to obtain the dependency tree and its adjacency matrix of the sentence, and on this basis, integrating sentiment knowledge and aspect term information; Step 3: taking the aspect term as the root, reconstructing the dependency tree by pruning, using a multi-head self-attention mechanism to fuse the feature of the reconstructed tree with edge information to extract the context representation, and integrating the output of the Bert model for supervised training; Step 4: constructing a lightweight aspect-level sentiment classification model based on first-order predicate logic, and using response-based knowledge distillation and feature-based knowledge distillation to optimize the effect of the classification model; The specific process of step 4 is: 4.1: using first-order predicate logic to construct a lightweight aspect-level sentiment classification model, and the corresponding predicate calculation module is composed of a neural network module; 4.2: represents the hidden layer output of the corresponding word by the bidirectional long short-term memory network; 4.3: Acquiring dependency relations with sentiment knowledge in a sentence is represented by the following formula (a): （a）； wherein, is an adjacency matrix containing distilled dependency knowledge, emotional common sense information; shares parameters with the relational graph attention network, containing information distilled from the edges of the dependency relations in the relational graph attention network; is a nonlinear function; 4.4: The context information is given different weights according to the relative distance of the aspect words, i.e. The closer the aspect words, the greater the weight, and the farther the aspect words, the smaller the weight. 4.5: Extract the words in the context that are related to the aspect words, i.e. ); 4.6: Incorporate the previously obtained sentiment common sense, relative distance weight, etc. constraints into the extracted words related to the aspect words in the context, i.e. ); 4.7: finally inputting into a fully connected network to obtain a sentiment probability distribution.

2. The aspect-level sentiment classification method based on knowledge distillation lightening according to claim 1, characterized in that, The specific process of step 1 is: 1.1: using a pre-trained GloVe vector to initialize the word embedding representation; 1.2: inputting the word embedding representation into a bidirectional long short-term memory network to obtain the hidden layer vector representation, which is the word feature representation with context information; 1.3: inputting the word embedding representation of the sentence into the BERT pre-training model in the form of "[CLS]sentence[SEP]aspect-terms[SEP]"; 1.4: fine-tuning the BERT model, and the output result of the training sample is denoted as P1, then BERT will be used as the first teacher to use P1 as the training target to supervise the relationship graph attention model and the lightweight aspect-level sentiment classification model.

3. The aspect-level sentiment classification method based on knowledge distillation lightening according to claim 1, characterized in that, The specific process of step 2 is: 2.1: using a parser to obtain the dependency tree of the sentence, and obtaining a 0-1 adjacency matrix according to the connectivity of the dependency tree; 2.2: using SenticNet to integrate sentiment information into the adjacency matrix, if the nodes corresponding to the two ends of the item in the adjacency matrix contain the words in SenticNet, then add the sentiment score of the word in SenticNet to the corresponding item; 2.3: integrating aspect term information into the adjacency matrix, if one of the nodes corresponding to the two ends of the item in the adjacency matrix contains the words in the aspect term, then add 1 to the corresponding item.

4. The aspect-level sentiment classification method based on knowledge distillation lightening according to claim 1, characterized in that, The specific process of step 3 is: 3.1: reconstructing the obtained dependency tree into a tree with aspect terms as nodes, taking the aspect term node as the root of the tree, retaining the nodes and edge information directly connected with the aspect term in the dependency tree, and using a virtual relationship n to connect the nodes not directly connected with the aspect term, wherein n represents the relative distance of the node to the aspect term; 3.2: iteratively update the node pair aspect word context features for feature fusion, use multi-head self-attention mechanism for the nodes of the reconstructed tree, use multi-head relationship mechanism to map the edge information between the nodes into vector representation, connect the vectors aggregated through the nodes with the vectors aggregated through the edge relationships, use an activation function to transform them into the original dimension, and obtain the last updated node representation; 3.3: iteratively update each node along the edge of the reconstructed tree, and finally take out the context feature representation of the root node to obtain the classification result by using softmax, and the output result P1 in the BERT model and the label are jointly used as the training target of the lightweight aspect-level sentiment classification model for supervision.

5. The aspect-level sentiment classification method based on knowledge distillation lightening according to claim 1, characterized in that, The predicate is represented as follows: aspect words, aspect words; dependency relations with affective knowledge; relative distance of aspect words to aspect words; aspect words ; aspect words; then there exists ).

Images