A deep learning-based aspect-level sentiment classification method

Abstract

The present invention discloses an aspect-level sentiment classification method based on deep learning, which is specifically as follows: constructing an aspect-level sentiment classification data set; embedding aspect-level features in the data set into a semantic space; using an encoder based on adversarial BERT to perform feature encoding; an aspect-level sentence feature compression layer based on linear transformation performs linear transformation on the encoded result to meet the needs of capsule network calculation; an aspect-level sentence feature extraction layer based on capsule network further extracts features from the compressed matrix to capture the semantic relationship between aspect words and their corresponding contexts; a sentiment classification layer integrating a label regularization mechanism performs aspect-level sentiment classification on text according to the content learned by the model, and can be used for aspect-level sentiment classification of social media text. The present invention is based on fully learning the aspect-level features of sentences and combining an adversarial training mechanism to maximize the effect of the data set, and can still show good results in complex scenarios, with the characteristics of high accuracy and good robustness.

Description

Technical Field

[0001] The present invention relates to an aspect-level sentiment classification method based on deep learning. The method can be used for social media aspect-level sentiment classification and belongs to the technical field of Internet and natural language processing. Background Art

[0002] With the continuous development of the Internet, people have changed from information acquirers to information producers. Most netizens are accustomed to using social media (such as Weibo, Twitter, etc.) to express their views and attitudes on news, current events, social reality and other events. Massive amounts of data are generated on these social media platforms every day. Major social media platforms have gradually become sensors for the development of real-world events. Online public opinion plays an increasingly important role in reflecting public opinion and reflecting reality. At the same time, the Internet is also full of various bad remarks, but cyberspace is not a lawless place. Using online public opinion analysis technology to regulate online public opinion is conducive to enabling government departments to understand public opinion, properly handle social public opinion events, and create a healthy and harmonious network environment. As an important part of public opinion analysis technology, sentiment analysis technology has made great progress in recent years. At the same time, online comments show diverse characteristics, that is, a comment often targets multiple aspects. Therefore, traditional document-level sentiment analysis technology and sentence-level sentiment classification cannot classify each aspect of a sentence separately, and it is necessary to introduce aspect-level sentiment classification technology.

[0003] In recent years, many scholars have conducted in-depth research on aspect-level sentiment classification technology. The research methods can be roughly divided into three categories: methods based on sentiment dictionaries and rules, methods based on machine learning, and methods based on deep learning. The methods based on sentiment dictionaries and rules can improve the accuracy of sentence sentiment classification to a certain extent by manually constructing sentiment dictionaries and designing sentiment calculation rules. However, the setting of classification rules is often related to the specific field of data. Once the algorithm is migrated to other fields, the effect is difficult to guarantee, and the generalization ability of the algorithm is poor. In addition, the construction of sentiment dictionaries is a labor-intensive project that requires a lot of manual labor. The quality of the sentiment dictionary directly determines the quality of classification. This method is costly and has poor flexibility. Classification methods based on machine learning must rely on manually extracted text features before classification. The quality of text features directly determines the classification results. The process of manual feature extraction is very inefficient, and the model has poor migration ability, making it difficult to apply in real scenarios. Compared with methods based on sentiment dictionaries and rules and methods based on machine learning, methods based on deep learning have significantly improved classification effects. However, deep learning models require a large amount of training corpus, and the training process consumes a lot of computing power, with poor interpretability. Moreover, the effectiveness of algorithm classification based on pre-trained models depends on the quality of the downstream designed network. When a sentence contains multiple aspect words, it is often difficult for the downstream network to accurately match the correspondence between aspect words and context.

[0004] In view of the current severe situation of cyberspace security and the problem that aspect words and their corresponding contextual relationships cannot be accurately matched in existing aspect-level sentiment classification research, the present invention proposes an aspect-level sentiment classification method based on deep learning, which uses the BERT model combined with adversarial training mechanism as the encoding layer of aspect-level sentences, and introduces a capsule network downstream to capture aspect words and their corresponding contexts, thereby improving the robustness of the aspect-level sentiment classification model and the accuracy of classification. Summary of the invention

[0005] In view of the problems and shortcomings of the prior art, the present invention provides an aspect-level sentiment classification method based on deep learning, which can achieve the goal of aspect-level sentiment classification of review texts. Compared with the existing methods, the method can more effectively perform aspect-level sentiment classification on review texts, thereby improving the problem of insufficient accuracy of existing classification methods.

[0006] In order to achieve the above purpose, the technical solution of the present invention is as follows: a method for aspect-level sentiment classification based on deep learning, which covers the whole process of aspect-level sentiment classification, mainly including aspect word and sentence encoding, adversarial perturbation, feature transformation, feature extraction and sentiment classification, and can effectively classify the aspect-level sentiment of the comment text, thereby improving the accuracy of the task. The method mainly includes three steps, as follows:

[0007] Step 1: Construct an aspect-level sentiment classification dataset. First, we collected the SemEval 2014 Restaurant dataset and the Laptop dataset, and then collected the ACL14 Twitter public dataset. Finally, we divided these datasets into training and validation sets in an 8:2 ratio, which were used for training and validation of the aspect-level sentiment classification model, respectively.

[0008] Step 2, aspect-level sentiment classification model training. First, it is sent to the model's aspect-level sentence encoding layer based on adversarial BERT to obtain the encoding result of the aspect-level sentence; then, the encoding result of the aspect-level sentence will be sent to the aspect-level sentence feature compression layer based on linear transformation to perform sentence feature compression to meet the input needs of the capsule network; next, the capsule network further extracts features from the output of the sentence feature compression layer to identify the correspondence between aspect words and the context in the sentence; finally, the sentiment classifier that integrates the label regularization mechanism gives the sentiment prediction result. During the training phase, the model's loss function will compare the model's predicted value with the true value and calculate the loss value, and update the model parameters through back propagation to make the model parameters better. In addition, after each round of training, the data of the validation set will be sent to the model for verification;

[0009] Step 2: Aspect-level sentiment classification model training. The implementation process of this step is divided into 6 sub-steps:

[0010] Sub-step 2-1, use the aspect-level sentence encoding layer based on adversarial BERT to encode sentences and aspect words. The specific process is as follows:

[0011] Definition E w represents the result of the overall embedding of three types of information, E T 、E s and E P They represent word embedding, fragment embedding, and position embedding respectively. The information embedding in BERT can be expressed as:

[0012] E w =E T +E S +E P (1)

[0013] After that, the multi-layer Transformer encoder encodes the embedded result. The input of the first layer encoder is defined as H0, then H0 = E w , the encoding process can be expressed as:

[0014] H i = Transformer(H i-1 ),i∈[1,L] (2)

[0015] Among them, H i represents the result of Transformer encoding at the i-th layer, and L represents the total number of layers of the BERT-base encoder Transformer;

[0016] In sub-step 2-2, the adversarial training mechanism perturbs the embedded result and sends the perturbed result to the model for secondary training. The specific process is as follows:

[0017] Define X t+1 Represents the input at time t+1, which is the input X at time t t The gradient at time t is calculated, Π X+S If the perturbation exceeds a certain range, it will be remapped to the range S, where S is the perturbation constraint space and ε is the perturbation step size. In addition, g t Represents the gradient loss function at time t, and the calculation process can be expressed as:

[0018]

[0019] Sub-step 2-3, use the aspect-level sentence feature compression layer based on linear transformation to transform the vector obtained by the encoding layer. The transformation strategy includes linear transformation and Squash activation. The specific process is as follows:

[0020] H squ =Squash(W h H L +b h ) (4)

[0021]

[0022] Among them, H L represents the output hidden state vector of the BERT encoding layer, Represents the output vector after feature compression, col represents the matrix H squ The number of column vectors, W h is the linear change weight matrix, b h is the bias value, Squash is the compression function;

[0023] Sub-step 2-4 uses a capsule network-based specific aspect-level sentence feature extraction layer to match the relationship between the aspect words in the sentence and their corresponding contexts. The specific process is as follows:

[0024] First, the sentiment matrix is ​​initialized by averaging the embedding results of sentiment words Among them, c is the number of emotion capsules, d m is the dimension of the embedding vector, assuming that the sentiment capsule is Z = {z1,z2,...,z c}, the parameter update process can be expressed as:

[0025] z i =Squash(M i ) (6)

[0026]

[0027] Among them, w ij represents the routing weight between the main capsule and the sentiment capsule, W r is the weight matrix, Represents the vector obtained after compression by the sentence feature compression layer;

[0028] Sub-steps 2-5 use the sentiment classification layer that integrates the label regularization mechanism to perform aspect-level sentiment classification on the text. The specific classification calculation method is as follows:

[0029] The sentiment classification layer integrating the label regularization mechanism is mainly responsible for the final sentiment classification. In addition, in order to improve the classification effect, the label regularization mechanism is integrated in this layer. The final classification result is calculated by the following algorithm:

[0030]

[0031] Among them, λ is the learnable parameter of the model, y j Represents the result predicted by the model. In fact, neutral sentiment is an ambiguous sentiment that is often difficult for the model to identify. The Label Smoothing Regularization (LSR) method is introduced to alleviate this problem. During the training process, LSR prevents the model from predicting the sentiment tendency of a certain word in a sentence in a fully probabilistic way. This process can be expressed as:

[0032] q(k|x)=(1-ξ)q(k|x)+ξu(k) (9)

[0033]

[0034] Among them, ξ represents the regularization parameter, x represents the training examples of the input model, q(k|x) represents the true label distribution, u(k) is the prior distribution of the label, and c is the number of emotion types;

[0035] Substeps 2-6 use the loss function to calculate the training loss value. The specific process is as follows:

[0036] The loss function used in this model consists of three parts: cross entropy loss value, LSR loss value L lsr And the adversarial training loss value L adv , which can be expressed as:

[0037]

[0038] in, Represents the one-hot encoding result with the true label, y c Represents the model prediction result. The loss value of the label regularization mechanism is equal to the prior label distribution u(k) and the network prediction distribution p θ The KL divergence between can be expressed as:

[0039] L lsr =-D KL (u(k)||p θ ) (12)

[0040] The loss of adversarial training can be expressed as:

[0041]

[0042] Among them, Pb represents the disturbance generated on the input, ψ represents the size of the disturbance, logp(y c |x) represents the result y given the input x c The probability of occurrence, θ represents the parameters of the model.

[0043] Step 3: Classify the test set of the data set. For the text to be tested, first send it to the model obtained through the training process in step 2. The model encodes the aspect words and sentences, transforms features, extracts features, and finally uses the sentiment classifier to obtain the aspect-level sentiment classification results.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] The present invention fully learns the characteristics of the data set. When the data set is put into model training, the utilization rate of the data set is further improved by adding perturbations to the embedded information, thereby enhancing the robustness of the trained model and making it possible to use the model on small sample data. In addition, the capsule network is used to capture aspect words and their corresponding contextual relationships, further improving the accuracy of model classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0047] Figure 2 It is a general framework diagram of the method of an embodiment of the present invention;

[0048] Figure 3 Detailed graphs for sentence and aspect word embeddings;

[0049] Figure 4Comparison diagram of vector neurons and ordinary scalar neurons in capsule networks. DETAILED DESCRIPTION

[0050] In order to deepen the recognition and understanding of the present invention, the present invention is further illustrated below in conjunction with specific embodiments.

[0051] Example 1: See Figure 1-Figure 4 , a method for sentiment classification based on aspect level, the specific implementation steps of the present invention are as follows:

[0052] Step 1: Construct an aspect-level sentiment classification dataset. First, we collected the SemEval2014 Restaurant dataset and the Laptop dataset, and then collected the ACL14 Twitter public dataset. Finally, we divided these datasets into training and validation sets in an 8:2 ratio, which were used for training and validation of the aspect-level sentiment classification model, respectively.

[0053] Step 2: Aspect-level sentiment classification model training. The implementation process of this step is divided into 6 sub-steps:

[0054] Sub-step 2-1, use the aspect-level sentence encoding layer based on adversarial BERT to encode sentences and aspect words. The specific process is as follows:

[0055] Definition E w represents the result of the overall embedding of three types of information, E T 、E s and E P They represent word embedding, fragment embedding, and position embedding respectively. The information embedding in BERT can be expressed as:

[0056] E w =E T +E S +E P (1)

[0057] After that, the multi-layer Transformer encoder encodes the embedded result. The input of the first layer encoder is defined as H0, then H0 = E w , the encoding process can be expressed as:

[0058] H i = Transformer(H i-1 ),i∈[1,L] (2)

[0059] Among them, H i represents the result of Transformer encoding at the i-th layer, and L represents the total number of layers of the BERT-base encoder Transformer.

[0060] In sub-step 2-2, the adversarial training mechanism perturbs the embedded result and sends the perturbed result to the model for secondary training. The specific process is as follows:

[0061] Define X t+1 Represents the input at time t+1, which is the input X at time t t The gradient at time t is calculated, Π X+S If the perturbation exceeds a certain range, it will be remapped to the range S, where S is the perturbation constraint space and ε is the perturbation step size. t Represents the gradient loss function at time t, and the calculation process can be expressed as:

[0062]

[0063] Sub-step 2-3, use the aspect-level sentence feature compression layer based on linear transformation to transform the vector obtained by the encoding layer. The transformation strategy includes linear transformation and Squash activation. The specific process is as follows:

[0064] H squ =Squash(W h H L +b h ) (4)

[0065]

[0066] Among them, H L represents the output hidden state vector of the BERT encoding layer, Represents the output vector after feature compression, col represents the matrix H squ The number of column vectors, W h is the linear change weight matrix, b h is the bias value, and Squash is the compression function.

[0067] Sub-step 2-4 uses a capsule network-based specific aspect-level sentence feature extraction layer to match the relationship between the aspect words in the sentence and their corresponding contexts. The specific process is as follows:

[0068] First, the sentiment matrix is ​​initialized by averaging the embedding results of sentiment words Among them, c is the number of emotion capsules, d m is the dimension of the embedding vector. Assume that the sentiment capsule is Z = {z1,z2,...,z c}, the parameter update process can be expressed as:

[0069] z i =Squash(M i ) (6)

[0070]

[0071] Among them, w ij represents the routing weight between the main capsule and the sentiment capsule, W r is the weight matrix, Represents the vector obtained after compression by the sentence feature compression layer;

[0072] Sub-steps 2-5 use the sentiment classification layer that integrates the label regularization mechanism to perform aspect-level sentiment classification on the text. The specific classification calculation method is as follows:

[0073] The sentiment classification layer integrating the label regularization mechanism is mainly responsible for the final sentiment classification. In addition, in order to improve the classification effect, the label regularization mechanism is integrated in this layer. The final classification result is calculated by the following algorithm:

[0074]

[0075] Among them, λ is the learnable parameter of the model, y j Represents the result predicted by the model. In fact, neutral sentiment is an ambiguous sentiment that is often difficult for the model to identify. The Label Smoothing Regularization (LSR) method is introduced to alleviate this problem. During the training process, LSR prevents the model from predicting the sentiment tendency of a certain word in a sentence in a fully probabilistic way. This process can be expressed as:

[0076] q(k|x)=(1-ξ)q(k|x)+ξu(k) (9)

[0077]

[0078] Among them, ξ represents the regularization parameter, x represents the training examples of the input model, q(k|x) represents the true label distribution, u(k) is the prior distribution of the label, and c is the number of emotion types;

[0079] Substeps 2-6 use the loss function to calculate the training loss value. The specific process is as follows:

[0080] The loss function used in this model consists of three parts: cross entropy loss value, LSR loss value L lsr And the adversarial training loss value L adv , which can be expressed as:

[0081]

[0082] in, Represents the one-hot encoding result with the true label, y cRepresents the model prediction result. The loss value of the label regularization mechanism is equal to the prior label distribution u(k) and the network prediction distribution p θ The KL divergence between can be expressed as:

[0083] L lsr =-D KL (u(k)||p θ ) (12)

[0084] The loss of adversarial training can be expressed as:

[0085]

[0086] Among them, Pb represents the disturbance generated on the input, ψ represents the size of the disturbance, logp(y c |x) represents the result y given the input x c The probability of occurrence, θ represents the parameters of the model.

[0087] Step 3: For the text to be detected, first send it to the model obtained through the training process in step 2. The model encodes the aspect words and sentences, performs feature transformation, feature extraction and other steps, and finally uses the sentiment classifier to obtain the aspect-level sentiment classification result.

[0088] In summary, the present invention first uses an encoder based on adversarial BERT to encode aspect words and sentences, then the adversarial training mechanism adds perturbations to the embedded information, and the perturbed results are sent to the model for secondary training. Next, the encoding results are transformed through the feature transformation layer, and then the capsule network is used to extract aspect-level features. Finally, the sentiment classifier is used to perform aspect-level sentiment classification on the text.

[0089] It should be noted that the above embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent modifications of the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

Claims

1. A deep learning-based aspect-level sentiment classification method, characterized in that: The method comprises the following steps: Step 1: Construct an aspect-level sentiment classification dataset. Step 2: Aspect-level sentiment classification model training. Step 3, classify the test set of the data set; Among them, step 1, constructing the aspect-level sentiment classification dataset, specifically as follows, first collect the SemEval2014Restaurant dataset and Laptop dataset, then collect the ACL14 Twitter public dataset, and finally divide these datasets into training set and validation set according to 8:2, which are used for training and validation of aspect-level sentiment classification model respectively; Step 2 is as follows: extract the sentences and aspect words from the data in step 1. First, send them to the aspect-level sentence encoding layer of the model based on adversarial BERT to obtain the encoding results of the aspect-level sentences; then, the encoding results of the aspect-level sentences will be sent to the aspect-level sentence feature compression layer based on linear transformation to perform sentence feature compression to meet the input needs of the capsule network; next, the capsule network further extracts features from the output of the sentence feature compression layer to identify the corresponding relationship between the aspect words and the context in the sentence; finally, the sentiment classifier that is integrated with the label regularization mechanism gives the sentiment prediction results. In the training stage, the model's loss function compares the model's predicted value with the true value and calculates the loss value, and updates the model parameters through back propagation to make the model parameters better; Step 2: Aspect-level sentiment classification model training. The implementation process of this step is divided into 6 sub-steps: Sub-step 2-1, use the aspect-level sentence encoding layer based on adversarial BERT to encode sentences and aspect words. The specific process is as follows: Definition E w represents the result of the overall embedding of three types of information, E T 、E s and E P They represent word embedding, fragment embedding, and position embedding respectively. The information embedding in BERT is represented as: AND w =And T +E S +E P (1) After that, the multi-layer Transformer encoder encodes the embedded result. The input of the first layer encoder is defined as H0, then H0 = E w , the encoding process is expressed as: H i =Transformer(H i-1 ),i∈[1,L](2) Among them, H i represents the result of Transformer encoding at the i-th layer, and L represents the total number of layers of the BERT-base encoder Transformer; In sub-step 2-2, the adversarial training mechanism perturbs the embedded result and sends the perturbed result to the model for secondary training. The specific process is as follows: Define X t+1 Represents the input at time t+1, which is the input X at time t t The gradient at time t is calculated, Π X+S If the perturbation exceeds a certain range, it will be remapped to the range S, where S is the perturbation constraint space and ε is the perturbation step size. In addition, g t Represents the gradient loss function at time t, and the calculation process is expressed as: Sub-step 2-3, use the aspect-level sentence feature compression layer based on linear transformation to transform the vector obtained by the encoding layer. The transformation strategy includes linear transformation and Squash activation. The specific process is as follows: H squ =Squash(W h H L +b h )(4) Among them, H L represents the output hidden state vector of the BERT encoding layer, Represents the output vector after feature compression, col represents the matrix H squ The number of column vectors, W h is the linear change weight matrix, b h is the bias value, Squash is the compression function; Sub-step 2-4 uses a capsule network-based specific aspect-level sentence feature extraction layer to match the relationship between the aspect words in the sentence and their corresponding contexts. The specific process is as follows: First, the sentiment matrix is ​​initialized by averaging the embedding results of sentiment words Among them, c is the number of emotion capsules, d m is the dimension of the embedding vector, assuming that the sentiment capsule is Z = {z1,z2,...,z c }, the parameter update process can be expressed as: z i =Squash(M i )(6) Among them, w ij represents the routing weight between the main capsule and the sentiment capsule, W r is the weight matrix, Represents the vector obtained after compression by the sentence feature compression layer; Sub-steps 2-5 use the sentiment classification layer that integrates the label regularization mechanism to perform aspect-level sentiment classification on the text. The specific classification calculation method is as follows: The sentiment classification layer that integrates the label regularization mechanism is mainly responsible for the final sentiment classification. In order to improve the classification effect, the label regularization mechanism is integrated in this layer. The final classification result is calculated by the following algorithm: Among them, λ is the learnable parameter of the model, y j The result of the model prediction is represented. The Label Smoothing Regularization (LSR) method is introduced to alleviate this problem. During the training process, LSR prevents the model from predicting the sentiment tendency of a certain word in a sentence in a full probability manner. This process is expressed as: q(k|x)=(1-ξ)q(k|x)+ξu(k)(9) Among them, ξ represents the regularization parameter, x represents the training examples of the input model, q(k|x) represents the true label distribution, u(k) is the prior distribution of the label, and c is the number of emotion types; Substeps 2-6 use the loss function to calculate the training loss value. The specific process is as follows: The loss function used in this model consists of three parts: cross entropy loss value, LSR loss value L lsr And the adversarial training loss value L adv , which can be expressed as: in, Represents the one-hot encoding result with the true label, y c Represents the model prediction result. The loss value of the label regularization mechanism is equal to the prior label distribution u(k) and the network prediction distribution p θ The KL divergence between can be expressed as: L lsr =-D KL (u(k)||p θ )(12) The loss of adversarial training can be expressed as: Among them, Pb represents the disturbance generated on the input, ψ represents the size of the disturbance, logp(y c |x) represents the result y given the input x c The probability of occurrence, θ represents the parameters of the model.

Images