A Chinese aspect-level sentiment classification method based on pre-trained sentiment embedding

Abstract

The present invention discloses a Chinese aspect-level sentiment classification method based on pre-trained sentiment embedding, and proposes a deep learning model combining pre-trained sentiment embedding and semantic embedding. The model converts the aspect-level sentiment classification problem into a sentence pair classification problem, fine-tunes the BERT model at the character level of Chinese comments, trains the sentiment feature vector at the word level using an adversarial learning method and a two-layer bidirectional LSTM encoder module, and dynamically combines the obtained sentiment information with the semantic information of the last few layers of BERT through a linear weighted and multi-head self-attention mechanism, thereby obtaining the semantic and sentiment information of a specific aspect category, and performing accurate sentiment classification. This application combines the characteristics of Chinese patient comments, not only providing patients with rich decision-making information, but also enabling some doctors who pay more attention to performance and weaken medical ethics and abilities to improve their work, providing important guarantees for improving the quality of online medical care and the level of doctor services.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an aspect-level sentiment classification method for Chinese patient reviews based on pre-trained sentiment embedding. Background Art

[0002] With the development of online medical service platforms, more and more patients choose to seek medical treatment online and are willing to share their feelings and experiences during the medical treatment process on medical platforms. Previous studies have classified the emotions of patient comments to determine whether the patient is satisfied or dissatisfied with the doctor. However, a patient comment may involve multiple aspects, and this classification method is too general to distinguish the emotional polarity of multiple aspects in a Chinese patient comment. For example, given the comment sentence "Dr. Lu's attitude is very cold, but after he performed surgery on me, my symptoms were significantly relieved", the patient expressed opposite emotions about the doctor's "attitude" and "medical skills", and it is very necessary to correctly predict the emotional polarity corresponding to each aspect.

[0003] In this context, aspect-level sentiment classification aims to detect the categories of various aspects of the commented entity and predict the sentiment polarity expressed by each aspect. Currently, the methods used to deal with aspect-level sentiment classification problems mainly include statistical learning-based methods and deep learning-based methods. They usually use pre-trained word embeddings as input and design some neural structures with complex syntactic features or attention mechanisms, and have achieved good results on English review datasets. However, there are relatively few aspect-level sentiment classification methods for Chinese, and there is currently no research on related methods for patient reviews. Summary of the invention

[0004] The purpose of this application is to provide an aspect-level sentiment classification method based on pre-trained sentiment embedding, which can understand patients' evaluation of doctors during their medical consultation from multiple aspects, and serve as a reference for other patients in choosing doctors, thereby improving the quality and service level of online medical care.

[0005] To achieve the above objectives, this application proposes a Chinese aspect-level sentiment classification method based on pre-trained sentiment embedding, which specifically includes:

[0006] Step 1: Annotate aspect-level sentiment labels and corresponding sentiment polarity in the patient's review data on the doctor;

[0007] Step 2: Preprocess the annotated review data and divide it into training set, validation set, and test set;

[0008] Step 3: The preprocessed comment data is segmented and then sent to the pre-trained sentiment embedding module. The vector obtained by the pre-trained sentiment embedding module is then sent to the two-layer bidirectional LSTM model for pre-training. The sentiment embedding of the comment sentence is extracted at the Chinese word level based on the distributed feature representation learned by the fully connected layer;

[0009] Step 4: Expand the preprocessed comment data and use the BERT model to map each Chinese character in the sentence to a vector space to extract the semantic embedding of the comment sentence at the Chinese character level;

[0010] Step 5: Integrate the sentiment embedding and the semantic embedding, use linear weighting and multi-head self-attention mechanism to obtain the final sentiment and semantic feature representation, and send the representation to the softmax layer for training classification to obtain the aspect categories involved in the comment sentence and the corresponding sentiment polarity.

[0011] Furthermore, the aspect-level sentiment labels include attitude, medical ethics and medical skills; the sentiment polarity includes positive, negative and no such aspect.

[0012] Furthermore, the annotated comment data is preprocessed, specifically, removing empty text, repeated text, and stop words, and then removing useless characters in the text through regular expressions.

[0013] Furthermore, the specific implementation of step 3 is as follows:

[0014] Step 31: Use Jieba word segmentation in Python to process the preprocessed comment data;

[0015] Step 32: The comment data after word segmentation is sent to the pre-trained sentiment embedding module to obtain the sentiment word vector matrix of each comment;

[0016] Step 33: Send the sentiment word vector matrix to a two-layer bidirectional LSTM (Long Short Term Memory) model for pre-training, and use a gating mechanism to capture the dependencies between word sequences, so that the model can learn which information needs to be forgotten and which information needs to be retained, and finally obtain a fixed-size feature vector S that reflects the sentiment information of the sentence;

[0017] Furthermore, the pre-trained sentiment embedding module uses an adversarial learning method ASWE (Adversarial Learning of Sentiment Word Representations) based on a multi-head attention mechanism to learn the sentiment characteristics of each word in a sentence, and the dimension of the word vector is set to 300.

[0018] Furthermore, the two-layer bidirectional LSTM model is a bidirectional recurrent neural network including a forward network and a backward network. Given a sentence T containing n words, T = (t1, t2, ..., t n ), the forward network passes the k-th input word in the output vector of the last layer of the network To predict the next word t k+1 , the backward network passes the output vector of the kth input word in the last layer of the network To predict the previous word t k-1 , the optimization objective function is:

[0019]

[0020] in and Represent the parameters of the forward and backward networks respectively, and the parameters in these two directions are independent.

[0021] Furthermore, the specific implementation of step 4 is as follows:

[0022] Step 41: The preprocessed comment data is expanded by constructing auxiliary sentences related to aspect categories and sentiment polarity (e.g., “attitude-positive”); the expanded comment data set includes the original comment X and the auxiliary sentences A, the label “0” indicates that the sentiment polarity for a certain aspect does not appear in the comment, and the label “1” indicates that the sentiment polarity for a certain aspect appears in the comment, and the final input sequence of the BERT model is represented as P=[CLS,X,SEP,A,SEP];

[0023] Step 42: Represent the input sequence of the BERT model as the sum of token embedding, segment embedding and position embedding, and send it to L consecutive transformer encoding blocks;

[0024] Step 43: After training, the hidden state of each encoding layer in the BERT model is:

[0025]

[0026] In order to dynamically fuse the spatial and hierarchical information contained in each BERT encoding layer, the hidden state sequence of CLS tokens is used Represents the semantic features of sentence pairs, rather than just As a global aggregate representation of a sentence.

[0027] Furthermore, the specific implementation of step 5 is as follows:

[0028] Step 51: Embed the sentiment S and the semantic embedding h CLS Connect them to get the sentiment and semantic feature vector representation with dimension d:

[0029]

[0030] Step 52: Use the fully connected layer to connect the d-dimensional features Q corresponding to each layer in step 51 d Mapped to the one-dimensional feature space, and the obtained one-dimensional vector Q 1 Send it to the softmax layer to get the weights corresponding to the features of each layer, and then dynamically combine all the features through linear weighting to get the initial sentiment and semantic feature representation O':

[0031] O′=Q d ·softmax(W q Q 1 +b q ) (4)

[0032] Where W q and b q They are the trainable parameter matrix and vector in the fully connected layer respectively;

[0033] Step 53: In order to allow the model to capture the correlation between different parts of the entire input, the multi-head self-attention mechanism is used to obtain the initial feature representation O', and the final sentiment and semantic feature representation O is obtained. The specific steps are as follows:

[0034] A1. First, define each input vector under a specific attention head h and Correlation between:

[0035]

[0036]

[0037] where ψ (h) (x,y) represents the similarity measure function of vectors x and y, and is the transformation matrix that maps the original embedding space to the new space;

[0038] A2. Through the correlation coefficient Update vector e m The expression is:

[0039]

[0040] in is the transformation matrix that maps the original embedding space to the new space, Represents the new vector learned by the multi-head self-attention mechanism;

[0041] Step 54: The Softmax function converts the feature representation O obtained in step 53 into a conditional probability distribution on the label set y = {0, 1}, and outputs the label with the maximum probability value as the final prediction result:

[0042]

[0043] Where W o and b o are trainable parameter matrices and vectors respectively;

[0044] Step 55: By minimizing the true value y and the predicted value in all data samples The cross entropy loss between them is used for training. The cross entropy loss function L is:

[0045]

[0046] Among them, i and j represent the indexes of data samples and label categories respectively; the model with the best classification effect is selected and loaded, and the Chinese patient review data is sent to the model to predict the aspect categories mentioned in the review sentences and their corresponding sentiment polarities.

[0047] By adopting the above technical scheme, the present invention can achieve the following technical effects: the present invention adopts a deep learning model that integrates pre-trained sentiment embedding and semantic embedding to perform fine-grained analysis of Chinese patient comments from multiple different aspects, which can determine the patient's emotional tendencies in some specific aspects of the doctor's evaluation, so that the public can know what is good and what is not good about the doctor, so that some doctors who pay more attention to performance and gradually weaken their morality and ability can improve their work, and also serve as a reference for other patients in choosing doctors, thereby improving the quality of online medical care and the level of doctor services. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Flowchart of an aspect-level sentiment classification method based on pre-trained sentiment embeddings;

[0049] Figure 2 This is the structure diagram of the pre-trained sentiment embedding module. DETAILED DESCRIPTION

[0050] The embodiments of the present invention are implemented on the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given, but the protection scope of the present invention is not limited to the following embodiments.

[0051] Example 1

[0052] The present invention extracts the comments of patients on doctors from the Good Doctor Online website, and constructs a Chinese patient comment data set based on the screening; furthermore, a deep learning model based on pre-trained sentiment embedding is provided to process the sentiment classification task at the aspect level, and two embedding methods are proposed according to the characteristics of Chinese comments: 1) at the level of Chinese words, the sentiment word embedding in the comment sentence is trained using an adversarial method, and the embedding vector is sent to a two-layer bidirectional LSTM encoder for pre-training to obtain the sentiment embedding of the comment sentence; 2) at the level of Chinese characters, the comment data set is expanded, and the semantic embedding of the comment sentence is obtained using the BERT model. Then, the extracted sentiment features and the knowledge of the semantic embedding layer are integrated using a linear weighting and multi-head self-attention mechanism, and then the aspect category involved in the comment and its corresponding sentiment polarity are predicted.

[0053] During the training process, the model uses the Adam optimizer, sets the batch size to 32, and the learning rate to 1e-5. The number of BERT network layers k is experimented from 1 to 12, and finally k=4 is selected as the default value of the number of layers. This model effectively solves the problem of sentiment classification based on aspect categories in Chinese patient reviews, and its accuracy is significantly improved compared with other mainstream methods.

[0054] The present invention is described in detail below in conjunction with embodiments and drawings so that those skilled in the art can implement the invention according to the description.

[0055] This example uses Pycharm as the development platform and Python as the development language, and is conducted on a Chinese patient review dataset. The specific process is as follows:

[0056] Step 1: Annotate aspect-level sentiment labels and corresponding sentiment polarity in the patient's review data on the doctor;

[0057] Specifically, we annotated aspect-level sentiment labels in 108,331 comments about doctors on the Haodaifu Online website. We predefined the aspect categories of the comment dataset, including the doctor's attitude, medical ethics, and medical skills. We also manually annotated the aspect labels (attitude, medical ethics, medical skills) and corresponding sentiment polarity (positive, negative, no such aspect) of some comment sentences.

[0058] Step 2: Preprocess the annotated review data and divide it into training set, validation set, and test set;

[0059] Specifically, the annotated comment data is preprocessed by removing empty text, repeated text, stop words, and then using regular expressions to remove useless characters in the text. The preprocessed comment data is divided into a ratio of training set: validation set: test set = 8:1:1.

[0060] Step 3: The pre-processed comment data is segmented and then sent to the pre-trained sentiment embedding module. The vector obtained by the pre-trained sentiment embedding module is then sent to the two-layer bidirectional LSTM model for pre-training. The sentiment embedding of the comment sentence is extracted at the Chinese word level based on the distributed feature representation learned by the fully connected layer; specifically:

[0061] Step 31: Segment the preprocessed comment data and remove punctuation marks and stop words;

[0062] Step 32: Use an adversarial learning method based on a multi-head attention mechanism to learn the sentiment characteristics of each word in the sentence and obtain a sentiment word vector matrix;

[0063] Step 33: sending the sentiment word vector matrix into a two-layer bidirectional LSTM model for pre-training, and obtaining a feature vector reflecting the sentiment information of the sentence through a fully connected layer;

[0064] Step 4: Expand the preprocessed comment data and use the BERT model to map each Chinese character in the sentence to the vector space to extract the semantic embedding of the comment sentence at the Chinese character level; specifically:

[0065] Step 41: construct auxiliary sentences related to aspect categories and sentiment polarity (e.g., “attitude-positive”);

[0066] Step 42: Send the auxiliary sentences and original comments as input sequences to the BERT model for training;

[0067] Step 43: The model outputs the hidden state representations of different network layers, which are used as different semantic feature representations of the sentence pairs;

[0068] Step 5: Integrate the sentiment embedding and the semantic embedding, use linear weighting and multi-head self-attention mechanism to obtain the final sentiment and semantic feature representation, and send the representation to the softmax layer for training classification to obtain the aspect categories and corresponding sentiment polarity involved in the comment sentence. Specifically:

[0069] Step 51: Connect the sentiment embedding and the semantic embedding to obtain sentiment and semantic feature vector representations with a dimension of d;

[0070] Step 52: Map the d-dimensional features corresponding to each layer to a one-dimensional feature space, and obtain the initial sentiment and semantic feature representation through linear weighting;

[0071] Step 53: Send the initial sentiment and semantic feature representations into the multi-head self-attention mechanism to obtain the final sentiment and semantic feature representations;

[0072] Step 54: The feature representation obtained in step 53 is sent to the fully connected layer for classification to predict the aspect category mentioned in the comment sentence and its corresponding sentiment polarity.

[0073] According to the above steps, the present invention conducts an ablation experiment to further study how the model benefits from each component, as shown in Table 1, where w / o indicates that a component is removed from the model. The experiment proves that the deep learning model based on pre-trained sentiment embedding in the present invention can significantly improve the accuracy of aspect-level sentiment classification tasks.

[0074] Table 1 Ablation experiment results of the model on the patient review dataset

[0075]

[0076] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A Chinese aspect-level sentiment classification method based on pre-trained sentiment embedding, characterized in that Specifically include: Step 1: Annotate aspect-level sentiment labels and corresponding sentiment polarities in the patient's review data on the doctor; Step 2: Preprocess the annotated review data and divide it into training set, validation set, and test set; Step 3: The preprocessed comment data is segmented and then sent to the pre-trained sentiment embedding module. The vector obtained by the pre-trained sentiment embedding module is then sent to the two-layer bidirectional LSTM model for pre-training. The sentiment embedding of the comment sentence is extracted at the Chinese word level based on the distributed feature representation learned by the fully connected layer; Step 4: Expand the preprocessed comment data and use the BERT model to map each Chinese character in the sentence to a vector space to extract the semantic embedding of the comment sentence at the Chinese character level; Step 5: Integrate the sentiment embedding and the semantic embedding, use linear weighting and multi-head self-attention mechanism to obtain the final sentiment and semantic feature representation, and send the representation to the softmax layer for training classification to obtain the aspect categories involved in the comment sentence and the corresponding sentiment polarity.

2. According to claim 1, a Chinese aspect-level sentiment classification method based on pre-trained sentiment embedding is characterized in that: The aspect-level sentiment labels include attitude, medical ethics and medical skills; the sentiment polarity includes positive, negative and no such aspect.

3. According to claim 1, a Chinese aspect-level sentiment classification method based on pre-trained sentiment embedding is characterized in that: The annotated comment data is preprocessed by removing empty text, repeated text, and stop words, and then using regular expressions to remove useless characters in the text.

4. According to claim 1, a Chinese aspect-level sentiment classification method based on pre-trained sentiment embedding is characterized in that: The specific implementation of step 3 is: Step 31: Use Jieba word segmentation in Python to process the preprocessed comment data; Step 32: The comment data after word segmentation is sent to the pre-trained sentiment embedding module to obtain the sentiment word vector matrix of each comment; Step 33: Send the sentiment word vector matrix to a two-layer bidirectional LSTM model for pre-training, use a gating mechanism to capture the front-end dependency of the word sequence, and obtain a fixed-size feature vector S that reflects the sentiment information of the sentence.

5. According to claim 4, a Chinese aspect-level sentiment classification method based on pre-trained sentiment embedding is characterized in that: The pre-trained sentiment embedding module uses an adversarial learning method based on a multi-head attention mechanism, ASWE, to learn the sentiment features of each word in a sentence.

6. A Chinese aspect-level sentiment classification method based on pre-trained sentiment embedding according to claim 1 or 4, characterized in that: The two-layer bidirectional LSTM model is a bidirectional recurrent neural network including a forward network and a backward network. Given a sentence T containing n words, T = (t1, t2, ..., t n ), the forward network passes the k-th input word in the output vector of the last layer of the network To predict the next word t k+1 , the backward network passes the output vector of the kth input word in the last layer of the network To predict the previous word t k-1 , the optimization objective function is: in and Represent the parameters of the forward and backward networks respectively, and the parameters in these two directions are independent.

7. According to claim 1, a Chinese aspect-level sentiment classification method based on pre-trained sentiment embedding is characterized in that: The specific implementation of step 4 is as follows: Step 41: The preprocessed comment data is expanded by constructing auxiliary sentences related to aspect categories and sentiment polarity; the expanded comment data set includes the original comment X and the auxiliary sentences A, the label "0" indicates that the sentiment polarity for a certain aspect does not appear in the comment, and the label "1" indicates that the sentiment polarity for a certain aspect appears in the comment, and the final input sequence of the BERT model is expressed as P = [CLS, X, SEP, A, SEP]; Step 42: Represent the input sequence of the BERT model as the sum of token embedding, segment embedding and position embedding, and send it to L consecutive transformer encoding blocks; Step 43: After training, the hidden state of each encoding layer in the BERT model is: In order to dynamically fuse the spatial and hierarchical information contained in each BERT encoding layer, the hidden state sequence of CLS tokens is used Represents the semantic features of a sentence pair.

8. According to claim 1, a Chinese aspect-level sentiment classification method based on pre-trained sentiment embedding is characterized in that: The specific implementation of step 5 is as follows: Step 51: Embed the sentiment S and the semantic embedding h CLS Connect them to get the sentiment and semantic feature vector representation with dimension d: Step 52: Use the fully connected layer to connect the d-dimensional features Q corresponding to each layer in step 51 d Mapped to the one-dimensional feature space, and the obtained one-dimensional vector Q 1 Send it to the softmax layer to get the weights corresponding to the features of each layer, and then dynamically combine all the features through linear weighting to get the initial sentiment and semantic feature representation O': O′=Q d ·softmax(W q Q 1 +b q ) (4) Where W q and b q They are the trainable parameter matrix and vector in the fully connected layer respectively; Step 53: Use the multi-head self-attention mechanism to obtain the initial feature representation O', and obtain the final sentiment and semantic feature representation O. The specific steps are as follows: A1. First, define each input vector under a specific attention head h and Correlation between: where ψ (h) (x,y) represents the similarity measure function of vectors x and y, and is the transformation matrix that maps the original embedding space to the new space; A2. Through the correlation coefficient Update vector e m The expression is: in is the transformation matrix that maps the original embedding space to the new space, Represents the new vector learned by the multi-head self-attention mechanism; Step 54: The Softmax function converts the feature representation O obtained in step 53 into a conditional probability distribution on the label set y = {0, 1}, and outputs the label with the maximum probability value as the final prediction result: Where W o and b o are trainable parameter matrices and vectors respectively; Step 55: By minimizing the true value y and the predicted value in all data samples The cross entropy loss between them is used for training. The cross entropy loss function L is: Among them, i and j represent the indexes of data samples and label categories respectively; the model with the best classification effect is selected and loaded, and the Chinese patient review data is sent to the model to predict the aspect categories mentioned in the review sentences and their corresponding sentiment polarities.

Images