Target-centered semi-supervised fine-grained sentiment analysis method

Abstract

The invention discloses a target-centered semi-supervised fine-grained sentiment analysis method, and belongs to the technical field of deep learning and natural language processing. The method comprises the following steps of: constructing a data set based on social media text data with an emotion label, encoding an input text by using a fusion word vector and a position vector, and enriching context semantic information of each word in the data set; constructing a PWT encoder, a gating classifier and a bidirectional PWT decoder; constructing an emotion analysis network model; performing semi-supervised training on the sentiment analysis network model by using the labeled data to obtain a trained model; and using the trained sentiment analysis network model to carry out sentiment score calculation on the unlabeled data, and outputting a sentiment score. According to the method, automatic public opinion acquisition in a big data era can be researched, and sentiment analysis is carried out from massive texts.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and natural language processing, and in particular to a target-centric semi-supervised fine-grained sentiment analysis method. Background Art

[0002] In recent years, the research on automatic public opinion acquisition has been highly valued by the academic community. Among them, sentiment analysis from massive texts to obtain the public opinion trends that people are interested in is a key technology that needs to be solved in the big data era. Fine-grained sentiment analysis is an important part of text information classification. Unlike coarse-grained sentiment analysis, which can only obtain the sentiment tendency of the entire text, fine-grained sentiment analysis can analyze the public opinion trend by analyzing the sentiment tendency of each person and thing in the text. Fine-grained sentiment analysis has more practical value and possibility of implementation.

[0003] Fine-grained sentiment analysis focuses on the target words in the text and is used to automatically analyze the emotional tendency of the target words. Existing fine-grained sentiment analysis methods are mainly fully supervised methods based on neural networks, among which deep learning methods based on recurrent neural networks are the most common. Among the fine-grained sentiment analysis methods based on deep learning, the method of combining the long short-term memory network (LSTM) with the attention mechanism is more intuitive and has achieved a certain degree of accuracy. This method uses LSTM to extract contextual features, and the attention mechanism is used to locate the context related to the target word. However, this type of method still has some shortcomings. For example, in the feature embedding layer, the traditional word vector that can only express the meaning of a single word is not used to determine the semantics of the word in a specific context. However, the pre-trained language model ELMo is used for feature representation, which can enrich the contextual semantic information of the word. It does not take into account that the fully supervised method requires all training data to be labeled, which is inefficient and costly, while the semi-supervised method can make full use of unlabeled data to achieve fine-grained sentiment analysis capabilities similar to those of the fully supervised method. For the current semi-supervised training method, the fine-grained analysis task is not modified, so that the semantic information and position information of the target word cannot be fully utilized, and the traditional LSTM-based method cannot be operated in parallel and has low efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a target-centric semi-supervised fine-grained sentiment analysis method to solve the problems existing in the above-mentioned prior art. The present invention has the characteristics of high efficiency, high accuracy, no need for manpower, wide applicability, and is not affected by environmental factors.

[0005] The technical solution adopted by the present invention is:

[0006] A target-centric semi-supervised fine-grained sentiment analysis method includes the following steps:

[0007] Step 1: Build a dataset based on social media text data with sentiment labels, and use fused word vectors and position vectors to encode the input text to enrich the contextual semantic information of each word in the dataset;

[0008] Step 2: construct a PWT encoder to encode the input text and obtain a sentence vector representation containing position association information; construct a gated classifier to implement fine-grained sentiment analysis and obtain a sentiment polarity score; construct a bidirectional PWT decoder to decode the mixed representation of the sentence vector representation and the sentiment polarity score;

[0009] Step 3, the PWT encoder, the gated classifier, and the bidirectional PWT decoder are connected in series in sequence, and the output of the PWT encoder is also transmitted to the bidirectional PWT decoder, so as to construct a sentiment analysis network model; the sentiment analysis network model is semi-supervised trained using labeled data to obtain a trained model;

[0010] Step 4: Use the trained sentiment analysis network model to calculate the sentiment score of the unlabeled data and output the sentiment score.

[0011] Furthermore, the specific method of step 1 is:

[0012] Collect text data containing sentiment labels from social media platforms. Sentiment labels include positive, negative, and neutral categories.

[0013] Clean and preprocess the collected text data to remove noise data and irrelevant information;

[0014] Use the pre-trained word vector model to convert each word into a corresponding word vector to capture the semantic information of the word and fuse the word vector with the position vector;

[0015] The ELMo pre-trained language model and the Glove word vector are used, and the fully connected layer is used for fusion to form an input vector containing contextual semantic information;

[0016] The dataset is divided into various training sets, validation sets, and test sets for model training and evaluation.

[0017] Further, in step 2, the PWT encoder includes a two-layer Transformer encoder;

[0018] The gated classifier adopts the ReLU activation function. The gated classifier uses a multi-level attention mechanism to calculate the attention of the target word from the full text, the previous text of the target word, and the following text of the target word, respectively, to obtain the attention weight and the respective feature representation, thereby enriching the contextual semantic information of the word; after obtaining the weighted context representation, the information irrelevant to the target word is further filtered through the GTRU gating mechanism;

[0019] The bidirectional PWT decoder includes two reverse PWT structures, starting from the target word and decoding its context simultaneously, so that the decoder continuously focuses on the target word information.

[0020] Furthermore, the specific method of semi-supervised training in step 3 is:

[0021] First, the gated classifier and the bidirectional PWT decoder are trained using labeled data to obtain a partially trained gated classifier model. Then, the gated classifier is used to calculate the sentiment score of the unlabeled data. At this time, the gated classifier model parameters are fixed. Finally, the generated sentiment score and the feature vector generated by the PWT encoder are input into the bidirectional PWT decoder to optimize the entire encoding and decoding structure to generate better feature vectors, thereby obtaining a more accurate sentiment score.

[0022] The present invention has the following beneficial effects:

[0023] 1. The present invention is simple, efficient, easy to implement, and can improve training speed and feature extraction capabilities.

[0024] 2. The present invention solves the problem of low utilization rate of unlabeled data in the prior art.

[0025] 3. The present invention can study the automatic acquisition of public opinion in the big data era and perform sentiment analysis from massive texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 is a schematic diagram of the principle of a target-centric semi-supervised fine-grained sentiment analysis method in an embodiment of the present invention;

[0028] Figure 2 Schematic diagram of the principle of a PWT encoder in an embodiment of the present invention;

[0029] Figure 3Schematic diagram of the principle of a bidirectional PWT decoder in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] A target-centric semi-supervised fine-grained sentiment analysis method, such as Figure 1 As shown in the figure, it includes feature vector generation, feature vector encoding, fine-grained sentiment analysis and bidirectional decoding. It first uses fused word vectors and position vectors to encode the input text to enrich the contextual semantics of each word; then builds a PWT encoder to encode the input to obtain a sentence vector representation containing position association information; builds a gated classifier to implement fine-grained sentiment analysis and obtain a sentiment polarity score, in which the parallel attention mechanism can calculate the contribution of each word to the target from multiple angles, and the GTRU gating mechanism realizes the control of the flow of contextual information; then, uses a bidirectional decoder to decode the mixed representation of the sentence vector representation and the sentiment polarity score, and compares it with the original sentence to achieve semi-supervised training; finally, uses the trained model to perform sentiment analysis.

[0033] The specific steps of this method are as follows:

[0034] (1) By collecting a large amount of social media text data with sentiment labels (from annotated datasets or obtained through manual annotation), a neural network training dataset that can be used for semi-supervised methods is formed. The input text is encoded using a fused multi-dimensional word vector and an absolute position vector to obtain the contextual information of the word and the position information of the target word, and a feature vector is generated; the ELMo pre-trained language model and the Glove word vector are fused using a fully connected layer to achieve a fused multi-dimensional word vector.

[0035] Specifically, in the word embedding layer, in order to obtain the semantics of each word in the context, for each input text sequence {x0, x1, ..., x t-1 , x t , x t+1 , ..., x n-1 , x n}, based on the traditional word embedding w, add the pre-trained word vector e of ELMo. Different from the traditional N-gram-based word embedding model, the word vector representation in the ELMo pre-training model will be dynamically adjusted according to the context semantics, so the introduction of ELMo word vector is necessary. Compared with BERT, ELMo requires much less computing resources, so the word vector representation s fused by the fully connected layer (FC) is:

[0036] s=FC(w+e) (1)

[0037] (2) Construct a PWT (Position Wise Transformer) encoder to encode the feature vector and obtain a sentence vector representation containing position-related information. The PWT encoder uses a two-layer Transformer encoder as the basic architecture to improve the training speed and feature extraction capability. The multi-head self-attention layer it contains introduces trainable relative position parameters.

[0038] like Figure 2 As shown, the standard input of Transformer includes word vector e xk and position vector p xk For the target-oriented sentiment classification task, in order to better distinguish the target and the context, segmentation tags are used to distinguish the target words and the context at the time of input. The target words are marked with 1, and the target word context is marked with 0, and converted into a vector s xi As input. At the same time, the relative position information of the target entity can also help the model better focus on the relationship between the target entity and the adjacent context. Therefore, a Transformer-based PWT encoder-decoder is proposed. In specific implementation, a trainable parameter representing the relative position is added when the Transformer model calculates the self-attention score, and the parameter is shared between each attention head. The process is formally expressed as:

[0039]

[0040] Among them, A i represents the self-attention score at position i, h i and h j For the word pair (x i , x j ) from the previous multi-head self-attention layer, W Q , W K and W V is the trainable parameter of the model, d z is the output dimension of the self-attention layer. Compared with the standard Transformer self-attention model, PWT adds two parameters and Used to represent the input x i , x j The boundary information of , and in this method, it is used to obtain the relative distance between the target entity and each word, that is, i only represents the target word, when i is a non-target word, and Set to zero. and The specific formula is as follows:

[0041]

[0042] dis(x,k)=max(-k,min(k,x)) (6)

[0043] A gated classifier is constructed to perform fine-grained sentiment analysis on the feature vector to obtain the sentiment polarity score. The multi-level attention mechanism of the gated classifier calculates the attention of the target word from the full text, the previous text of the target word, and the following text of the target word, and obtains the attention weight and the respective feature representation, thereby enriching the contextual semantic information of the word. The GTRU gating mechanism of the gated classifier uses the attention mechanism to calculate the contribution of each word to the target. After obtaining the weighted context representation, it realizes the control of the flow of contextual information, and further filters the information unrelated to the target word through the GTRU gating mechanism, thereby ignoring the interference of non-target word related information.

[0044] The gated classifier uses the ReLU activation function instead of the traditional Sigmoid function. Its negative interval zeroing feature can filter out irrelevant information and only retain the emotional information related to the specific target.

[0045] Specifically, the gated classifier first uses the attention mechanism to calculate the contribution of each word to the target. For the input word vectors s0, s1, ..., s n , use BILSTM network to extract semantics and get vectors h0, h1, ..., h n :

[0046] [h0,h1,...,h n ] = BILSTM([s0, s1, ..., s n ]) (7)

[0047] The target word is represented as h in the latent vector t When the target word is a short text composed of multiple words, the target word vector is represented by the text representation [h t0 ,h t1 , ..., h tm ]. In the attention layer, the full text, the target word preceding text, and the target word following text respectively calculate the attention of the target word, and obtain the attention weight α iand their respective feature representations s, s l ,s r , the specific formula is as follows:

[0048]

[0049]

[0050] The following takes the calculation of s as an example to illustrate the calculation method of the attention mechanism:

[0051]

[0052] The weight β i It is calculated from the target word representation and the context word representation:

[0053] β i =U T tanh(W1·[h i :h t ]+b1) (12)

[0054] After obtaining the weighted representation, the GTRU gating mechanism is used to control the flow of information. Sentiment information can be reflected by the context of the target word, the context of the target word, or the entire sentence. Therefore, three gates z, z l , z r To control the target word and the corresponding context:

[0055] z=ReLU(W1s+U1h t +b1) (13)

[0056] z l =ReLU(W2s l +U2h t +b2) (14)

[0057] z r =ReLU(W3s r +U3h t +b3) (15)

[0058] Among them, s, s l ,s r The linear interpolation formula between is:

[0059]

[0060] The final predicted sentiment label distribution is:

[0061]

[0062] A bidirectional PWT decoder is constructed to decode the mixed representation of sentence vector representation and sentiment polarity score and compare it with the original sentence to achieve semi-supervised training.

[0063] The bidirectional PWT decoder method consists of two reverse PWT structures, starting from the target word and decoding its context simultaneously, so that the decoder can continuously focus on the target word information.

[0064] like Figure 3 As shown, the input of the bidirectional PWT decoder consists of four parts: the latent vector z from the encoder, the label vector y from the classifier, and the input word vector e xk and position vector p xk Therefore, the calculation formula for the conditional probability distribution is:

[0065] logp θ (x|y,a,z)=logp θ (x l |y,a,z)+logp θ (x r |y,a,z) (18)

[0066]

[0067]

[0068] This bidirectional PWT structure is also equivalent to using two decoders to generate two sentences. When generating the left half or the right half, the target word will be processed first, so the decoder always focuses on the target information.

[0069] (3) A semi-supervised method is used for the sentiment analysis network model composed of a PWT encoder, a gated classifier, and a bidirectional PWT decoder: first, the gated classifier and the bidirectional PWT decoder are trained using labeled data to obtain a partially trained classifier model. Then, the gated classifier is used to calculate the sentiment score of the unlabeled data. At this time, the classifier model parameters are fixed. Finally, the generated sentiment score and the feature vector generated by the encoder are input into the decoder together to optimize the entire encoding and decoding structure to generate a better feature vector, thereby obtaining a more accurate sentiment score.

[0070] Specifically, the semi-supervised training method is used for training. The entire semi-supervised model divides the autoencoder classification model into three parts: encoder, decoder and classifier. The encoding network will encode each word pair (x, y) in space, and the z distribution will be a Gaussian distribution q φ (z|x, y) parameterization. The decoder will generate conditions based on the vector z and the category label y. For the labeled sample pair (x, y), the corresponding latent variable variational lower bound is:

[0071]

[0072] For unlabeled samples, their category labels need to be passed through the classification model q φ (y|x) is used for prediction, so the variational lower bound is as follows:

[0073]

[0074] Therefore, for the entire data set, the objective function is the sum of the above two objective functions:

[0075]

[0076] Among them, S1 and S u They represent labeled datasets and unlabeled datasets respectively.

[0077] During the training process, all models and variational parameters should be learned together, but the loss function of the labeled dataset does not include the prediction of the label. Therefore, to solve this problem, a classifier loss needs to be added so that the distribution q can also be learned in the labeled dataset. φ (y|x), the modified objective function is as follows:

[0078]

[0079] The hyperparameter α is used to control the weight between the variational autoencoder model and the classifier model.

[0080] (4) Finally, use the trained sentiment analysis network model to calculate the sentiment scores of the unlabeled data, output the sentiment scores, and realize sentiment analysis.

[0081] The present invention can study the automatic acquisition of public opinion in the big data era and perform sentiment analysis from massive texts.

[0082] The described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, without departing from the design spirit of the present invention, various modifications and improvements made by ordinary technicians in this field to the technical solution of the present invention should fall within the protection scope determined by the claims of the present invention.

Claims

1. A target-centric semi-supervised fine-grained sentiment analysis method, characterized in that: The following steps are involved: Step 1: Build a dataset based on social media text data with sentiment labels, and use fused word vectors and position vectors to encode the input text to enrich the contextual semantic information of each word in the dataset; Step 2: construct a PWT encoder to encode the input text and obtain a sentence vector representation containing position association information; construct a gated classifier to implement fine-grained sentiment analysis and obtain a sentiment polarity score; construct a bidirectional PWT decoder to decode the mixed representation of the sentence vector representation and the sentiment polarity score; Step 3, the PWT encoder, the gated classifier, and the bidirectional PWT decoder are connected in series in sequence, and the output of the PWT encoder is also transmitted to the bidirectional PWT decoder, so as to construct a sentiment analysis network model; the sentiment analysis network model is semi-supervised trained using labeled data to obtain a trained model; Step 4: Use the trained sentiment analysis network model to calculate the sentiment score of the unlabeled data and output the sentiment score.

2. According to claim 1, a target-centric semi-supervised fine-grained sentiment analysis method is characterized in that: The specific method of step 1 is: Collect text data containing sentiment labels from social media platforms. Sentiment labels include positive, negative, and neutral categories. Clean and preprocess the collected text data to remove noise data and irrelevant information; Use the pre-trained word vector model to convert each word into a corresponding word vector to capture the semantic information of the word and fuse the word vector with the position vector; The ELMo pre-trained language model and the Glove word vector are used, and the fully connected layer is used for fusion to form an input vector containing contextual semantic information; The dataset is divided into various training sets, validation sets, and test sets for model training and evaluation.

3. The target-centric semi-supervised fine-grained sentiment analysis method according to claim 1, characterized in that: In step 2, the PWT encoder includes a two-layer Transformer encoder; The gated classifier adopts the ReLU activation function. The gated classifier uses a multi-level attention mechanism to calculate the attention of the target word from the full text, the previous text of the target word, and the following text of the target word, respectively, to obtain the attention weight and the respective feature representation, thereby enriching the contextual semantic information of the word; after obtaining the weighted context representation, the information irrelevant to the target word is further filtered through the GTRU gating mechanism; The bidirectional PWT decoder includes two reverse PWT structures, starting from the target word and decoding its context simultaneously, so that the decoder continuously focuses on the target word information.

4. The target-centric semi-supervised fine-grained sentiment analysis method according to claim 1, characterized in that: The specific method of semi-supervised training in step 3 is: First, the gated classifier and the bidirectional PWT decoder are trained using labeled data to obtain a partially trained gated classifier model. Then, the gated classifier is used to calculate the sentiment score of the unlabeled data. At this time, the gated classifier model parameters are fixed. Finally, the generated sentiment score and the feature vector generated by the PWT encoder are input into the bidirectional PWT decoder to optimize the entire encoding and decoding structure to generate better feature vectors, thereby obtaining a more accurate sentiment score.

Images