A target-centered semi-supervised fine-grained sentiment analysis method

Abstract

The application discloses a target-centered semi-supervised fine-grained sentiment analysis method and belongs to the technical field of deep learning and natural language processing. The method comprises the following steps: constructing a data set based on social media text data with sentiment labels, and using fusion word vectors and position vector encoding input text to enrich the context semantic information of each word in the data set; constructing a PWT encoder, a gated classifier and a bidirectional PWT decoder; constructing a sentiment analysis network model; using labeled data to perform semi-supervised training on the sentiment analysis network model to obtain a trained model; and using the trained sentiment analysis network model to perform sentiment score calculation on unlabeled data and outputting the sentiment score. The application can be used for researching automatic public opinion acquisition in the big data era and performing sentiment analysis on massive texts.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning and natural language processing, and in particular to a target-centered semi-supervised fine-grained sentiment analysis method. BACKGROUND

[0002] In recent years, the research on automatic public opinion acquisition has always been highly valued by the academic circle. Among them, sentiment analysis from massive texts to obtain the public opinion trend of people's interest is a key technology to be solved in the big data era. Fine-grained sentiment analysis is an important part of text information classification. Unlike coarse-grained sentiment analysis which can only obtain the sentiment tendency of the entire text, fine-grained sentiment analysis can analyze the sentiment tendency of each person or thing in the text to analyze the public opinion trend. Fine-grained sentiment analysis has more practical value and landing possibility.

[0003] Fine-grained sentiment analysis focuses on the target word in the text for automatic analysis of the sentiment tendency of the target word. Existing fine-grained sentiment analysis methods are mainly based on neural network full-supervised methods, among which the deep learning method based on recurrent neural network is the most common. In the fine-grained sentiment analysis method based on deep learning, the method combining long short-term memory (LSTM) and attention mechanism is more intuitive and has achieved certain accuracy. This method uses LSTM for context feature extraction, and attention mechanism is used to locate the context related to the target word. However, this kind of method still has some shortcomings. For example, in the feature embedding layer, the traditional word vector which can only express the meaning of a single word cannot well determine the semantic meaning of the word in the specific context, while using the pre-trained language model ELMo for feature representation can well enrich the context semantic information of the word; the full-supervised method needs to label all training data, which is low in efficiency and high in cost, while the semi-supervised method can make full use of unlabeled data to achieve similar fine-grained sentiment analysis ability as the full-supervised method; the current semi-supervised training method does not modify the fine-grained analysis task, so that the semantic information and position information of the target word cannot be fully utilized, and the traditional LSTM-based method cannot be operated in parallel, which is low in efficiency. SUMMARY

[0004] The purpose of the present application is to provide a target-centered semi-supervised fine-grained sentiment analysis method to solve the problems existing in the prior art. The present application has the characteristics of high efficiency, high precision, no need for manpower and wide applicability, and is not affected by environmental factors.

[0005] The technical scheme adopted by the present application is:

[0006] A target-centered semi-supervised fine-grained sentiment analysis method, comprising the following steps:

[0007] Step 1, based on social media text data with sentiment labels to build a dataset, and use the fusion word vector, position vector to encode the input text, enrich the context semantic information of each word in the dataset;

[0008] Step 2, build PWT encoder, encode the input text to obtain sentence vector representation containing position association information; build a gated classifier to realize fine-grained sentiment analysis and obtain sentiment polarity score; build a bidirectional PWT decoder to decode the mixed representation of sentence vector representation and sentiment polarity score;

[0009] Step 3, connect the PWT encoder, gated classifier and bidirectional PWT decoder in sequence, and the output of the PWT encoder is also transmitted to the bidirectional PWT decoder, thereby constructing a sentiment analysis network model; use labeled data to semi-supervised train the sentiment analysis network model to obtain a trained model;

[0010] Step 4, use the trained sentiment analysis network model to calculate the sentiment score of unlabeled data and output the sentiment score.

[0011] Further, the specific way of step 1 is:

[0012] Collect text data containing sentiment labels from social media platforms, and the sentiment labels include positive, negative and neutral three categories;

[0013] Clean and preprocess the collected text data to remove noise data and irrelevant information;

[0014] Convert each word into a corresponding word vector using a pre-trained word vector model to capture the semantic information of the word, and fuse the word vector and the position vector;

[0015] Use ELMo pre-trained language model and Glove word vector, and adopt the method of full connection layer fusion to form an input vector containing context semantic information;

[0016] Divide the dataset into multiple training sets, validation sets and test sets for model training and evaluation.

[0017] Further, in step 2, the PWT encoder comprises a double-layer Transformer encoder;

[0018] The gating classifier adopts a ReLU activation function, the gating classifier uses a multi-level attention mechanism to perform attention calculation on the target word from the full text, the context before the target word and the context after the target word respectively, obtain attention weights and respective feature representations, so as to enrich the context semantic information of the word; after obtaining the weighted context representation, the information irrelevant to the target word is further filtered through the GTRU gating mechanism.

[0019] The bidirectional PWT decoder includes two reverse PWT structures, which simultaneously decode the context of the target word from the target word, so that the decoder continuously pays attention to the information of the target word.

[0020] Further, the specific manner of the semi-supervised training in step 3 is as follows:

[0021] First, the gating classifier and the bidirectional PWT decoder are trained using labeled data, a partially trained gating classifier model is obtained, then the gating classifier is used to calculate the sentiment score of unlabeled data, at this time the gating classifier model parameters are fixed, finally the generated sentiment score and the feature vector generated by the PWT encoder are input into the bidirectional PWT decoder, the whole encoding and decoding structure is optimized to generate a better feature vector, so as to obtain a more accurate sentiment score.

[0022] The present application has the following beneficial effects:

[0023] 1. The present application is simple, efficient and easy to implement, and can improve the training speed and feature extraction capability.

[0024] 2. The present application solves the problem of low utilization rate of unlabeled data in the prior art.

[0025] 3. The present application can study automatic public opinion acquisition in the big data era and perform sentiment analysis on massive texts. DETAILED DESCRIPTION

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0027] Figure 1 is a principle diagram of a target-centered semi-supervised fine-grained sentiment analysis method in the embodiments of the present application;

[0028] Figure 2 is a principle diagram of a PWT encoder in the embodiments of the present application;

[0029] Figure 3This is a schematic diagram of the bidirectional PWT decoder in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] A goal-centered, semi-supervised, fine-grained sentiment analysis method, such as Figure 1 As shown, the process includes feature vector generation, feature vector encoding, fine-grained sentiment analysis, and bidirectional decoding. First, it uses fused word vectors and position vectors to encode the input text, enriching the contextual semantics of each word. Then, a PWT encoder is constructed to encode the input, obtaining a sentence vector representation containing positional information. Next, a gated classifier is built to perform fine-grained sentiment analysis, obtaining a sentiment polarity score. The parallel attention mechanism calculates the contribution of each word to the target from multiple perspectives, while the GTRU gating mechanism controls the flow of contextual information. Then, a bidirectional decoder is used to decode the hybrid representation of the sentence vector representation and the sentiment polarity score, comparing it with the original sentence to achieve semi-supervised training. Finally, the trained model is used for sentiment analysis.

[0033] The specific steps of this method are as follows:

[0034] (1) By collecting a large amount of social media text data with sentiment labels (from labeled datasets or obtained through manual annotation), a neural network training dataset that can be used for semi-supervised methods is formed. Input text is encoded using fused multidimensional word vectors and absolute position vectors to obtain the contextual information of words and the position information of target words, and to generate feature vectors; among them, the ELMo pre-trained language model and GloVe word vectors are fused using a fully connected layer to achieve fused multidimensional word vectors.

[0035] Specifically, in the word embedding layer, in order to obtain the semantics of each word in the context, for each input text sequence {x0, x1, ..., x...} t-1 x t x t+1 , ..., x n-1 x n},in which the word vector e pre-trained by ELMo is added on the basis of the traditional word vector embedding w. Unlike the traditional N-gram-based word embedding model, the word vector representation in the ELMo pre-training model is dynamically adjusted according to the context semantics, so the introduction of the ELMo word vector is necessary, and compared with BERT, the demand for computing resources of ELMo is much smaller, so the word vector representation s fused by the full connection layer (FC) is:

[0036] s=FC(w+e) (1)

[0037] (2) A PWT (Position Wise Transformer) encoder is constructed to encode the feature vector and obtain a sentence vector representation containing position-related information. The PWT encoder uses a double-layer Transformer encoder as the basic architecture to improve the training speed and feature extraction capability, and the multi-head self-attention layer contained therein introduces trainable relative position parameters.

[0038] As shown in Figure 2 , the standard input of the Transformer includes the word vector e xk and the position vector p xk . For the target-oriented sentiment classification task, in order to better distinguish the target and the context, the segmented markers are used for distinction when inputting, the target words are marked with 1, the context of the target words is marked with 0, and the vector s xi is converted as the input. At the same time, the relative position information of the target entity can also help the model to better focus on the relationship between the target entity and the adjacent context, so the PWT encoder based on the Transformer is proposed, and in the specific implementation, the parameters representing the relative position are added when calculating the self-attention score of the Transformer model, and the parameters are shared among the attention heads. The process is formally represented as:

[0039]

[0040] Wherein, A i represents the self-attention score at position i, h i and h j are the representations of the word pair (x i , x j ) from the previous multi-head self-attention layer, W Q , W K and W V are the trainable parameters of the model, and d z is the output dimension of the self-attention layer. Compared with the standard Transformer self-attention model, the PWT increases two parameters and x is used to represent the input x i , x j boundary information, but in this method, it is used to obtain the relative distance between the target entity and each word, that is, i only represents the target word, when i is a non-target word, and set to zero. and The specific formula is as follows:

[0041]

[0042] dis(x, k) = max(-k, min(k, x)) (6)

[0043] The gated classifier is used to perform fine-grained sentiment analysis on the feature vector to obtain a sentiment polarity score. The multi-level attention mechanism of the gated classifier performs attention calculation on the target word from the full text, the pre-context of the target word and the post-context of the target word respectively to obtain attention weights and respective feature representations, thereby enriching the context semantic information of the words. The GTRU gating mechanism of the gated classifier uses the attention mechanism to calculate the contribution of each word to the target, and after obtaining the weighted context representation, it controls the flow direction of the context information, and further filters the information irrelevant to the target word, thereby ignoring the interference of the non-target word related information.

[0044] The gated classifier uses a ReLU activation function instead of a traditional Sigmoid function, and the zeroing feature in the negative interval can filter out irrelevant information and only retain specific target related sentiment information.

[0045] Specifically, the gated classifier first uses the attention mechanism to calculate the contribution of each word to the target, and for the input word vectors s0, s1,..., s n , uses a BILSTM network to extract semantics to obtain vectors h0, h1,..., h n :

[0046] [h0, h1,..., h n ] = BILSTM ([s0, s1,..., s n ]) (7)

[0047] In the hidden vector, the target word is represented as h t , when the target word is a short text composed of multiple words, the target word vector is represented as the mean value of the segment text representation [h t0 , h t1 ,..., h tm ]. In the attention layer, the full text, the pre-context of the target word and the post-context of the target word perform attention calculation on the target word respectively to obtain attention weights α iand each feature representation s, s l , s r , and the specific formula is as follows:

[0048]

[0049]

[0050] The calculation method of attention is illustrated below taking the calculation of s as an example:

[0051]

[0052] wherein the weight β i is calculated from the target word representation and the context word representation:

[0053] β i = U T tanh(W1·[h i : h t ]+b1) (12)

[0054] After obtaining the weighted representation, a GTRU gating mechanism is used to control the flow of information. Emotional information can be embodied by the context of the target word, the context after the target word, or the entire sentence. Therefore, three gates z, z l , z r are used to control the target word and the corresponding context:

[0055] z = ReLU(W1s + U1h t +b1) (13)

[0056] z l = ReLU(W2s l +U2h t +b2) (14)

[0057] z r = ReLU(W3s r +U3h t +b3) (15)

[0058] wherein the linear interpolation formula between s, s l , s r is:

[0059]

[0060] The final predicted emotional label distribution is:

[0061]

[0062] A bidirectional PWT decoder is constructed to decode the hybrid representation of sentence vector representation and sentiment polarity score, compared with the original sentence to realize semi-supervised training.

[0063] The bidirectional PWT decoder method includes two reverse PWT structures, which simultaneously decode the context of the target word from the target word, so that the decoder continuously focuses on the target word information.

[0064] As shown in Figure 3 , the input of the bidirectional PWT decoder includes four parts, which are the hidden vector z from the encoder, the label vector y from the classifier, the input word vector e xk and the position vector p xk . Therefore, the calculation formula of the conditional probability distribution is:

[0065] logp θ (x|y,a,z) = logp θ (x l |y,a,z) + logp θ (x r |y,a,z) (18)

[0066]

[0067]

[0068] This bidirectional PWT structure can also be equivalent to using two decoders to generate two sentences, when generating the left half or the right half, the target word will be processed first, so the decoder always focuses on the target information.

[0069] (3) The semi-supervised method is used for the sentiment analysis network model composed of PWT encoder, gated classifier and bidirectional PWT decoder: first, use the labeled data to train the gated classifier and bidirectional PWT decoder, obtain the partially trained classifier model, then use the gated classifier to calculate the sentiment score of the unlabeled data, at this time the classifier model parameters are fixed, finally the generated sentiment score and the feature vector generated by the encoder are input into the decoder together, the whole encoder-decoder structure is optimized to generate better feature vectors, so as to obtain more accurate sentiment scores.

[0070] Specifically, the semi-supervised training method is used for training, and the whole semi-supervised model divides the autoencoder classification model into three parts: encoder, decoder and classifier. The encoding network encodes each word pair (x, y) in space, and the z distribution is parameterized by a Gaussian distribution q φ (z|x,y). The decoder generates conditionally according to the vector z and the category label y. For the labeled sample pair (x, y), the corresponding hidden variable variational lower bound is:

[0071]

[0072] And for the unlabeled samples, their class labels need to be predicted by the classification model q φ (y|x), so the variational lower bound is as follows:

[0073]

[0074] Therefore, for the whole dataset, the objective function is the sum of the above two objective functions:

[0075]

[0076] Where S1 and S u respectively represent the labeled dataset and the unlabeled dataset.

[0077] During the training process, all models and variational parameters should be learned together, but the prediction of the label is not included in the loss function of the labeled dataset, so a classifier loss needs to be added to solve this problem, so that the distribution q φ (y|x) can also be learned in the labeled dataset, and the modified objective function is as follows:

[0078]

[0079] The hyperparameter alpha is used to control the weight between the variational autoencoder model and the classifier model.

[0080] (4) Finally, the trained sentiment analysis network model is used to calculate the sentiment score of the unlabeled data, and the sentiment score is output, realizing sentiment analysis.

[0081] The present application can research automatic public opinion acquisition in the big data era, and perform sentiment analysis from massive texts.

[0082] The described embodiments are part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, various modifications and improvements of the technical solutions of the present application made by those skilled in the art without departing from the design spirit of the present application shall fall within the protection scope of the claims of the present application.

Claims

1.A target-centered semi-supervised fine-grained sentiment analysis method, characterized in that, The method comprises the following steps: Step 1, constructing a dataset based on social media text data with emotional labels, and using fused word vectors and position vectors to encode the input text, enriching the context semantic information of each word in the dataset; Step 2, constructing a PWT encoder to encode the input text and obtain a sentence vector representation containing position-related information; constructing a gated classifier to realize fine-grained sentiment analysis and obtain a sentiment polarity score; constructing a bidirectional PWT decoder to decode the mixed representation of the sentence vector representation and the sentiment polarity score; Step 3, connecting the PWT encoder, the gated classifier and the bidirectional PWT decoder in sequence, and the output of the PWT encoder is also transmitted to the bidirectional PWT decoder, thereby constructing a sentiment analysis network model; using labeled data to perform semi-supervised training on the sentiment analysis network model to obtain a trained model; wherein the specific way of semi-supervised training is: First, use labeled data to train the gated classifier and the bidirectional PWT decoder to obtain a partially trained gated classifier model, then use the gated classifier to calculate the sentiment score of unlabeled data, at this time the gated classifier model parameters are fixed, and finally the generated sentiment score and the feature vector generated by the PWT encoder are input into the bidirectional PWT decoder to optimize the entire encoding and decoding structure; Step 4, using the trained sentiment analysis network model to calculate the sentiment score of unlabeled data and output the sentiment score. 2.The target-centered semi-supervised fine-grained sentiment analysis method according to claim 1, wherein, The specific way of step 1 is: Collect text data containing emotional labels from social media platforms, and the emotional labels include positive, negative and neutral three categories; Clean and preprocess the collected text data to remove noise data and irrelevant information; Convert each word into a corresponding word vector using a pre-trained word vector model to capture the semantic information of the word, and fuse the word vector and the position vector; Use the ELMo pre-trained language model and the Glove word vector to form an input vector containing context semantic information by using a full connection layer for fusion; Divide the dataset into multiple training sets, validation sets and test sets for model training and evaluation. 3.The target-centered semi-supervised fine-grained sentiment analysis method of claim 1, wherein, In step 2, the PWT encoder comprises a double-layer Transformer encoder; The gated classifier uses a ReLU activation function, and the gated classifier uses a multi-level attention mechanism to perform attention calculation on the target word from the full text, the pre-context of the target word and the post-context of the target word respectively to obtain attention weights and respective feature representations, thereby enriching the context semantic information of the word; after obtaining the weighted context representation, the GTRU gating mechanism is used to further filter the information irrelevant to the target word; The bidirectional PWT decoder includes two reverse PWT structures, which simultaneously decode the context of the target word from the target word, so that the decoder continuously focuses on the target word information.

Images