A sentiment analysis method and device based on RoBERTa model
By connecting the two-way independent recurrent neural network and attention mechanism on the RoBERTa model, the problem of complex feature engineering and low data in Chinese network text sentiment analysis is solved, and the accuracy and efficiency of sentiment analysis is improved, especially suitable for Weibo and Zhida The polar classification of comments and comments on platforms such as Hu.
Patent Information
- Application Number
- CN202210301411.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-03-24
AI Technical Summary
In the analysis of Chinese online text sentiment, the problem of complex feature engineering, model accuracy depends on sample quality and low data volume in the existing technology, especially in the classification of comments and comments sentiment polarity of platforms such as Weibo and Zhihu, which is difficult to be effective. Improve the feature extraction and calculation speed of the model.
The RoBERTa model is used for pre-training and externally connected to the bidirectional independent recurrent neural network. Through random mask processing and attention mechanism, the RoBERTa-WWM-ext model is built for sentiment analysis and improve the model's feature extraction ability and calculation speed.
It has achieved the improvement of the accuracy of text sentiment analysis under low resource data conditions, especially in the sentiment analysis tasks with complex semantic structure and large text span of comments and messages on Chinese social platforms, which significantly improves the accuracy and efficiency of emotion classification.
Smart Images

Figure CN114742066B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of natural language processing, and in particular relates to a sentiment analysis method and device based on a RoBERTa model. Background Art
[0002] With the promotion of the Internet, users are more inclined to express their opinions on public events on social media or comment websites. Designing sentiment analysis tasks based on text comments on online platforms can help relevant parties understand public opinion tendencies and grasp the impact of social events related to public opinion. Sentiment analysis is an important branch of natural language processing. The common form is text-based sentiment polarity classification, which belongs to supervised learning tasks. Its purpose is to take comments from actual scenarios as input and predict their sentiment tendencies. Sentiment analysis technology can be used to collect opinions as feedback information, which has practical significance in promoting social services.
[0003] Sentiment classification methods based on traditional machine learning, such as support vector machines, sentiment dictionary construction, and naive Bayes, require complex feature engineering, and the accuracy of the model depends on the quality of the sample. The feedforward neural network model improves the model's ability to understand deep-level abstract semantic features by deepening the hidden layer, but it still requires a lot of manual feature engineering. Convolutional neural networks can extract information through convolution windows and map it into sentiment categories after multi-layer convolution processing. Recurrent neural networks can map text sequences of indefinite length into fixed-length word representations. Chinese network text sentiment analysis data has the characteristics of weak sentiment tendency and small amount of labeled data. The bidirectional encoder model BERT based on the self-attention mechanism of the Transformer encoder and the more robust pre-trained model RoBERTa, through the domain adaptation method of self-supervised pre-training, large-scale parameter quantity and training intensity, enable the model to obtain powerful feature extraction and context understanding capabilities, and can achieve better classification results than traditional deep learning models in sentiment analysis tasks using such data sets.
[0004] Based on the above content, the present invention proposes to further pre-train the RoBERTa model, migrate this model as a basic model for learning long-distance dependent independent recurrent neural network (Indpendently Recurrent NeturlNetwork), and introduce the attention mechanism to improve the calculation speed and fitting effect. The present invention can effectively classify the sentiment polarity of netizens' comments and messages such as Weibo and Zhihu. Summary of the invention
[0005] In order to solve the above-mentioned defects, the present invention proposes a sentiment analysis method and device based on the RoBERTa model.
[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0007] A sentiment analysis method based on the RoBERTa model, comprising:
[0008] Obtain sentiment analysis data sets and preprocess them to obtain training sets, validation sets, and test sets;
[0009] Extract the review texts from the training set and validation set and convert them into unlabeled datasets;
[0010] Construct a RoBERTa-WWM-ext model, randomly mask the comment text in the unlabeled dataset to obtain the mask flags, and then train the RoBERTa-WWM-ext model by predicting words on the mask flags to obtain a trained RoBERTa-WWM-ext model;
[0011] Construct and train a bidirectional independent recurrent neural network. Connect the last hidden layer of the RoBERTa-WWM-ext model that completes the mask prediction training task to the bidirectional independent recurrent neural network to obtain the final sentiment analysis model. The bidirectional independent recurrent neural network needs to assign weights to the feature vectors output by the bidirectional independent recurrent neural network.
[0012] Obtain the final sentiment analysis model, input the comment text in the test set into the sentiment analysis model for sentiment polarity prediction, and output the sentiment category label.
[0013] A further improvement of the method is that the extraction of the review texts of the training set and the validation set into an unlabeled data set comprises:
[0014] Convert the format of the unlabeled dataset into a specified table format.
[0015] A further improvement of the method is that: the RoBERTa-WWM-ext model is constructed, the unlabeled text is randomly masked to obtain masked flags, and then the RoBERTa-WWM-ext model is trained by predicting words on the masked flags, and the trained RoBERTa-WWM-ext model includes:
[0016] Through the pre-training process of mask prediction, the parameters of the RoBERTa-WWM-ext model are updated to obtain a more reasonable initialization for the RoBERTa-WWM-ext model when performing sentiment analysis tasks.
[0017] A further improvement of this method is that: the bidirectional independent recurrent neural network is constructed and trained, and the last hidden layer of the RoBERTa-WWM-ext model that completes the mask prediction training task is connected to the bidirectional independent recurrent neural network to obtain the final sentiment analysis model, wherein the bidirectional independent recurrent neural network needs to perform weight distribution on the feature vector output by the bidirectional independent recurrent neural network, including:
[0018] The sample sequence is encoded as an input sequence containing word vectors, position vectors, and segment vectors, which are input into the core processing layer of the RoBERTa-WWM-ext model composed of a multi-head self-attention mechanism for feature extraction. A bidirectional independent recurrent neural network is constructed to process all hidden vectors output by the last Transformer layer of the RoBERTa-WWM-ext model, and to learn the long-distance dependency information in the sentence by combining the feature vectors of the forward sequence and the reverse sequence.
[0019] A further improvement of this method is that the structural formula of the bidirectional independent recurrent neural network is:
[0020] h t =σ(Wx t +U⊙h t-1 +b)
[0021] In the formula, σ is the ReLU activation function, W is the weight matrix, and x t represents the input at time t, U represents the weight of the loop layer, and h t-1 represents the sequence output at time t-1, b is the bias, and the operator ⊙ represents the matrix U and h t-1 Multiply the corresponding positions.
[0022] A further improvement of this method is that the weight distribution formula is:
[0023]
[0024] Among them, u i is the nonlinearization of the hidden state information, u w is a randomly initialized attention matrix, α i Used to calculate the information relevance of the i-th item in the hidden layer for the text sequence in the sentiment analysis task.
[0025] A further improvement of this method is that: the sentiment analysis model is trained by a cross entropy function to obtain an optimized sentiment analysis model.
[0026] A sentiment analysis device based on the RoBERTa model, comprising:
[0027] The acquisition module is used to obtain the comment text and preprocess it to obtain the training set, validation set and test set, and extract the comment text of the training set and validation set to convert them into unlabeled data sets;
[0028] The first construction module is used to construct a RoBERTa-WWM-ext model, randomly mask the comment text in the unlabeled data set to obtain mask flags, and then train the RoBERTa-WWM-ext model by predicting words on the mask flags to obtain a trained RoBERTa-WWM-ext model;
[0029] The second construction module is used to construct and train a bidirectional independent recurrent neural network. The last hidden layer of the RoBERTa-WWM-ext model that completes the mask prediction pre-training task is connected to the bidirectional independent recurrent neural network to obtain the final sentiment analysis model. The bidirectional independent recurrent neural network needs to perform weight allocation on the feature vector output by the bidirectional independent recurrent neural network.
[0030] The output module inputs the comment text in the test set into the final sentiment analysis model for sentiment polarity prediction and outputs the sentiment category label.
[0031] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is as follows: This application designs a self-supervised masked language model pre-training task, making full use of limited data to encode semantic knowledge in the field into the model level. The pre-trained RoBERTa-WWM-ext model is migrated, retaining the super feature extraction ability of the large neural network model. The last layer of all hidden state information of the RoBERTa-WWM-ext model is used as the input of the bidirectional independent recurrent neural network to obtain global contextual structure knowledge. This application proposes the design of an attention mechanism that can make the model pay more attention to feature information that is more useful for sentiment analysis tasks and speed up the calculation process. The present invention can design a model for the low-resource type of training data in sentiment analysis tasks, the complex semantic structure of comments and messages on Chinese social platforms, and the large text span, which can effectively improve the accuracy of text sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 is a model diagram of the sentiment analysis system of the present invention;
[0033] Figure 2 This is a simplified diagram of the bidirectional independent recurrent neural network structure of the present invention. DETAILED DESCRIPTION
[0034] The present invention is further described in detail below in conjunction with the accompanying drawings:
[0035] like Figure 1 As shown, the present invention proposes a sentiment analysis method and device based on the RoBERTa model, the method comprising the following steps:
[0036] Obtain sentiment analysis data sets and preprocess them to obtain training sets, validation sets, and test sets;
[0037] Extract the review text of the training set and validation set and convert it into an unlabeled dataset;
[0038] Construct the RoBERTa-WWM-ext model, randomly mask the comment text in the unlabeled dataset to obtain the mask flag, and then train the RoBERTa-WWM-ext model by predicting the words on the mask flag to obtain the trained RoBERTa-WWM-ext model
[0039] Construct and train a bidirectional independent recurrent neural network. Connect the last hidden layer of the RoBERTa-WWM-ext model that completes the mask prediction training task to the bidirectional independent recurrent neural network to obtain the final sentiment analysis model. The bidirectional independent recurrent neural network needs to assign weights to the feature vectors output by the bidirectional independent recurrent neural network.
[0040] Obtain the final sentiment analysis model, input the comment text in the test set into the sentiment analysis model for sentiment polarity prediction, and output the sentiment category label.
[0041] A device based on the above-mentioned sentiment analysis method based on the RoBERTa model includes: an acquisition module, a first construction module, a second construction module and an output module.
[0042] Among them, the acquisition module is used to obtain the sentiment analysis data set and preprocess it to obtain the training set, validation set and test set, and extract the comment texts of the training set and validation set to convert them into unlabeled data; the first construction module is used to build the RoBERTa-WWM-ext model, randomly mask the comment texts in the unlabeled data set to obtain the mask flags, and then train the RoBERTa-WWM-ext model by predicting words on the mask flags to obtain the trained RoBERTa-WWM-ext model; the second construction module is used to build and train a bidirectional independent recurrent neural network, and connect the last hidden layer of the RoBERTa-WWM-ext model that completes the mask prediction training task to a bidirectional independent recurrent neural network to obtain the final sentiment analysis model, wherein the bidirectional independent recurrent neural network needs to assign weights to the feature vectors output by the bidirectional independent recurrent neural network; the output module is used to predict the sentiment polarity of the comment texts in the test set through the final sentiment analysis model, and output the sentiment category label.
[0043] It should be noted that the comment text can come from the Internet, and can use the comment text from Bilibili, Weibo, Kuaishou, etc.
[0044] The following is combined with the following embodiments:
[0045] Step 1: The sentiment analysis dataset comes from DataFountain's open source dataset. The samples are derived from real content in Weibo, including 900,000 unlabeled Weibo comments and 10 manually annotated data that can be used for supervised learning. After obtaining the dataset, first filter out the sample data with incorrect annotations and garbled comment text in the 100,000 labeled data. The labeled data after preliminary filtering contains 25,365 samples with positive sentiment categories, 57,277 neutral samples, and 16,896 negative samples. Randomly extract the labeled data and divide it into the sentiment analysis task training set, validation set, and test set in a ratio of 8:1:1.
[0046] Step 2: Extract the training set and validation set of the sentiment analysis dataset, remove the sentiment polarity label as unlabeled data. Introduce large-scale unlabeled microblog comment content to expand the unsupervised dataset, and save each comment sample line by line in UTF-8 format.
[0047] Step 3: Load the RoBERTa-WWM-ext model for further pre-training of self-supervised mask prediction. In the optimization study of the bidirectional encoder model for Chinese tasks, the full-word masking strategy can reduce the uncertainty of word meaning and speed up the calculation. The unsupervised data set preprocessed in step 2 is randomly masked with a probability of 15%, and the original word at the [Mask] position is restored by prediction to train the RoBERTa-WWM-ext model to obtain bidirectional semantic information. For example, the sentence: "If you have a fever, you must pay attention." After masking, "If [Mask][Mask] appears, you must pay attention." The training loss function of the self-supervised mask language model (Mask Language Model, MLM) is as follows:
[0048]
[0049]
[0050] Where x is the token sequence {[s], x 1 ,...,x n [ / s]}, m(x) represents the set of masked tokens, x\ m(x)is the token that has not been replaced in the input sequence, η is the learning rate during model MLM training, and λ is the L2 regularization parameter used to stabilize the value of the model parameter θ to avoid overfitting.
[0051] Step 4: After the migration is completed, the RoBERTa-WWM-ext model in step 3 obtains more reasonable initialization parameters than the original model. A bidirectional recurrent network based on IndRNN is constructed by using nonlinearization, layer regularization, and full connection. Its basic internal structure is as follows: Figure 2 , all hidden states of the last layer of the RoBERTa-WWM-ext model are output as bidirectional independent recurrent neural networks. The IndRNN basic unit can improve the gradient vanishing and gradient exploding problems in neural networks and is more conducive to preprocessing serialized long dependencies. The structural expression is as follows:
[0052] h t =σ(Wx t +U⊙h t-1 +b)
[0053] In the formula, σ represents the ReLU activation function, W is the weight matrix, and x t represents the input at time t, U represents the weight of the loop layer, and h t-1 represents the sequence output at time t-1, b is the bias, and the operator ⊙ represents the matrix U and h t-1 Use a bidirectional network structure to learn global semantic features and output the IndRNN forward. and reverse output For splicing, the expression is as follows:
[0054]
[0055] Step 5: Combine the attention mechanism to assign weights to the output vectors of the bidirectional independent recurrent neural network, and introduce more attention to the representation of important words in the feature vector. The specific calculation process is as follows:
[0056] u i =tanh(W W h i +b w )
[0057]
[0058]
[0059] The hidden state h output by the bidirectional independent recurrent neural network i Perform nonlinear activation to obtain u i , W W is the weight matrix parameter bw is the bias parameter. i is the weight parameter, v i It is the feature vector after weight assignment.
[0060] Step 6: Use the softmax classifier to identify the category attributes of the text features output by the attention mechanism. The formula is:
[0061]
[0062] Where W w is the weight parameter, b i is the bias, v i is the input of the classifier. The parameters of the model are trained using the gradient descent method, and the cross entropy loss function is expressed as follows:
[0063]
[0064]
[0065] K is the total number of categories for sentiment classification, y i is the true sentiment category label corresponding to the sample, is the sentiment label predicted by the model, λ k is the L2 regularization coefficient, Θ is the set of model parameters, and γ is the learning rate of the current layer update parameters.
[0066] In order to illustrate the practicality and effectiveness of the present invention, a specific experimental scheme is given below, and the evaluation indicators of the test set are accuracy Acc and macro F1. The hyperparameters are set as follows: the number of training rounds in the masked language learning stage is 3, the random inactivation probability is 0.1, the learning rate is 5e-5, the batch size is 64, and the L2 regularization coefficient is 1e-5. The training rounds of the sentiment analysis task are 4, the learning rate is 5e-5, the batch size is 32, and the regularization coefficient is 1e-5. The hidden layer dimension of the independent cyclic network is 128 dimensions, and the random inactivation probability is set to 0.5. The maximum text length is set to 256 in the pre-training stage and the fine-tuning stage of the sentiment analysis task, and the Adam optimizer is used to iterate and correct the parameters multiple times. Based on sentiment analysis, the test set is used to perform three classifications of sentiment polarity {positive, neutral, negative}, and the accuracy is 74.49% accuracy and 71.23% F1 value.
[0067] The embodiments described above are merely descriptions of preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. A sentiment analysis method based on the RoBERTa model, characterized in that: include: Obtain sentiment analysis data sets and perform data preprocessing to obtain training sets, validation sets, and test sets; Extract the review text of the training set and validation set and convert it into an unlabeled dataset; Construct a RoBERTa-WWM-ext model, perform random masking on the comment text in the unlabeled dataset to obtain mask flags, and then train the RoBERTa-WWM-ext model by predicting words on the mask flags to obtain a trained RoBERTa-WWM-ext model. Construct and train a bidirectional independent recurrent neural network. Connect the last hidden layer of the RoBERTa-WWM-ext model that completes the mask prediction training task to the bidirectional independent recurrent neural network to obtain the final sentiment analysis model. The bidirectional independent recurrent neural network needs to assign weights to the feature vectors output by the bidirectional independent recurrent neural network. The final sentiment analysis model is used to predict sentiment polarity, and the comment text in the test set is input into the sentiment analysis model to output the sentiment category label; The bidirectional independent recurrent neural network is constructed and trained, and the last hidden layer of the RoBERTa-WWM-ext model that completes the mask prediction training task is connected to the bidirectional independent recurrent neural network to obtain the final sentiment analysis model, wherein the bidirectional independent recurrent neural network needs to perform weight distribution on the feature vector output by the bidirectional independent recurrent neural network, including: The sample sequence is encoded into an input sequence containing word vectors, position vectors, and segment vectors, and input into the core processing layer of the RoBERTa-WWM-ext model composed of a multi-head self-attention mechanism for feature extraction. A bidirectional independent recurrent neural network is constructed to process all hidden vectors output by the last Transformer layer of the RoBERTa-WWM-ext model, and the feature vectors of the forward sequence and the reverse sequence are combined to learn the long-distance dependency information in the sentence.
2. The sentiment analysis method based on the RoBERTa model according to claim 1, characterized in that: The extraction of the review text of the training set and the validation set into unlabeled text includes: Convert the format of the unlabeled dataset into a specified table format.
3. The sentiment analysis method based on the RoBERTa model according to claim 1, characterized in that: The RoBERTa-WWM-ext model is constructed by randomly masking the comment text in the unlabeled data set to obtain the mask flag, and then the RoBERTa-WWM-ext model is trained by predicting words on the mask flag. The trained RoBERTa-WWM-ext model includes: Through the pre-training process of mask prediction, the parameters of the RoBERTa-WWM-ext model are updated to obtain a more reasonable initialization for the RoBERTa-WWM-ext model when performing sentiment analysis tasks.
4. The sentiment analysis method based on the RoBERTa model according to claim 1, characterized in that: The structural formula of the bidirectional independent recurrent neural network is: h t =σ(Wx t +U⊙h t-1 +b) In the formula, σ is the ReLU activation function, W is the weight matrix, and x t represents the input at time t, U represents the weight of the loop layer, and h t-1 represents the sequence output at time t-1, b is the bias, and the operator ⊙ represents the matrix U and h t -1 Multiply the corresponding positions.
5. The sentiment analysis method based on the RoBERTa model according to claim 1, characterized in that: The weight distribution formula is: Among them, u i is the nonlinearization of the hidden state information, u w is a randomly initialized attention matrix, α i Used to calculate the information relevance of the i-th item in the hidden layer for the text sequence in the sentiment analysis task.
6. The sentiment analysis method based on the RoBERTa model according to claim 1, characterized in that: The sentiment analysis model is trained by a cross entropy function to optimize the sentiment analysis model.
7. A sentiment analysis device using the sentiment analysis method based on the RoBERTa model of claim 1, characterized in that: include: The acquisition module is used to obtain the sentiment analysis data set and preprocess it to obtain the training set, validation set and test set, and extract the comment text of the training set and validation set to convert them into an unlabeled data set; The first construction module is used to construct a RoBERTa-WWM-ext model, randomly mask the comment text in the unlabeled data set to obtain mask flags, and then train the RoBERTa-WWM-ext model by predicting words on the mask flags to obtain a trained RoBERTa-WWM-ext model; The second construction module is used to construct and train a bidirectional independent recurrent neural network. The last hidden layer of the RoBERTa-WWM-ext model that completes the mask prediction training task is connected to the bidirectional independent recurrent neural network to obtain the final sentiment analysis model. The bidirectional independent recurrent neural network needs to perform weight allocation on the feature vector output by the bidirectional independent recurrent neural network. The output module obtains the final sentiment analysis model, inputs the comment text in the test set into the sentiment analysis model for sentiment polarity prediction, and outputs the sentiment category label.
