A fine-grained sentiment analysis model based on transfer learning
By improving the Senti-transfer model of the BERT language model and combining local and global sentiment semantics, the problem of insufficient data in fine-grained sentiment analysis was solved, achieving higher accuracy and effectiveness, and improving the management capabilities of online public opinion analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2021-12-21
- Publication Date
- 2026-05-15
AI Technical Summary
Existing fine-grained sentiment analysis methods suffer from insufficient labeled data and inadequate model training, resulting in insufficient accuracy in online public opinion analysis and difficulty in effectively identifying and managing complex multi-dimensional sentiment tendencies in the network.
This fine-grained sentiment analysis model based on transfer learning improves the BERT language model by introducing the Senti-transfer model, combining local and global sentiment semantics, designing a sentiment dictionary and a strategy for adding global sentiment information, and performing pre-training and fine-tuning of the model to enhance the accuracy of sentiment analysis.
It improves the accuracy of fine-grained sentiment analysis, enabling better identification and management of multi-dimensional sentiment tendencies in online public opinion, and enhancing the effectiveness of online public opinion supervision.
Smart Images
Figure CN114266257B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing, and more specifically, to a fine-grained sentiment analysis model based on transfer learning. Background Technology
[0002] Online public opinion contains a wealth of information; however, the sheer volume of information online makes it difficult to distinguish between truth and falsehood, and the public's ability to analyze and discern information has not improved. They are easily misled by biased, one-sided accounts, leading to irrational statements. In recent years, there have been numerous egregious incidents of a few malicious individuals deliberately spreading inflammatory misinformation online to mislead the public, reducing the public's enthusiasm for using the internet to understand and express their opinions, and bringing negative social impacts. Therefore, the analysis and management of online public opinion is of great significance.
[0003] According to the definition of public opinion, the ultimate focus of public opinion analysis is the analysis of sentiment. The strong emotional inclinations of the public towards public events can reflect their stance and demands. Because online public opinion is often complex, and people tend to hold different attitudes towards different aspects involved, document-level sentiment analysis can provide relatively limited information, necessitating the introduction of fine-grained sentiment analysis. However, existing fine-grained sentiment analysis methods suffer from problems such as insufficient labeled data and inadequate model training. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention proposes a fine-grained sentiment analysis model based on transfer learning. This model is an improvement on the BERT Masked Language Model (MLM) and proposes a Senti-transfer model, which includes learning both local and global sentiment semantics.
[0005] First, a priority masking strategy is implemented for sentiment words, focusing on learning the semantic information of sentiment words, including sentiment words and negative words and conjunctions that have a significant impact on the overall sentiment tendency (Senti-MLM). Senti-MLM learns local sentiment semantics. Considering the holistic and complex nature of sentence sentiment expression, two strategies for adding global sentiment information are specifically designed to better learn the relationship between local sentiment words and the overall sentiment tendency.
[0006] The specific technical solution of the present invention is as follows:
[0007] A fine-grained sentiment analysis model based on transfer learning is constructed through the following steps:
[0008] S1: Data Processing and Dictionary Construction
[0009] Select multiple known sentiment dictionaries, remove sentiment bias and sentiment score, keep only sentiment words, then clean and remove duplicates, add negative words and conjunctions to obtain the sentiment dictionaries (as shown in Table 1);
[0010] S2: Input for building the model;
[0011] S21: Basic inputs for building the model;
[0012] S22: Add global sentiment information;
[0013] S23: Words that obscure the focus on emotions;
[0014] S24: Vector transformation;
[0015] S3: Pretraining
[0016] The input vector obtained in step S24 is used as the model input, the original word at the occluded position is used as the target word, BERT-base is used as the initial model, features are extracted and encoded through BERT layers, and the output is obtained through fully connected layers and classification layers. The parameters are adjusted by comparing the difference with the target word, and the training of the MLM task is completed to obtain a pre-trained model that has learned emotional semantics.
[0017] S4: Fine-tuning training
[0018] The model obtained from the pre-training in step S3 is replaced with a new classification layer, and then trained and evaluated using a fine-grained sentiment corpus to obtain a fine-grained sentiment analysis model based on transfer learning.
[0019] Table 1 Size of the English Emotional Dictionary
[0020] Emotional word types Emotional words negative words conjunction Comparative words quantity 8396 33 98 70
[0021] Furthermore, comparison terms are introduced when constructing the dictionary in step S1.
[0022] Furthermore, step S21 specifically includes: reading the original natural language document data, performing sentence segmentation and word segmentation processing to form a set of sentences to be selected; selecting a portion of continuous sentences as input token_a, and forming token_b according to different strategies and whether it is random; and adding special symbols required by the BERT model after padding and truncation to form the overall tokens input.
[0023] Furthermore, step S22 specifically includes adding global sentiment information prompts based on the basic input obtained in step S21.
[0024] Furthermore, the strategy for adding global sentiment information in step S22 is as follows: based on the format of the basic input, determine the sentiment tendency of the documents containing token_a and token_b, and add them respectively after token_a and token_b to form the model input (e.g., Figure 2 (As shown).
[0025] Furthermore, the strategy for adding global sentiment information in step S22 is as follows: after parsing the sentiment tendency of token_a, the corresponding sentiment tendency word is used as token_b to form the model input (e.g., Figure 2 (As shown).
[0026] Furthermore, step S23 specifically includes:
[0027] (1) Load the sentiment dictionary constructed in step S1;
[0028] (2) Perform word segmentation on the model input obtained in step 22 to obtain a word list;
[0029] (3) Based on the sentiment dictionary, determine whether each word except the global sentiment information prompt words belongs to sentiment words, negation words and conjunctions, save the words in tokens that meet the judgment conditions and their position numbers as candidate words, and construct a candidate word set;
[0030] (4) Shuffle the candidate word set, randomly select a candidate token, and replace the token with [MASK] with 80% probability, replace it with a random word from the entire word list with 10% probability, and leave it unchanged with 10% probability.
[0031] (5) Repeat step (4) until the number of words that are covered meets the requirement;
[0032] (6) Return the occluded tokens, the position number of the occluded word, and the original tokens.
[0033] Furthermore, step S24 specifically includes:
[0034] Each token in the occluded tokens is converted into a word ID by referring to the vocabulary list. The word vector is extracted based on the ID. Then, the position vector and sentence vector are calculated based on the lengths of token_a and token_b, thus obtaining the input vector of the model.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] 1. This invention introduces the idea of transfer learning during model construction, transferring relevant sentiment information from document-level corpora to improve the accuracy of fine-grained sentiment analysis tasks. This method is of great significance for improving the value research of the entire network public opinion monitoring and analysis system.
[0037] 2. This invention introduces a transfer learning method, designs a transfer strategy and method for sentiment semantics, learns sentiment semantics from large-scale document-level corpora in a targeted manner, and then applies it to fine-grained sentiment analysis tasks on the terminal. It proposes and implements a fine-grained sentiment analysis model training method based on transfer learning, which can effectively improve the accuracy of text sentiment analysis. Attached Figure Description
[0038] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are the implementation process and details of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0039] Figure 1 These are two global emotion addition strategies of this invention;
[0040] Figure 2 This is a structural diagram of the fine-grained sentiment analysis model based on transfer learning of the present invention;
[0041] Figure 3 This is a schematic diagram of the fine-grained sentiment analysis fine-tuning process of the present invention. Detailed Implementation
[0042] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.
[0043] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.
[0044] To facilitate understanding of the above technical solutions of the present invention, the following detailed description of the above technical solutions of the present invention is provided through specific embodiments.
[0045] Example 1
[0046] 1. Data Preprocessing
[0047] (1) Pre-training data processing
[0048] For document-level sentiment analysis used in the transfer, the Yelp dataset and Amazon Electronic dataset were selected, with data sizes shown in Table 2. The Yelp dataset selected data from the restaurant category within the entire dataset. This involved parsing the JSON data, extracting review and rating fields, segmenting and labeling the entire review text, and then cleaning and formatting the input for BERT. Additionally, to ensure validation validity, sentences appearing in the fine-grained sentiment dataset were filtered out from the Amazon dataset. The Amazon Electronic dataset selected data from the electronics category. First, product IDs for electronic products were filtered from the product description files. Then, the corresponding review and rating data were filtered based on these IDs, followed by cleaning and formatting the input for BERT.
[0049] Table 2. Size of the pre-trained dataset
[0050]
[0051]
[0052] (2) Fine-tuning data processing
[0053] The dataset used to validate the fine-grained sentiment analysis is SemEval2014Task4, currently the most influential dataset in the industry, which includes two domains: restaurant and laptop. Furthermore, for ease of comparison, the "CONFLICT" label was removed, retaining only the three-level rating system: "Positive," "Neutral," and "Negative." Specific data is shown in Table 3.
[0054] Table 3 Information on the SemEval2014 Task 4 Fine-grained Sentiment Analysis Dataset
[0055]
[0056] 2. Input for model construction
[0057] After reading the document, it is segmented into sentences and words to form a set of candidate sentences. Then, a portion of consecutive sentences is selected as the input token_a. Token_b is then constructed according to different strategies, padded and truncated, and combined with special symbols required by the BERT model to form the overall input. Subsequently, after occlusion processing and vector transformation, the occluded sentence, the occluded position and the original word, and vectors representing the position and sentence are obtained, which serve as the input to the model.
[0058] 3. Pretraining phase training
[0059] After processing the document-level sentiment corpus according to the above steps, the model input and target words are obtained, with BERT-base as the initial model (e.g., Figure 2 As shown, the input undergoes feature extraction and encoding through multiple encoder layers, followed by a feedforward layer and a classification layer to obtain the output. The parameters are adjusted by comparing the output with the target word to complete the training of the MLM task. Finally, a pre-trained model that has learned sentiment semantics is obtained.
[0060] After encoding by the BERT model and passing through fully connected layers, the hidden state of the final output is obtained. After passing through a softmax layer, the probability distribution over the entire vocabulary is obtained. The training loss can be calculated based on the correct target word.
[0061]
[0062]
[0063]
[0064] Where W and b are the parameters to be trained, and when the i-th word is occluded, m i =1, otherwise m i =0, while y i Represents the original x i The one-hot representation of the corresponding word.
[0065] 4. Conduct training and prediction during the fine-tuning phase.
[0066] The sentiment text to be analyzed is used as input to the model obtained in step 3, and the final output is the sentiment classification result after analyzing the text (e.g., Figure 3 As shown in the figure, the results were evaluated using accuracy (acc) and F1.
[0067]
[0068]
[0069] Where n correct n represents the number of samples that correctly predicted sentiment. total This represents the total number of samples. Precision represents the number of correctly identified samples divided by the total number of predicted samples; recall represents the number of samples that were actually positive but were also correctly predicted.
[0070] Experimental Analysis
[0071] The fine-grained sentiment analysis dataset used to validate the results is the publicly available SemEval2014 Task 4 dataset, which includes two domains: restaurant and laptop. During the experimental phase, it can be seen that the model method of this invention outperforms the baseline on both datasets; among them, the back-hint method (Senti-MLM with back hint: SentiMP) achieves an average accuracy 5% higher than BERT_base.
[0072] Table 4 Results of the Senti-transfer experiment
[0073]
[0074] In addition, to more rigorously determine the effectiveness of the transfer network, this invention also conducted an ablation experiment, the specific results of which are shown in Table 5 below. It can be seen that using Senti-MLM improved the effect by 1.27%; adding global sentiment information further improved it by 1.51%, thus proving the effectiveness of the sentiment transfer network proposed in this invention, and that both Senti-MLM and global sentiment information are actually effective.
[0075] Table 5 Ablation Experiment Results
[0076] Model-Results acc f1 random_nohint_tokena 84.82％ 76.65％* sentiment_nohint_tokena 85.09％* 76.72％* sentiment_tokena 86.60％ 78.69％
[0077] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Any modifications or variations made by those skilled in the art, without departing from the scope of the present invention, using the disclosed technical content, are equivalent embodiments of the present invention. Therefore, all equivalent changes made based on the concept of the present invention without departing from the scope of the present invention should be covered within the protection scope of the present invention.
Claims
1. A method for constructing a fine-grained sentiment analysis model based on transfer learning, characterized in that, The model is constructed through the following steps: S1: Data Processing and Dictionary Construction Select multiple known sentiment dictionaries, remove sentiment bias and sentiment score, keep only sentiment words, then clean and remove duplicates, add negative words and conjunctions to obtain the sentiment dictionary; S2: Input for building the model S21: Basic inputs for building the model; S22: Add global sentiment information; S23: Words that obscure the focus on emotions; S24: Vector transformation; S3: Pretraining The input vector obtained in step S24 is used as the model input, the original word at the occluded position is used as the target word, BERT-base is used as the initial model, features are extracted and encoded through BERT layers, and the output is obtained through fully connected layers and classification layers. The parameters are adjusted by comparing the difference with the target word, and the training of the MLM task is completed to obtain a pre-trained model that has learned emotional semantics. The hidden state of the final output is then obtained through encoding by the BERT model and fully connected layers. After passing through a softmax layer, the probability distribution over the entire vocabulary is obtained. The training loss is then calculated based on the correct target word. in and These are the parameters to be trained when the i-th word is occluded. =1, otherwise =0, and Indicates the original The one-hot representation of the corresponding word; S4: Fine-tuning training The model obtained from the pre-training in step S3 is replaced with a new classification layer, and then trained and evaluated using a fine-grained sentiment corpus to obtain a fine-grained sentiment analysis model based on transfer learning. The evaluation metrics are accuracy (acc) and F 1; in This represents the number of samples that correctly predicted sentiment. The total number of samples is represented by precision, which is the number of correctly identified samples divided by the total number of predicted samples; recall is the number of samples that were actually positive and were also correctly predicted. The specific steps of S21 include: reading the original natural language document data, performing sentence segmentation and word segmentation processing to form a set of sentences to be selected; selecting a portion of continuous sentences as input token_a, and forming token_b according to different strategies and whether it is random; and adding special symbols required by the BERT model after padding and truncation to form the overall tokens input. Step S22 specifically includes: adding global sentiment information prompts based on the basic input obtained in step S21; Step S23 specifically includes: (1) Load the sentiment dictionary constructed in step S1; (2) Perform word segmentation on the model input obtained in step 22 to obtain a vocabulary list; (3) Based on the sentiment dictionary, determine whether each word except the global sentiment information prompt words belongs to sentiment words, negation words and conjunctions, save the words in tokens that meet the judgment conditions and their position numbers as candidate words, and construct a candidate word set; (4) Disorder the candidate word set, randomly select a candidate token, replace the token with [MASK] with 80% probability, replace it with a random word in the entire word list with 10% probability, and keep it unchanged with 10% probability. (5) Repeat step (4) until the number of words that are covered meets the requirement; (6) Return the occluded tokens, the position number of the occluded word, and the original tokens; Step S24 specifically includes: Each token in the occluded tokens is converted into a word ID by referring to the vocabulary list. The word vector is extracted based on the ID. Then, the position vector and sentence vector are calculated based on the lengths of token_a and token_b, thus obtaining the input vector of the model. In step S1, comparison terms are also introduced when constructing the dictionary; The strategy for adding global sentiment information in step S22 is as follows: based on the format of the basic input, determine the sentiment tendency of the documents where token_a and token_b are located, and add them to token_a and token_b respectively to form the model input; The strategy for adding global sentiment information in step S22 is as follows: after parsing the sentiment tendency of token_a, use the corresponding sentiment tendency word as token_b to form the model input.