Aspect-level sentiment classification method based on context text generation and filtering

Abstract

The invention discloses an aspect-level sentiment classification method based on context text generation and filtering. The invention provides an aspect-level sentiment classification method based on context text generation and filtering in order to solve the problems of few samples and low resources in part of fields in aspect-level sentiment classification tasks. The method comprises the following steps: firstly, carrying out context text generation by virtue of a Conditional Bert language model; secondly, a text screening and filtering algorithm is used for screening and filtering context text generation data to generate a high-quality text, and a data set is expanded; and finally, combining context text generation and filtering algorithm data to enhance the generation model. Experimental results show that the method provided by the invention effectively expands the data set, and improves the effect of aspect-level sentiment classification.

Description

Technical Field

[0001] This invention relates to the field of natural language processing sentiment analysis technology, and proposes an aspect-level sentiment method based on contextual text generation and filtering. Background Technology

[0002] Many users post comments online to express their opinions and attitudes on various issues. These comments can provide valuable insights for people making decisions in real life. However, due to the complexity and diversity of textual data, analyzing and researching this data requires significant resources. Therefore, automatically acquiring and analyzing the opinions and attitudes expressed in comment texts is a valuable research task. Holistic sentiment analysis of text may overlook the analysis of the opinion target and sometimes fail to accurately reflect the sentiment expression. In recent years, fine-grained sentiment analysis (ABSA) has received considerable attention from academia and industry. An aspect is an attribute of the object entity in the comment, and sentiment polarity refers to the sentiment tendency expressed in the aspect. Aspect-category sentiment classification (ACSC) determines the sentiment polarity information expressed by the comment text for each aspect category. It is more capable of providing corresponding sentiment tendency judgments for each specific aspect, thus offering a more comprehensive and in-depth analysis.

[0003] Currently, aspect-level sentiment classification has been extensively studied in single-domain and single-language contexts, and traditional neural network models based on deep learning have been widely applied. However, research on sentiment classification in few-shot scenarios still needs improvement. The goal of few-shot learning is to learn a problem-solving model when there are few or no labeled samples, a common problem in many applications. Therefore, how to achieve sentiment classification through few-shot learning has become a pressing issue. Text classification requires a large amount of labeled data to train deep learning models. However, labeling text datasets is a difficult and time-consuming task, resulting in a limited number of labeled datasets available for sentiment classification. To overcome this problem, data augmentation techniques can be employed, including synonym replacement, sentence restructuring, and sentence swapping.

[0004] In the field of natural language processing, data augmentation faces significant challenges due to the complexity of language semantics. Data augmentation techniques are often task-specific, requiring manually crafted systems or training task-specific language models. To reduce the workload of data annotation, data augmentation methods are commonly used to generate homogeneous but heterogeneous text data, thereby expanding labeled datasets. Because data augmentation is generated by machine rules, it is more efficient than manual annotation and therefore has high practical value.

[0005] This invention proposes an aspect-level sentiment classification method based on contextual text generation and filtering, combining the Bet contextual text generation method and text filtering algorithm to generate a model. Specifically, this model leverages the BERT model to fully mine contextual information, focusing more deeply on the connections between sentences and effectively combining the relationship between sentences and their corresponding tags, thereby generating a preliminary data-enhanced corpus. Subsequently, a filtering algorithm is introduced to optimize the initially generated corpus, removing data that does not meet the generation quality standards, thus obtaining the final enhanced text data. The aspect-level sentiment classification method based on BERT text generation and filtering algorithm data augmentation can further improve the model's performance in aspect-level sentiment classification. Summary of the Invention

[0006] This invention aims to solve aspect-level sentiment classification tasks by combining contextual text generation and filtering algorithms with data augmentation generative models.

[0007] To achieve the above objectives, and considering the impact of contextual text generation and filtering, this invention proposes an aspect-level sentiment classification method based on contextual text generation and filtering, comprising the following steps:

[0008] S1, Overall Framework of Aspect-Level Sentiment Classification Method Based on Contextual Text Generation and Filtering; This invention proposes an aspect-level sentiment classification model based on contextual text generation and filtering, mainly including BERT contextual text and text filtering algorithm modules; The BERT contextual text generation module generates a preliminary data augmentation corpus with the help of the BERT context prediction model; Secondly, the text filtering algorithm module performs quality evaluation on the BERT text generation data to select high-quality text to augment the original dataset.

[0009] S2, data preprocessing is the first step in performing aspect-level sentiment classification tasks. It processes the collected data into the input format required by the sentiment classification model, including word segmentation, adding CLS tags, padding or truncation, and generating masks.

[0010] S3. Construct a contextual text generation model based on the Conditional Bert language model. This model identifies words closely related to aspect polarity through a BERT model with conditional constraints, and does not mask them, thus making better use of the context for text generation.

[0011] S4. A text data filtering algorithm is designed. The algorithm defines a scoring function to evaluate the quality of the generated text, filters out poor-quality text based on a threshold, and generates new high-quality text to expand the original dataset.

[0012] This invention proposes an aspect-level sentiment classification method based on contextual text generation and filtering algorithms for data augmentation. First, the invention utilizes a Conditional BERT language model for contextual text generation. Second, it employs a text filtering algorithm to filter and refine the generated contextual text, producing high-quality text and expanding the dataset. Finally, it combines the contextual text generation and filtering algorithms to augment the data into a model. Experimental results demonstrate that the proposed method effectively expands the dataset and improves the performance of aspect-level sentiment classification.

[0013] According to an embodiment of the present invention, step S1 includes:

[0014] S11, the Bert context text generation module utilizes the Bert language model to add conditional constraints and optimize it into a constrained Conditional Bert context text generation model, which combines context information and label information to generate context text;

[0015] S12, the text filtering algorithm module, performs quality assessment on the preliminary data augmentation corpus generated by the Bert context text based on the filtering algorithm, selects new high-quality text, and uses it together with the original dataset as the text augmentation dataset.

[0016] According to one embodiment of the present invention, step S2 further includes:

[0017] S21, word segmentation is based on the word segmentation tool SpaCy and Python's string splitting function Split to segment text data into individual words or tags;

[0018] S22, add CLS and SEP markers at the beginning and end of the text sequence respectively. The CLS marker represents the semantics of the entire input sequence in the Bert pre-trained model.

[0019] S23, standardize the length of the input text sequence by fixing the length of the text sequence and padding or truncating the text sequence according to the actual situation;

[0020] S24, using MASK tags to mark one or more words in the text to be replaced, this process is based on the improved Conditional Bert language model constraint to mask unimportant words;

[0021] According to one embodiment of the present invention, step S3 further includes:

[0022] S31, Prepare the original dataset and the condition dataset; you need to prepare an original text dataset and a text dataset for generating conditions; the condition dataset can be a dataset containing keywords, labels, and category information;

[0023] S32, Train the Conditional Bert model; Using the prepared conditional dataset, train the Conditional Bert model; During training, take the original text data as input and the conditional data as conditional input, and train the model to generate condition-related text; On the labeled dataset D, use the conditional MLM task to fine-tune the pre-trained Bert, change the sentence block embedding to aspect label embedding, and continue fine-tuning until convergence.

[0024] S33, Generate augmented dataset: Using a trained Conditional BERT model, input the original text data and conditional data to generate conditionally related augmented text data; randomly select a sentence in D and mask k words, then use a fine-tuned Conditional BERT to predict the label-compatible words at the masked positions to generate a new sentence S′;

[0025] In the Conditioned Bert language model, an additional item X is added to the Context field, which represents the previous sentence that has already been generated;

[0026]

[0027] Where X is the generation target, C is the added context, and x i For the i-th word, x i-1 Let i be the (i-1)th word, and I be the length of the sentence;

[0028] S34, merge the original dataset and the augmented dataset; merge the generated augmented text data with the original dataset to obtain a new augmented dataset; add the new sentence S′ to the original dataset D to form the augmented dataset D′, and perform downstream tasks on D′ to verify the effect;

[0029] S35, Train the model using the expanded dataset; Train the model using the expanded dataset to improve the model's accuracy and generalization ability;

[0030] According to one embodiment of the present invention, step S4 further includes:

[0031] S41 defines a scoring function called Score to evaluate the quality of the generated text. Evaluation metrics include fluency score, grammar score, and relevance score. The relevance score is obtained by calculating the edit distance between the generated text and the original text; a smaller edit distance indicates a greater similarity between the enhanced text and the original text, resulting in a higher score. The overall score formula is as follows:

[0032]

[0033] Among them, fluency score The grammar score represents fluency. score For grammar score, relevance score To obtain the relevance score, the final overall score is calculated based on these three indicators. The overall score formula is the average of the scores of the three indicators.

[0034] S42, some low-quality datasets are filtered out through a screening mechanism; the present invention represents the overall score filtering threshold as a, where a is 3.5; therefore, when a is less than 3.5, the algorithm filters it out; and when a is greater than or equal to 3.5, it is retained.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) This invention utilizes a context-based vocabulary enhancement method, employing an improved BERT (ConditionalBERT) pre-trained language model to predict replacement words around the context of a given original word. By predicting more diverse replacement words through the language model, words can be randomly replaced to enhance tagged sentences. Randomly replacing words with other words predicted at their positions, and replacing the original word with other words, allows the model to expand sentences without compromising label compatibility.

[0037] (2) The present invention designs a text data filtering algorithm after text generation, which can filter and select the generated data, remove data with low generation quality, and maintain the accuracy and fluency of text grammar without destroying the core meaning of the original text, thereby generating the final enhanced data. Attached Figure Description

[0038] Figure 1 This is a general framework diagram of aspect-level sentiment classification based on contextual text generation and filtering according to an embodiment of the present invention.

[0039] Figure 2 This is a flowchart of aspect-level sentiment classification based on contextual text generation and text filtering. Detailed Implementation

[0040] The present invention will now be further described with reference to the accompanying drawings.

[0041] like Figure 1 As shown, the framework of this invention mainly consists of the following four steps, which are interconnected and ultimately integrated. The learning process mainly includes the following steps:

[0042] S1, Overall Framework of Aspect-Level Sentiment Classification Method Based on Contextual Text Generation and Filtering; This invention proposes an aspect-level sentiment classification model based on contextual text generation and filtering, mainly including BERT contextual text and text filtering algorithm modules; The BERT contextual text generation module generates a preliminary data augmentation corpus with the help of the BERT context prediction model; Secondly, the text filtering algorithm module performs quality evaluation on the BERT text generation data to select high-quality text to augment the original dataset.

[0043] Step S1 also includes:

[0044] S11, the Bert context text generation module utilizes the Bert language model to add conditional constraints and optimize it into a constrained Conditional Bert context text generation model, which combines context information and label information to generate context text;

[0045] S12, the text filtering algorithm module, performs quality assessment on the preliminary data augmentation corpus generated by the Bert context text based on the filtering algorithm, selects new high-quality text, and uses it together with the original dataset as the text augmentation dataset.

[0046] The basic process of aspect-level sentiment classification based on contextual text generation and text filtering is as follows: Figure 2 As shown, the dataset is first preprocessed, including word segmentation, adding CLS tags, padding or truncation, and mask generation. Next, contextual text generation is performed using a constrained Conditional BERT contextual text generation model to replace predicted contextual words and generate new text. Then, a text filtering algorithm is used to filter and select the generated data, removing low-quality data and generating the final augmented data. Finally, a sentiment classification model is used to validate the experimental results.

[0047] S2, data preprocessing is the first step in performing aspect-level sentiment classification tasks. It processes the collected data into the input format required by the sentiment classification model, including word segmentation, adding CLS tags, padding or truncation, and generating masks.

[0048] Step S2 also includes:

[0049] S21, word segmentation is based on the word segmentation tool SpaCy and Python's string splitting function Split to segment text data into individual words or tags;

[0050] S22, add CLS and SEP markers at the beginning and end of the text sequence respectively. The CLS marker represents the semantics of the entire input sequence in the Bert pre-trained model.

[0051] S23, standardize the length of the input text sequence by fixing the length of the text sequence and padding or truncating the text sequence according to the actual situation;

[0052] S24, using MASK tags to mark one or more words in the text to be replaced, this process is based on the improved Conditional Bert language model constraint to mask unimportant words;

[0053] S3. Construct a contextual text generation model based on the Conditional Bert language model. This model identifies words closely related to aspect polarity through a BERT model with conditional constraints, and does not mask them, thus making better use of the context for text generation.

[0054] Step S3 also includes:

[0055] S31, Prepare the original dataset and the condition dataset; you need to prepare an original text dataset and a text dataset for generating conditions; the condition dataset can be a dataset containing keywords, labels, and category information;

[0056] S32, Train the Conditional Bert model; Using the prepared conditional dataset, train the Conditional Bert model; During training, take the original text data as input and the conditional data as conditional input, and train the model to generate condition-related text; On the labeled dataset D, use the conditional MLM task to fine-tune the pre-trained Bert, change the sentence block embedding to aspect label embedding, and continue fine-tuning until convergence.

[0057] S33, Generate augmented dataset: Using a trained Conditional BERT model, input the original text data and conditional data to generate conditionally related augmented text data; randomly select a sentence in D and mask k words, then use a fine-tuned Conditional BERT to predict the label-compatible words at the masked positions to generate a new sentence s′;

[0058] In the Conditioned Bert language model, an additional item X is added to the Context field, which represents the previous sentence that has already been generated;

[0059]

[0060] Where X is the generation target, C is the added context, and x i For the i-th word, x i-1 Let i be the (i-1)th word, and I be the length of the sentence;

[0061] S34, merge the original dataset and the augmented dataset; merge the generated augmented text data with the original dataset to obtain a new augmented dataset; add the new sentence S′ to the original dataset D to form the augmented dataset D′, and perform downstream tasks on D′ to verify the effect;

[0062] S35, Train the model using the expanded dataset; Train the model using the expanded dataset to improve the model's accuracy and generalization ability;

[0063] This invention focuses on datasets in the area of ​​aspect-level sentiment classification, researching contextual text generation to achieve data augmentation. The entire algorithm process is as follows: First, this invention extracts and integrates the XML-formatted dataset to obtain each comment text and its corresponding aspect terms and polarity labels. Next, using information embedding, the aspect terms are embedded into a BERT pre-trained language model to obtain more accurate semantic information. On the labeled dataset D, the pre-trained BERT is fine-tuned using a conditional MLM task, changing sentence block embedding to aspect label embedding, and fine-tuning continues until convergence. Then, a sentence is randomly selected from D and k words are masked. Next, the fine-tuned conditional BERT is used to predict the label-compatible words at the masked positions, generating a new sentence S′. Finally, the new sentence S′ is added to the original dataset D to form the augmented dataset D′, and downstream tasks are performed on D′ to verify the effect.

[0064] S4. A text data filtering algorithm was designed. The algorithm defines a scoring function to evaluate the quality of the generated text, filters out poor-quality text based on a threshold, and generates new high-quality text to expand the original dataset.

[0065] Step S4 also includes:

[0066] S41 defines a scoring function called Score to evaluate the quality of the generated text. Evaluation metrics include fluency score, grammar score, and relevance score. The relevance score is obtained by calculating the edit distance between the generated text and the original text; a smaller edit distance indicates a greater similarity between the enhanced text and the original text, resulting in a higher score. The overall score formula is as follows:

[0067]

[0068] Among them, fluency score The grammar score represents fluency. score For grammar score, relevance score To obtain the relevance score, the final overall score is calculated based on these three indicators. The overall score formula is the average of the scores of the three indicators.

[0069] S42, some low-quality datasets are filtered out through a screening mechanism; the present invention represents the overall score filtering threshold as a, where a is 3.5; therefore, when a is less than 3.5, the algorithm filters it out; and when a is greater than or equal to 3.5, it is retained.

[0070] Text data filtering algorithms can retain relatively complete original text information, outputting a score from 1 to 5, with 5 being the highest quality. For example, if the original sentence is "But was so horrible us," and the enhanced text is "But the staff was so horrible," the algorithm will output a score of 2, indicating relatively good enhancement. This score combines text fluency, grammatical correctness, and similarity to the original text, and can serve as an indicator to evaluate the effectiveness of text data enhancement, filtering out poor-quality data and thus improving sentiment classification performance.

[0071] This invention preserves the aspect label information of the Semeval-Lap14 and Rest14 datasets and augments them using a Conditional BERT pre-trained language model. Then, a filtering mechanism is used to remove some low-quality datasets. The final selected dataset consists of data with an overall score of 3.5 or higher after the filtering algorithm. This data will be used as augmentation of the original text, and together with the original dataset, it will form the augmented text. The ultimate goal is to generate new text by performing a text filtering algorithm while maintaining sentence labels and polarity. This ensures that the core meaning of the original text remains unchanged, while increasing the number of original texts from 2599 to 4599, representing a 1.8-fold expansion of the Semeval-2014 restaurant review dataset.

[0072] To enhance the word-level changes, the original XML data needs to be preprocessed. Table 1 shows the data format of the Semeval-2014 restaurant dataset after preprocessing. Each text data point corresponds to an aspect term in the text, which can be used as label information for sentiment classification and corresponds to the aspect polarity of the text.

[0073] Table 1 Example of preprocessing the original data set

[0074]

[0075] The data generated through word-level and sentence-level data augmentation (1629 records in total) was merged with the original Semeval2014-Restaurant dataset training set (3174 records in total), resulting in a training set of 4803 records. Among them, there are 1610 records with a label of -1 and 3193 records with a label of 1, a ratio of approximately 2:1.

[0076] The BERT-based contextual text generation model and text filtering algorithm proposed in this invention combine text internal structure changes and post-generation filtering. To investigate the applicability of this combination of methods in aspect-level sentiment classification tasks and to explore the impact of the order of text augmentation methods on model performance, a set of comparative experiments were conducted based on Convolutional Neural Networks (CNN) and Recent Neural Networks (RNN) models. The data augmentation methods compared included augmentation methods with synonym substitution (+Synonym), the unmodified unlabeled conditional structure BERT text generation method (+Contextual), and the proposed ConditionalBERT method (+Con-BERT). These classifiers and data augmentation methods generated eight combinations: CNN, CNN+Synonym, CNN+Contextual, CNN+Con-BERT, RNN, RNN+Synonym, RNN+Contextual, and RNN+Con-BERT.

[0077] Table 2 shows the classification results of different comparison methods on the Semeval-2014 Laptop and Restaurant datasets. Table 2 indicates that the combination of the Conditional Bert contextual text generation language model and filtering improves the accuracy of the CNN model classifier on the Semeval-2014 Laptop dataset from 57.1% to 63.3%; and the accuracy using the RNN model on the Semeval-2014 Restaurant dataset from 62.1% to 64.0%. Compared to previous synonym-based augmentation methods (+Synonym), this combination of the Bert contextual text generation language model and filtering is effective for improving labeled datasets. Furthermore, the performance improvement is more significant on different datasets from the aspect-level sentiment classification domain. The CNN+Con-Bert method improves performance by 4.3% and 0.1% on the two datasets compared to the CNN+Synonym method, respectively, while the RNN+Con-Bert method improves performance by 3.7% and 3.8% on the two datasets compared to the RNN+Synonym method, respectively.

[0078] The text data filtering algorithm proposed in this invention is combined with an improved BERT model and validated through experiments. Compared with the unimproved BERT pre-trained language model (+Contextual), on the Semeval-2014 restaurant and laptop datasets, the CNN+Con-Bert method improves performance by 1.2% and 2.3% respectively compared to the CNN+Synonym method, and the RNN+Con-Bert method improves performance by 2.3% and 2.9% respectively compared to the RNN+Synonym method. This indicates that the Conditional BERT pre-trained language model can acquire aspect terms and their relationships with context, efficiently distinguishing aspect words from other contextual words. Simultaneously, the text data filtering algorithm can eliminate some poorly generated text data, thereby obtaining high-quality generated text. Therefore, the method proposed in this invention has good classification performance in aspect-level sentiment classification tasks.

[0079] Table 2. Classification results of different comparison methods on the Semeval-2014 Laptop and Restaurant data.

[0080]

[0081]

[0082] This description, in conjunction with the accompanying drawings and specific embodiments, is solely for the purpose of aiding understanding the methods and core concepts of the present invention. The methods described in this invention are not limited to the embodiments depicted in the specific implementation details. Other implementation methods derived by those skilled in the art based on the methods and concepts of this invention also fall within the scope of the technical innovations of this invention. The content of this specification should not be construed as limiting the invention.

Claims

1. An aspect-level sentiment classification method based on contextual text generation and filtering, characterized in that, Includes the following steps: S1, Overall Framework of Aspect-Level Sentiment Classification Method Based on Contextual Text Generation and Filtering; This invention proposes an aspect-level sentiment classification model based on contextual text generation and filtering, mainly including BERT contextual text and text filtering algorithm modules; The BERT contextual text generation module generates a preliminary data augmentation corpus with the help of the BERT context prediction model; Secondly, the text filtering algorithm module performs quality evaluation on the BERT text generation data to select high-quality text to augment the original dataset. Step S1 also includes: S11, the Bert context text generation module utilizes the Bert language model to add conditional constraints and optimize it into a constrained Conditional Bert context text generation model, which combines context information and label information to generate context text; S12, the text filtering algorithm module, performs quality assessment on the preliminary data augmentation corpus generated by the Bert context text based on the filtering algorithm, selects new high-quality text, and uses it together with the original dataset as the text augmentation dataset. S2, data preprocessing is the first step in performing aspect-level sentiment classification tasks. It processes the collected data into the input format required by the sentiment classification model, including word segmentation, adding CLS tags, padding or truncation, and generating masks. Step S2 also includes: S21, word segmentation is based on the word segmentation tool SpaCy and Python's string splitting function Split to segment text data into individual words or tags; S22, add CLS and SEP markers at the beginning and end of the text sequence respectively. The CLS marker represents the semantics of the entire input sequence in the Bert pre-trained model. S23, standardize the length of the input text sequence by fixing the length of the text sequence and padding or truncating the text sequence according to the actual situation; S24, using MASK tags to mark one or more words in the text to be replaced, this process is based on the improved Conditional Bert language model constraint to mask unimportant words; S3. Construct a contextual text generation model based on the Conditional Bert language model. This model identifies words closely related to aspect polarity through a BERT model with conditional constraints, and does not mask them, thus making better use of the context for text generation. Step S3 also includes: S31, Prepare the original dataset and the condition dataset; you need to prepare an original text dataset and a text dataset for generating conditions; the condition dataset can be a dataset containing keywords, labels, and category information; S32, Train the ConditionalBert model; Using the prepared conditional dataset, train the ConditionalBert model; During training, take the original text data as input and the conditional data as conditional input, and train the model to generate condition-related text; On the labeled dataset D, use the conditional MLM task to fine-tune the pre-trained Bert, change the sentence block embedding to aspect label embedding, and continue fine-tuning until convergence. S33, Generate augmented dataset: Using a trained Conditional BERT model, input the original text data and conditional data to generate conditionally related augmented text data; randomly select a sentence in D and mask k words, then use a fine-tuned Conditional BERT to predict the label-compatible words at the masked positions to generate a new sentence S′; In the Conditioned Bert language model, an additional item X is added to the Context field, which represents the previous sentence that has already been generated; Where X is the generation target, C is the added context, and x i For the i-th word, x i-1 Let i be the (i-1)th word, and I be the length of the sentence; S34, merge the original dataset and the augmented dataset; merge the generated augmented text data with the original dataset to obtain a new augmented dataset; add the new sentence S′ to the original dataset D to form the augmented dataset D′, and perform downstream tasks on D′ to verify the effect; S35, Train the model using the expanded dataset; Train the model using the expanded dataset to improve the model's accuracy and generalization ability; S4. A text data filtering algorithm was designed. The algorithm defines a scoring function to evaluate the quality of the generated text, filters out poor-quality text based on a threshold, and generates new high-quality text to expand the original dataset. Step S4 also includes: S41 defines a scoring function called Score to evaluate the quality of the generated text. Evaluation metrics include fluency score, grammar score, and relevance score. The relevance score is obtained by calculating the edit distance between the generated text and the original text; a smaller edit distance indicates a greater similarity between the enhanced text and the original text, resulting in a higher score. The overall score formula is as follows: Among them, fluency score The grammar score represents fluency. score For grammar score, relevance score To obtain the relevance score, the final overall score is calculated based on these three indicators. The overall score formula is the average of the scores of the three indicators. S42, some low-quality datasets are filtered out through a screening mechanism; the present invention represents the overall score filtering threshold as a, where a is 3.5; therefore, when a is less than 3.5, the algorithm filters it out; and when a is greater than or equal to 3.5, it is retained.

Images