A comparative sentence sentiment analysis method and system based on BERT adversarial training

By using pre-trained language model BERT and adversarial training techniques in comparative sentence sentiment analysis, deep semantic and logical relationships are generated, the problem of insufficient number of comparison sentence samples in the dataset is solved, and sentiment analysis performance is improved, especially in the 'WORSE' category Identify the effect.

CN114911931BActive Publication Date: 2024-11-15XI AN JIAOTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210421996.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-21
Publication Date
2024-11-15
Estimated Expiration
2042-04-21

AI Technical Summary

Technical Problem

The existing comparative sentence sentiment analysis methods fail to make full use of the deep semantic and contextual dependencies of the comment text, and the low sentiment analysis performance is due to the small number of comparison sentence samples in the dataset, especially the small number of ‘WORSE’ category samples.

Method used

The pre-trained language model BERT is used to encode comment texts, and combined with the adversarial training process, generate text semantic information, context dependencies and logical relationships, obtain the emotional probability distribution of the comparison sentences through the Softmax function, and solve sample imbalance using Focal-loss loss function. question.

Benefits of technology

Effectively identify the deep semantic and logical relationships in comparison sentences, reduce the interference effects of non-comparative sentences, improve the accuracy of emotion recognition in category ‘WORSE’, and significantly improve the performance of emotion analysis in comparison sentences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114911931B_ABST
    Figure CN114911931B_ABST
Patent Text Reader

Abstract

The present invention discloses a comparative sentence sentiment analysis method and system based on BERT adversarial training. By combining the pre-trained language model BERT with the adversarial training process, the method innovatively obtains richer semantic feature information in the comment text, reduces the noise impact caused by a large number of non-comparative sentences in the data set, and improves the performance of comparative sentence sentiment analysis. The present invention introduces the pre-trained language model BERT to encode the input comment text, effectively identifying the deep semantics and logical relationships in the comparative sentence text; by introducing the adversarial training process, the interference caused by a large number of non-comparative sentences in the data set can be reduced, and the "WORSE" category text with a small data sample in the comparative sentence can be identified, thereby effectively predicting the emotional tendency in the comment. Extensive experiments conducted by the present invention on real data sets have confirmed that the present invention has better performance than the most advanced models.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of comparative sentence recognition, and relates to a comparative sentence sentiment analysis method and system based on BERT adversarial training. Background Art

[0002] In recent years, with the rapid development of the Internet and the increasing popularity of social media, the number of online shopping users has increased year by year, and more and more users are willing to comment on products on the website. These comments carry users' feelings and suggestions about the products and contain a lot of valuable information. They can not only provide references for other consumers, but also provide improvement suggestions for merchants. However, due to problems such as network information overload, users do not have time to fully read and browse all the comments, and they may also be affected by a large amount of information interference information, and cannot make correct and effective decisions on the quality or purchase tendency of the product in a short time. In particular, there are some comparative or contrasting comments in user comments. Such comparative sentences usually express the priority preference ranking of multiple entities on a certain attribute feature. These comparative comments can not only help consumers make effective judgments quickly, but also help merchants more intuitively find the gap between them and their competitors and make corresponding improvements. Therefore, in recent years, how to perform sentiment analysis on comparative sentences in user comments and determine which comparative entity the commentator prefers has become one of the key issues that need to be urgently solved in the e-commerce and academic circles.

[0003] The existing comparative sentence recognition methods first extract the syntactic structure features of the comparative sentences from the sentences, and then use machine learning methods to supervise the extracted features. Comparative sentence sentiment analysis, on the other hand, further uses methods such as label sequence labeling (LSR), semantic role labeling, and CRF to discriminate the sentiment tendency and priority ranking of entities in comparative sentences based on comparative sentence recognition. Since the sentiment analysis task in comparative sentences needs to determine which object the reviewer prefers between the two comparative entity objects, there is an obvious logical relationship between them. However, most traditional research methods do not make full use of the deep semantics and contextual dependencies inherent in the comment text, and cannot effectively learn the logical information therein. In addition, since the public datasets for comparative sentence research are very limited, the number of comparative sentence samples in the existing sentiment analysis datasets is small, accounting for only about 13% of the total dataset. And among these comparative sentence categories, the "BETTER" category accounts for about 70% ("BETTER" means that the reviewer prefers the first comparative entity). A large number of non-comparative sentences will bring a lot of interference information to the learning of the model; at the same time, due to the small number of "WORSE" category data samples in comparative sentences, the recognition of this type of emotion has also become a major difficulty, and the classic comparative sentence sentiment analysis research method has not paid attention to the problem of the small number of comparative sentence samples in the data set. Summary of the invention

[0004] The purpose of the present invention is to solve the problems in the prior art and provide a comparative sentence sentiment analysis method and system based on BERT adversarial training. The main task of the present invention is to innovatively obtain richer semantic feature information in the comment text by combining the pre-trained language model BERT with the adversarial training process, and reduce the noise impact caused by a large number of non-comparative sentences in the data set, thereby improving the performance of comparative sentence sentiment analysis.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A comparative sentence sentiment analysis method based on BERT adversarial training includes the following steps:

[0007] Use the pre-trained language model BERT to encode the input review text and the two comparison objects in the review;

[0008] Conduct adversarial training on the encoded review text to obtain text semantic information, context dependencies, and logical relationships;

[0009] According to the text semantic information, context dependency and logical relationship, the Softmax function is used to obtain the final comparison sentence sentiment probability distribution and calculate the loss.

[0010] The above method is further improved in that:

[0011] The use of the pre-trained language model BERT to encode the input comment text and the two comparison objects in the comment includes:

[0012] The pre-trained language model BERT is used to encode the input comment text S in batch data, and the comment text S to be input and the two comparison objects in the comment are expressed in the format of "[CLS] comment text [SEP] comparison object 1, comparison object 2 [SEP]"; where the user's comment text S = {w1, w2, w3, ..., w n-1 ,w n}, comparison object in comment {C=c_obj1,c_obj2},w n is the word in the review; the hidden layer vector h of the review text using the last layer output of the pre-trained language model BERT as input i , H represents the dimension of the hidden layer after encoding, and n represents the sequence length of the input comment text.

[0013] The comment text is encoded using the double-sentence input mode of the pre-trained language model BERT.

[0014] The adversarial training of the encoded review text includes:

[0015] After each pair of batch data is encoded, a random number τ is generated. If the random number τ is less than 0.5, adversarial training is started. The random number τ is as follows:

[0016] τ=random() (1)

[0017] Generate adversarial interference according to the gradient in an iterative manner. When the first round of iteration begins, the parameter weight W of word_embedding of the original input word embedding of the pre-trained language model BERT is word Add the adversarial perturbation r obtained according to the gradient at , get the parameter weight t after adding interference in the current iteration round k :

[0018] t k =W word +r at (2)

[0019]

[0020] Among them, k is the current iteration round; θ is the parameter; G is the gradient, ||G|| is the norm of the gradient, so that the adversarial perturbation rat The direction is consistent with the gradient G;

[0021] The threshold ε is used to control the difference d between the semantics of the text after adding interference and the semantics of the original text. If the norm of the difference d is greater than the threshold ε, the truncation Chopper is used to truncate the difference:

[0022]

[0023] d=tW word (5)

[0024] Among them, g represents the current round of adversarial interference determined after passing through the truncation device;

[0025] The parameter weight of word_embedding in the current iteration round is O k for:

[0026] O k =W word +g (6)

[0027] In subsequent iterations, the parameter weight t after adding interference in the current iteration is calculated. k When , the parameter weight O of word_embedding obtained in the previous iteration k-1 Add adversarial perturbation r on the basis of at :

[0028] t k =O k-1 +r at (7)

[0029] After the iteration is completed, the newly generated word_embedding weight parameter O after adding perturbation is used in the pre-trained language model BERT k The input review text is encoded to obtain adversarial samples, and adversarial training is performed to obtain text semantic information, context dependencies, and logical relationships.

[0030] The method of using Softmax to obtain the final comparison sentence sentiment probability distribution and calculate the loss includes:

[0031] The last layer output of the pre-trained language model BERT is used as the word hidden layer h i , and use the Softmax function for prediction:

[0032] y=softmax(W p ·h i +b p ) (8)

[0033] Among them, y represents the true label, W p and b p Represents the weight matrix and bias of the fully connected layer;

[0034] Through the class weight parameter α and the simple / difficult sample adjustment factor (1-p t )Solve the problem of sample imbalance:

[0035] FL(p t )=-α(1-p t )log(p t ) (9)

[0036] Among them, FL() represents the loss function, -log(p t ) is the initial cross entropy loss function, α is the weight parameter between categories, (1-p t ) is the easy / difficult sample adjustment factor.

[0037] A comparative sentence sentiment analysis system based on BERT adversarial training, including:

[0038] The semantic information module is used to encode the input review text and the two comparison objects in the review using the pre-trained language model BERT;

[0039] Adversarial training module, used to perform adversarial training on the encoded review text to obtain text semantic information, contextual dependencies, and logical relationships;

[0040] The classification prediction module is used to obtain the final comparison sentence sentiment probability distribution and calculate the loss based on text semantic information, context dependencies and logical relationships using the Softmax function.

[0041] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0042] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention introduces the pre-trained language model BERT to encode the input comment text, effectively identifying the deep semantics and logical relations in the comparative sentence text; by introducing the adversarial training process, the interference caused by a large number of non-comparative sentences in the data set can be reduced, and the "WORSE" category text with a smaller data sample in the comparative sentence can be identified, thereby effectively predicting the emotional tendency in the comment.

[0045] The present invention uses the pre-trained language model BERT to encode the input comment text, effectively learns the deep semantic information and contextual dependencies of the comment text, and thus learns the internal logical relationship of the comparative sentence comment text. In order to enable the model to better understand the comparative relationship in the comparative sentence comment, the double-sentence input mode of the pre-trained language model BERT is used to encode the input comparative sentence comment text and the two comparative entities in the comment, and the dependency relationship between them is learned through the attention mechanism inside the pre-trained language model BERT. Extensive experiments conducted by the present invention on real data sets have confirmed that the present invention has better performance than the most advanced models.

[0046] The present invention introduces adversarial training, and trains adversarial samples and original samples at the same time, reducing the interference of a large number of non-comparative sentences in the data set on the model classifier. For some sentences with weak comparative relationships, the classifier is very easy to predict such sentences as sentences without obvious comparative relationships, that is, the "NONE" category. This module increases the model's understanding and recognition of such comment texts. And when calculating the loss, the Focal-loss loss function is used, and by introducing the α category weight parameter and (1-P t )The easy / difficult sample adjustment factor further reduces the interference problem caused by a large number of non-comparative sentences in the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 Flow chart of the method of the present invention.

[0049] Figure 2 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0053] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0054] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0055] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0056] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0057] See also Figure 1 The embodiment of the present invention discloses a comparative sentence sentiment analysis method based on BERT adversarial training, comprising the following steps:

[0058] S1, use the pre-trained language model BERT to encode the input comment text and the two comparison objects in the comment;

[0059] The input comment text S is batch-encoded using the two-sentence input mode of the pre-trained language model BERT. The comment text S to be input and the two comparison objects in the comment are represented in the format of “[CLS] comment text [SEP] comparison object 1, comparison object 2 [SEP]”; where the user’s comment text S = {w1, w2, w3, ..., w n-1 ,w n}, comparison object in comment {C=c_obj1,c_obj2},w n is the word in the review; the hidden layer vector h of the review text using the last layer output of the pre-trained language model BERT as input i , H represents the dimension of the hidden layer after encoding, and n represents the sequence length of the input comment text.

[0060] S2, conduct adversarial training on the encoded comment text to obtain text semantic information, context dependency and logical relationship; the adversarial training is specifically as follows:

[0061] After each pair of batch data is encoded, a random number τ is generated. If the random number τ is less than 0.5, adversarial training is started. The random number τ is as follows:

[0062] τ=random() (1)

[0063] Generate adversarial interference according to the gradient in an iterative manner. When the first round of iteration begins, the parameter weight W of word_embedding of the original input word embedding of the pre-trained language model BERT is word Add the adversarial perturbation r obtained according to the gradient at , get the parameter weight t after adding interference in the current iteration round k :

[0064] t k =W word +r at (2)

[0065]

[0066] Among them, k is the current iteration round; θ is the parameter; G is the gradient, ||G|| is the norm of the gradient, so that the adversarial perturbation r at The direction is consistent with the gradient G;

[0067] The threshold ε is used to control the difference d between the semantics of the text after adding interference and the semantics of the original text. If the norm of the difference d is greater than the threshold ε, the truncation Chopper is used to truncate the difference:

[0068]

[0069] d=tW word (5)

[0070] Among them, g represents the current round of adversarial interference determined after passing through the truncation device;

[0071] The parameter weight of word_embedding in the current iteration round is O k for:

[0072] O k =W word +g (6)

[0073] In subsequent iterations, the parameter weight t after adding interference in the current iteration is calculated. k When , the parameter weight O of word_embedding obtained in the previous iteration k-1 Add adversarial perturbation r on the basis of at :

[0074] t k =O k-1 +r at (7)

[0075] After the iteration is completed, the newly generated word_embedding weight parameter O after adding perturbation is used in the pre-trained language model BERT k The input review text is encoded to obtain adversarial samples, and adversarial training is performed to obtain text semantic information, context dependencies, and logical relationships.

[0076] S3, based on the text semantic information, context dependency and logical relationship, use the Softmax function to obtain the final comparison sentence sentiment probability distribution and calculate the loss. The specific method is as follows:

[0077] The last layer output of the pre-trained language model BERT is used as the word hidden layer h i , and use the Softmax function for prediction:

[0078] y=softmax(Wp ·h i +b p ) (8)

[0079] Among them, y represents the true label, W p and b p Represents the weight matrix and bias of the fully connected layer;

[0080] Through the class weight parameter α and the simple / difficult sample adjustment factor (1-p t )Solve the problem of sample imbalance:

[0081] FL(p t )=-α(1-p t )log(p t ) (9)

[0082] Among them, FL() represents the loss function, -log(p t ) is the initial cross entropy loss function, α is the weight parameter between categories (0-1 binary classification), (1-p t ) is the easy / difficult sample adjustment factor.

[0083] like Figure 2 As shown, the embodiment of the present invention discloses a comparative sentence sentiment analysis system based on BERT adversarial training, including:

[0084] The semantic information module is used to encode the input comment text and the two comparison objects in the comment using the pre-trained language model BERT; in order to learn the deep semantic features and contextual dependencies of the text and enable the model to better understand the logical relationship in the text, the pre-trained language model BERT is used to encode the input comment text. In addition, in order to enable the model to better understand the comparison relationship in the sentence, the two-sentence mode of the pre-trained language model BERT is adopted, and the input comment text and the two comparison objects in the comment are represented as "[CLS] comment text [SEP] comparison object 1, comparison object 2 [SEP]" and input into the pre-trained language model BERT for encoding.

[0085] The adversarial training module is used to perform adversarial training on the encoded comment text to obtain text semantic information, contextual dependencies and logical relationships. In order to reduce the interference to the model caused by the large data samples of non-comparative sentences and the small data samples of comparative sentences in the data set, the pre-trained language model BERT generates a random number every time a batch of data is encoded. If the random number meets the preset conditions, the adversarial training process in the adversarial training module will be executed. In the adversarial training module, adversarial interference is generated in an iterative manner according to the gradient, and the obtained adversarial interference is added to the weight parameter obtained in the previous iteration. It is worth noting that if it is the first iteration, adversarial perturbation is added based on the parameter weight of word_embedding of the original input word embedding of the pre-trained language model BERT; after the iteration is completed, the pre-trained language model BERT encodes the input comment text through the new word_embedding weight parameter, and the adversarial training process is completed.

[0086] The classification prediction module uses the Softmax function to obtain the final comparative sentence sentiment probability distribution based on the text semantic information, context dependencies, and logical relationships learned through the above process, and uses Focal-Loss to calculate the loss.

[0087] Principle of the present invention:

[0088] In view of the defects in the current sentiment analysis algorithm of comparative sentences, the present invention uses the pre-trained language model BERT to learn rich semantic information in the text, and increases the robustness of the model through adversarial training, thereby proposing a comparative sentence sentiment analysis method based on BERT adversarial training. Among them, the method mainly includes three modules, namely, a semantic information module, an adversarial training module and a classification prediction module. In the semantic module, the pre-trained language model BERT uses a double sentence mode to encode the input comment text and the comparison object, and learns richer deep semantics and contextual dependencies from the comment text, so as to better understand the logical relationship in the text. The adversarial training module adds adversarial sample training on the basis of the original sample training, reduces the noise caused by a large number of non-comparative sentences in the data set and the challenges faced by comparative sentence analysis under small sample conditions; the classification prediction module classifies and predicts the input comment text, judges the emotional tendency expressed by the commentator in the comment for the two comparison objects, and uses Focal-Loss to calculate the loss.

[0089] The present invention provides a comparative sentence sentiment analysis method based on pre-trained language model BERT adversarial training, comprising the following steps:

[0090] Step 0: Consider the user's comment text S = {w1,w2,w3,...,w n-1 ,w n} and the comparison object {C = c_obj1, c_obj2} in the comment, where w n is the word in the comment.

[0091] Step 1: Use the pre-trained language model BERT to encode the input comment text. Here, the pre-trained language model BERT's two-sentence input mode is used. The comment text to be input and the two comparison objects in the comment are represented in the format of "[CLS] comment text [SEP] comparison object 1, comparison object 2 [SEP]". Use the last layer output of the pre-trained language model BERT as the hidden layer vector representation of the input comment text. H represents the dimension of the hidden layer after encoding, and n represents the sequence length of the input comment text.

[0092] Step 2: In order to reduce the interference to the model caused by the small number of comparative sentence data samples and the large number of non-comparative sentence data samples, an adversarial training module is introduced to train adversarial samples and original samples at the same time to increase the robustness of the model. First, each time the semantic information module completes the encoding of a round of batch data, it will generate a random number τ between 0 and 1. If the random number τ is less than 0.5, adversarial training will begin.

[0093] τ=random() (1)

[0094] Step 3: Generate adversarial interference according to the gradient in an iterative manner. In the process of generating adversarial interference, only a small step is taken each time. Each iteration will limit the disturbance to a specified range. The number of iterations is η. When the first round of iterations begins, the parameter weight W of word_embedding of the original input word embedding of the pre-trained language model BERT is word Add the adversarial perturbation r obtained according to the gradient at , get the parameter weight t after adding interference in the current iteration round k :

[0095] t k =W word +r at (2)

[0096]

[0097] Where k represents the current iteration round. This is the first iteration round, so k = 1; θ represents the parameter; G is the gradient, and ||G|| is the norm of the gradient, which ensures the anti-disturbance r at The direction is consistent with the gradient.

[0098] Step 4: In order to ensure the consistency of the semantics of the text after the interference and the original text, a threshold ε is set to control the difference d between the two. If the norm of the difference d between the two is greater than the threshold ε, it is considered that the adversarial sample after the interference is no longer semantically consistent with the original data sample. In this case, a truncation (Chopper) is used to truncate the difference.

[0099]

[0100] d=tW word (5)

[0101] Step 5: Finally, the parameter weight of word_embedding in the current iteration round is O k for:

[0102] O k =W word +g (6)

[0103] Step 6: In subsequent iterations, calculate the parameter weight t after adding interference in the current iteration k When , the parameter weight O of word_embedding obtained in the previous iteration k-1 Add adversarial perturbation r on the basis of at .

[0104] t k =O k-1 +r at (7)

[0105] Step 7: Repeat steps 4 and 5 above to get the parameter weight O of word_embedding in the current iteration round. k .

[0106] Step 8: After the iteration is completed, use the newly generated word_embedding weight parameter O after adding the perturbation in the pre-trained language model BERT k Encode the input review text to obtain adversarial samples and perform adversarial training.

[0107] Step 9: Use the last layer output of the pre-trained language model BERT as the word hidden layer representation h i , and use the Softmax function to predict the probability distribution of the comparison sentence sentiment:

[0108] y=softmax(W p ·h i +b p ) (8)

[0109] Among them, Wp and b p are the weight matrices and biases of the fully connected layers.

[0110] Step 10: In order to further reduce the interference problem caused by the small number of comparative sentence data samples and the large number of non-comparative sentence data samples, the loss function of the present invention uses the Focal-loss loss function. Since there are a large number of non-comparative sentences in the data set, the number of negative samples is too large, accounting for a large part of the total loss, and most of them are easy to classify. Therefore, in this case, the optimization direction of the model is not what you want. And because the comparative sentence data samples are small, the addition of a large number of simple samples with small losses will drown out the samples with small data volumes. The Focal-Loss loss function is improved on the basis of the standard cross entropy loss function, by introducing the α category weight parameter and (1-P t )The simple / difficult sample adjustment factor solves the sample imbalance problem.

[0111] FL(p t )=-α(1-p t )log(p t ) (9)

[0112] like Figure 1 As shown, the comparative sentence sentiment analysis system based on the pre-trained language model BERT adversarial training of the present invention includes:

[0113] Example

[0114] The method of the present invention is suitable for application environments where users can understand related things through online comments and make their own decisions, such as online shopping. Users can more intuitively and quickly understand related things based on the sentiment analysis results of comparative sentences and make correct decisions in a short time. Merchants can more intuitively see user feedback on products, find deficiencies between them and other competing merchants, make corresponding improvements more accurately, and improve the performance of related applications and user experience.

[0115] The present invention is comprehensively compared with the baseline model.

[0116] BOW+XGBoost: BOW uses the bag-of-words model to represent the shallow semantic features of the text and inputs it into the gradient-enhanced XGBoost with decision trees as learners for sentiment prediction.

[0117] InferSent+XGBoost: The InferSent model operates on sentence pairs. Two sentences share one encoder and obtain their own text vector expressions. Then the vectors are concatenated, multiplied, and subtracted to extract the interactive information between vectors for the subsequent fully connected layer to obtain the sentence representation. This is then input into the XGBoost prediction result.

[0118] ELMo+LogReg: ELMo uses its pretrained BiLM to generate word embedding representations and uses logistic regression as a classifier.

[0119] LSTM: Use LSTM to encode the input text and predict the sentiment tendency.

[0120] Table 1 shows the experimental performance of the present invention under the CompSent-19 dataset. It can be seen from the experimental results that under the dataset CompSent-19, the BAT model proposed in the present invention has a significant improvement in performance compared with other baseline models. Compared with the best model before, the present invention has improved the f1 value of BETTER by 7.7%, the f1 value of WORSE by 26.4%, the f1 value of NONE by 1.5%, and the average f1 value by 11.9%. From the BOW bag of words model to the Infersent and then to the ELMO model, the feature information that can be learned when encoding sentences is gradually enriched, and the Micro-F1 value gradually increases. It is worth noting that compared with other models, the improvement of the present invention in the "WORSE" category is the most significant, which can explain that the model in this paper can better learn the deep semantic information and contextual dependencies of the input text in the semantic information module, so as to better learn the logical relationship contained in the input text. And the adversarial training module can reduce the noise introduced by the smaller data samples of comparative sentences and the larger data samples of non-comparative sentences in the data set to a certain extent.

[0121] Table 1 Experimental performance of the present invention under CompSent-19 dataset

[0122]

[0123]

[0124] Since the present invention includes multiple modules, in order to prove the importance of each module on the final result, an ablation experiment was conducted, and the results are shown in Table 2.

[0125] Table 2 Comparison of separation performance of different modules of the present invention under CompSent-19 dataset

[0126]

[0127] Only-Bert+FocalLoss uses only BERT embeddings as the word embedding representation of the sequence and predicts through softmax, with the loss function being Focal-Loss.

[0128] Glove+Transformer+Focal-Loss uses Glove as word embedding, Transformer as classifier, and FocalLoss as loss function.

[0129] Bert+Adversarial-Training+Focal-Loss uses BERT’s embeddings as the word embedding representation of the sequence, predicts through softmax, adds adversarial training on this basis, and uses Focal-Loss as the loss function.

[0130] Bert+Adversarial-Training+CELoss uses BERT embeddings as the word embedding representation of the sequence, predicts through softmax, adds adversarial training on this basis, and uses cross entropy as the loss function.

[0131] According to the ablation experiment results in Table 2, it can be found that:

[0132] Effectiveness of semantic information module: Compared with Glove+Transformer+Focal-Loss, Only-Bert+FocalLoss improves performance, with accuracy, recall, and f1 value increased by 0.7%, 1.2%, and 1.3% respectively on the dataset. This shows that using Bert in this task can obtain richer syntactic and semantic features and dependencies between contexts, and can better learn the logical information contained in the input text.

[0133] Effectiveness of the adversarial training module: Compared with Only-Bert+Focal-Loss, Bert+Adversarial-Training+FocalLoss improves accuracy, recall, and f1 by 1.6%, 1.4%, and 1.5% respectively on the dataset. This shows that training adversarial samples and original samples simultaneously in this task can help the model learn better and reduce the interference caused by a large number of non-comparative sentences.

[0134] Effectiveness of FocalLoss: Compared with Bert+Adversarial-Training+CELoss, the performance of Bert+Adversarial-Training+Focal-Loss has been improved. The accuracy, recall and f1 on the dataset have been increased by 0.8%, 0.4% and 0.8% respectively, which shows that the use of Focal-Loss in this task can further reduce the impact of a large number of non-comparative sentences in the dataset.

[0135] It can be seen that each module in the present invention has a very important contribution to the final experimental performance. Among them, the adversarial training module has the greatest contribution to improving the model effect, especially the model's recognition of the "WORSE" category text. After adding the adversarial training module, the F1 value of the "WORSE" category increased by 4.5%, which also indirectly confirms that the model can greatly reduce the interference noise brought to the model by a large number of non-comparative sentences and comparative sentences with a large number of "BETTER" categories.

[0136] An embodiment of the present invention provides a terminal device. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented, for example Figure 1 Alternatively, when the processor executes the computer program, the functions of each module in the above-mentioned device embodiments are implemented, such as the adversarial training module.

[0137] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0138] The terminal device may be a computing device such as a desktop computer, a notebook, a palmtop computer, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0139] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0140] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0141] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0142] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A comparative sentence sentiment analysis method based on BERT adversarial training, It is characterized in that The following steps are involved: Use the pre-trained language model BERT to encode the input comment text and the two comparison objects in the comment; the details are as follows: Use the pre-trained language model BERT to input the comment text Encode the batch data and enter the comment text The two comparison objects in the comment are expressed in the format of "[CLS] comment text [SEP] comparison object 1, comparison object 2 [SEP]"; where the user's comment text , the comparison object in the comment , is the word in the review; the hidden layer vector of the review text using the last layer output of the pre-trained language model BERT as input , , H represents the dimension of the hidden layer after encoding, n Indicates the sequence length of the input comment text; The encoded comment text is subjected to adversarial training to obtain text semantic information, context dependency, and logical relationship; the adversarial training of the encoded comment text is specifically as follows: After each pair of batch data is encoded, a random number is generated. τ , if the random number τ <0.5 to start adversarial training, the random number τ as follows: Generate adversarial interference according to the gradient in an iterative manner. When the first round of iteration begins, the parameter weight of word_embedding of the original input word embedding of the pre-trained language model BERT is Add adversarial perturbations based on the gradient , get the parameter weight after adding interference in the current iteration round : in, k is the current iteration round; θ is the parameter; G is the gradient, is the norm of the gradient, making the adversarial perturbation Direction and gradient G Consistency; Pass Threshold ε Control the difference between the semantics of the text after adding interference and the semantics of the original text d , if the difference d The norm of is greater than the threshold ε , then use the truncation operator Chopper to truncate the difference: in, g represents the adversarial interference of the current round determined after finally passing through the truncation device; The parameter weight of word_embedding in the current iteration for: In subsequent iterations, the parameter weights after adding interference in the current iteration are calculated. , the parameter weight of word_embedding obtained in the previous iteration Add adversarial perturbations based on : After the iteration is completed, the newly generated word_embedding weight parameters after adding perturbations are used in the pre-trained language model BERT Encode the input review text to obtain adversarial samples, perform adversarial training, and obtain text semantic information, context dependencies, and logical relationships; According to the text semantic information, context dependency and logical relationship, the Softmax function is used to obtain the final comparison sentence sentiment probability distribution and calculate the loss.

2. The comparative sentence sentiment analysis method based on BERT adversarial training according to claim 1, It is characterized in that The comment text is encoded using the double-sentence input mode of the pre-trained language model BERT.

3. The comparative sentence sentiment analysis method based on BERT adversarial training according to claim 1, It is characterized in that The method of using Softmax to obtain the final comparison sentence sentiment probability distribution and calculate the loss includes: Use the last layer output of the pre-trained language model BERT as the word hidden layer , and use the Softmax function for prediction: in, y represents the true label, and Represents the weight matrix and bias of the fully connected layer; By class weight parameter α Adjustment factor for easy / difficult samples Solve the problem of sample imbalance: in, represents the loss function, is the initial cross entropy loss function, is the weight parameter between categories, is the easy / difficult sample adjustment factor.

4. A comparative sentence sentiment analysis system based on BERT adversarial training for implementing the method of claim 1, It is characterized in that include: The semantic information module is used to encode the input review text and the two comparison objects in the review using the pre-trained language model BERT; Adversarial training module, used to perform adversarial training on the encoded review text to obtain text semantic information, contextual dependencies, and logical relationships; The classification prediction module is used to obtain the final comparison sentence sentiment probability distribution and calculate the loss based on text semantic information, context dependencies and logical relationships using the Softmax function.

5. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.