A Text Sentiment Classification Method and Device Based on Adversarial Training and TF-IDF

Abstract

The present invention discloses a text sentiment classification method and device based on adversarial training and TF-IDF. For the text sentiment classification task, research is carried out from two perspectives: generating interpretable adversarial samples and text feature extraction, to ensure that the deep neural network model improves the classification accuracy while enhancing the interpretability of adversarial samples. Regarding the problem of how to generate interpretable adversarial samples during model training, the present invention specifically proposes applying adversarial training to the BERT embedding layer and normalizing the perturbation direction of text words to the direction of existing neighboring words in the embedding space to improve the interpretability of adversarial samples. Regarding how to extract more text features, the present invention proposes using an attention mechanism and an improved TF-IDF algorithm to obtain additional text features, so that the model can draw a more comprehensive and accurate text sentiment tendency.

Description

Technical Field

[0001] The present invention relates to the field of Internet technologies, and in particular, to a text sentiment classification method and device based on adversarial training and TF-IDF. Background Art

[0002] Sentiment analysis is an important research direction in the field of natural language processing. What it focuses on is not the topic meaning of paragraphs and sentences, but the viewpoints they express. Sentiment analysis has a wide range of applications in multiple fields, from tracking the opinions of users on products or current events on social media platforms to predicting public behavior and improving decision-making, etc. With the explosive development of artificial intelligence technology, online review websites, personal blogs, and social media platforms rich in personal viewpoints are becoming increasingly popular. New opportunities and challenges have emerged in the field of sentiment analysis because people can now use information technology to seek and understand the viewpoints of others. In the era of big data with an explosion of information, a huge amount of data information is generated every minute on various social network platforms, and these social media information contain rich sentiment knowledge. Extracting people's opinions and the sentiment hidden in the text information published by users through artificial intelligence technology plays an important role in a wide range of applications, such as public opinion monitoring, predicting public behavior, and recommendation systems, etc.

[0003] With the replacement of computer software technology and hardware over time, various types of chips emerge in an endless stream, and the processing power is different from the past. Deep learning technology has gradually come into everyone's view and is widely used. This method does not require a large amount of manpower to label the text, and the deep learning technology can adaptively mine sentiment features and context sentiment information from the text. However, with just a little tiny perturbation added to the original input of the deep learning model, the deep learning model can output incorrect recognition results.

[0004] To cope with the security threats brought by adversarial attacks, existing defense methods add adversarial perturbations to the deep learning model training process. By jointly training the generated text adversarial samples and the original samples, the model is allowed to learn the adversarial samples to enhance the generalization ability and robustness of the model. In the field of natural language processing, two methods are usually used to generate adversarial samples: text-based and gradient-based.

[0005] The inventors of the present application found that the methods of the prior art have at least the following technical problems in the process of implementing the present invention:

[0006] The text-based adversarial method generates adversarial examples by replacing words or characters in the original sample. Compared with the gradient-based adversarial method, although it has higher interpretability, it lacks more attack diversity and relies more on human knowledge, thus limiting the diversity of adversarial patterns. On the contrary, in the process of model training, the gradient-based adversarial method inputs the tiny perturbations calculated by the gradient into the word embedding space and participates in the model training to optimize the model parameters. Although the method of adding perturbations to the input word embedding space improves the performance of natural language processing tasks, the generated adversarial examples may contain words that do not exist in the corpus, that is, to a certain extent, it reduces the interpretability of the generated adversarial examples.

[0007] It can be seen from this that the methods in the prior art cannot achieve improving the interpretability of the generated adversarial examples while ensuring the classification accuracy. Summary of the Invention

[0008] The purpose of the present invention is to alleviate the interpretability of generating adversarial examples in the text sentiment classification task and improve the sentiment classification accuracy. By normalizing the adversarial perturbation to the existing word embedding space, the interpretability of generating adversarial examples is improved. Aiming at the problem of how to improve the interpretability of the generated adversarial examples while ensuring the classification accuracy of the model, the present invention adds adversarial perturbations to the embedding space during the BERT fine-tuning process, and at the same time restricts the direction of the adversarial perturbation to the direction of the neighboring words of the original text, and the neighboring words of the original text are obtained by the masked language model of BERT; aiming at how to extract additional text features, the present invention takes the existing text classification information in the training set into account in the TF-IDF algorithm to extract deeper text features in the text, thereby improving the performance of the final model.

[0009] The technical solution adopted by the present invention is as follows:

[0010] In the first aspect, a text sentiment classification method based on adversarial training and TF-IDF is provided, including:

[0011] S1: Obtain the original text and divide the training set from the original text;

[0012] S2: Preprocess the training set;

[0013] S3: Construct a text sentiment classification model based on adversarial training and TF-IDF. The model includes an embedding module, a first feature extraction module, a second feature extraction module, and a prediction module. Among them, the embedding module generates adversarial text through adversarial training, and merges the adversarial text with the input text as the input word vector representation; the first feature extraction module uses a bidirectional long short-term memory network model, takes the output of the embedding module as the input to extract text features, and then uses an attention mechanism to weight the hidden states in the bidirectional long short-term memory network model, giving different weights to each hidden state; the second feature extraction module uses an improved TF-IDF algorithm to obtain the frequency weights of each word in each category for the input words, and uses a bidirectional long short-term memory network to further extract text features. The prediction module is used to obtain the classification result based on the outputs of the first feature extraction module and the second feature extraction module;

[0014] S4: Train the text sentiment classification model based on the preprocessed training set to obtain a trained text sentiment classification model;

[0015] S5: Use the trained text sentiment classification model for text sentiment classification.

[0016] In one implementation, step S2 includes:

[0017] S2.1: Obtain an array of sentiment categories according to the divided training set;

[0018] S2.2: Obtain a word library from the training set and the number of times each word appears in each sentiment category;

[0019] S2.3: Traverse the training set, convert the words into subscripts in the word library, parse each sentence, and obtain the word vector, segment vector, and position encoding vector of the sentence.

[0020] In one implementation, in the model constructed in step S3, the processing process of the embedding module includes:

[0021] Use the vector obtained after preprocessing the original text as the input of the BERT pre-training model. The vector obtained after preprocessing the original text includes the word vector, segment vector, and position encoding vector of the sentence;

[0022] Use the BERT pre-training model to predict the probability of the words in the vocabulary appearing at the corresponding positions in a sentence. Specifically: find the first K nearest neighbor word vectors of each word in a sentence, denoted as K T ;

[0023] Respectively obtain the direction representation d (t,k) between each word and the K nearest neighbor word vectors, and obtain the adversarial perturbation r t of the word vector xt :

[0024]

[0025] where \(t\) represents the vector of the \(t\)-th word in the sentence, \(k\) represents the \(k\)-th nearest neighbor word vector, \(|K|\) represents the number of nearest neighbor word vectors, and \(\alpha\) (t,k) represents the weight vector for perturbing word \(t\) to the nearest neighbor word \(k\);

[0026] The adversarial word vector is obtained by adding a perturbation vector to the word vector of the sentence and used as an adversarial sample. Its calculation method is as follows:

[0027]

[0028] where \(x\) t represents the vector of the \(t\)-th word in the sentence, \(T\) represents the number of word vectors in the sentence, and \(X\) +r is the adversarial sample, and the optimal perturbation direction is obtained by calculating the gradient:

[0029]

[0030]

[0031] where \(X\) +r is used as the input, \(Y\) represents the label, \(\Theta\) represents the model parameters, represents the loss function for a single data, \(\epsilon\) represents the threshold of the adversarial perturbation, and \(g\) t represents the gradient, and \(\nabla\) represents the concatenation of all \(g\) t connected,

[0032] The word vectors and adversarial samples obtained after preprocessing the original text are input into the BERT encoder to obtain word vector representations containing context semantics.

[0033] In one implementation, in the model constructed in step S3, the processing process of the first feature extraction module includes:

[0034] The output of the embedding module is passed into a bidirectional long short-term memory network model to extract text feature information from both directions of each word feature vector;

[0035] The attention mechanism is used to weight the hidden state at each step of the bidirectional long short-term memory network model, giving different weights to the hidden states at each moment, and taking the weighted sum of the hidden states at all moments as the final feature vector output.

[0036] In one implementation, in the model constructed in step S3, the processing process of the second feature extraction module includes:

[0037] Calculate the word w according to the following formula t The term frequency TF value of the occurrence of

[0038]

[0039] Among them, Indicates each word w t In the sentiment label C j The number of occurrences in Indicates the sentiment label C j The total number of words in, TF(w t ,C j ) is the term frequency TF value of the word w t The term frequency TF value of the occurrence of

[0040] In IDF calculation, consider the existing sentiment classification information to implement a supervised TF-IDF algorithm. Specifically: find the number of sentiment categories n containing the word w t And calculate the IDF value of the word:

[0041]

[0042] Among them, |k| represents the total number of all sentiment categories;

[0043] According to the term frequency TF value and IDF value of the word w t Calculate the TF-IDF value:

[0044] TF-IDF = TF × IDF

[0045] The TF-IDF value is used to represent the frequency weight of each word appearing in each category;

[0046] Input the feature vector represented by the TF-IDF value into the bidirectional long short-term memory network for further feature extraction to obtain the text vector representation.

[0047] In one implementation, the prediction module specifically uses the softmax function to solve the probability that the text belongs to a certain category, so as to obtain the final classification result.

[0048] In one implementation, during the training process of S4, the gradient descent method and the backpropagation method are used to propagate the error to each layer of the model, so as to adjust the parameter values of the model according to the error, and continuously iterate the training until the optimal solution is finally obtained, and the cross-entropy loss function is used to calculate the final classification loss The calculation process is expressed as:

[0049]

[0050] Among them, D represents the size of the training set, C represents the number of categories, q (i,j)Indicates whether it belongs to the current category, logp (i,j) Indicates the probability that the feature word i is predicted as category j, Indicates the loss of text adversarial attack.

[0051] Based on the same inventive concept, the second aspect of the present invention provides a text sentiment classification device based on adversarial training and TF-IDF, including:

[0052] A dataset acquisition module for acquiring the original text and partitioning the training set from the original text;

[0053] A preprocessing module for preprocessing the training set;

[0054] A model construction module for constructing a text sentiment classification model based on adversarial training and TF-IDF. The model includes an embedding module, a first feature extraction module, a second feature extraction module, and a prediction module. Among them, the embedding module uses adversarial training to generate adversarial text and merges the adversarial text with the input text as the input word vector representation; the first feature extraction module uses a bidirectional long short-term memory network model to extract text features with the output of the embedding module as the input, and then uses an attention mechanism to weight the hidden states in the bidirectional long short-term memory network model to give different weights to each hidden state; the second feature extraction module uses an improved TF-IDF algorithm to obtain the frequency weights of each word in each category for the input words, and uses a bidirectional long short-term memory network to further extract features. The prediction module is used to obtain the classification result according to the outputs of the first feature extraction module and the second feature extraction module;

[0055] A training module for training the text sentiment classification model based on the preprocessed training set to obtain a trained text sentiment classification model;

[0056] A classification module for performing text sentiment classification using the trained text sentiment classification model.

[0057] Based on the same inventive concept, the third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method described in the first aspect.

[0058] Based on the same inventive concept, the fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method described in the first aspect.

[0059] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0060] A text sentiment classification method based on adversarial training and TF-IDF provided by the present invention constructs a text sentiment classification model based on adversarial training and TF-IDF. The model includes an embedding module, a first feature extraction module, a second feature extraction module, and a prediction module. To improve the generalization ability of the model, adversarial training is added to the embedding layer of the BERT model in the embedding module. Secondly, to improve the interpretability of the generated adversarial samples, the present invention proposes to limit the adversarial perturbation to the direction of neighboring words of the original word in the corpus, where the neighboring words are obtained by BERT prediction with a masked language model head. In the second feature extraction module, the present invention improves the TF-IDF algorithm by adding known classification information to the calculation of IDF, obtains the probability of the feature word appearing in each classification category, and improves the unsupervised TF-IDF algorithm into a supervised TF-IDF algorithm; the first feature extraction module uses an attention mechanism and a Bi-LSTM model to solve the long-dependency problem in the text, and designs a dual-channel feature fusion model to further optimize the text sentiment classification model. Therefore, the method provided by the present invention can improve the interpretability of adversarial texts while ensuring classification accuracy. Description of the Drawings

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0062] Figure 1 It is a flowchart of the text sentiment classification method based on adversarial training and TF-IDF in the embodiments of the present invention;

[0063] Figure 2 It is an architecture diagram of the overall model (text sentiment classification model based on adversarial training and TF-IDF) that integrates adversarial training and the improved TF-IDF algorithm in the embodiments of the present invention;

[0064] Figure 3 It is an architecture diagram of the model that introduces adversarial training into the BERT embedding layer in the embodiments of the present invention. Detailed Embodiments

[0065] This paper studies the text sentiment classification task from two perspectives: generating explainable adversarial samples and extracting text features. This approach ensures that deep neural network models improve classification accuracy while also improving the interpretability of adversarial samples. Regarding the issue of how to generate explainable adversarial samples during model training, this paper specifically proposes applying adversarial training to the BERT embedding layer, normalizing the perturbation direction of text words to the direction of existing neighboring words in the embedding space to improve the interpretability of adversarial samples. Regarding how to extract more text features, this paper proposes using an attention mechanism and an improved TF-IDF algorithm to obtain additional text features, enabling the model to derive a more comprehensive and accurate text sentiment tendency.

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0067] Example 1

[0068] The embodiment of the present invention provides a text sentiment classification method based on adversarial training and TF-IDF, including:

[0069] S1: Get the original text and divide the training set from the original text;

[0070] S2: Preprocess the training set;

[0071] S3: Construct a text sentiment classification model based on adversarial training and TF-IDF. The model includes an embedding module, a first feature extraction module, a second feature extraction module, and a prediction module. The embedding module uses adversarial training to generate adversarial text, and merges the adversarial text with the input text as the input word vector representation; the first feature extraction module uses a bidirectional long short-term memory network model, and uses the output of the embedding module as input to extract text features. Then, the attention mechanism is used to weight the hidden states in the bidirectional long short-term memory network model, giving different weights to each hidden state; the second feature extraction module uses an improved TF-IDF algorithm for the input words to obtain the frequency weight of each word in each category, and uses a bidirectional long short-term memory network to further extract text features. The prediction module is used to obtain classification results based on the outputs of the first and second feature extraction modules;

[0072] S4: training the text sentiment classification model based on the preprocessed training set to obtain a trained text sentiment classification model;

[0073] S5: Classify the text sentiment using the trained text sentiment classification model.

[0074] Please refer to Figure 1 , which is the flowchart of the text sentiment classification method based on adversarial training and TF-IDF in the embodiments of the present invention.

[0075] Specifically, step S1 is data acquisition, and step S2 is data preprocessing, which specifically includes performing data cleaning operations on the input text. For ease of processing, in the deep neural network, the original unstructured text needs to be converted into a word vector representation.

[0076] Step S3 is model construction. Among them, the embedding module uses the BERT model to implement pre-training, and converts the preprocessed word vector plus the adversarial vector (text) into a word vector representation containing context semantics. Specifically, adversarial training is applied to the BERT embedding layer to generate adversarial text, and the adversarial text is combined with the preprocessed text as the input word vector representation to improve the robustness and generalization ability of the deep neural network model. Among them, the specific model structure diagram is as Figure 3 shown. In the processing of this embedding module, an original sequence of n words (w1, w2,..., w N ) will be converted into X = (x1, x2,..., x N ), where represents the d-dimensional vector of the i-th feature word, X represents the word vector matrix, which is used as the subsequent input, and w1, w N represent the first and N-th words in the original sequence respectively.

[0077] The first feature extraction module passes the output of the embedding module into a bidirectional long short-term memory network model to extract text information from two directions of each word feature vector, so as to be able to preserve the sequence information before and after each feature word, and further alleviate the problem that long-distance features cannot be stored in natural language processing. Then, the attention mechanism is used to weight the hidden states in the bidirectional long short-term memory network model, and different weights are given to each hidden state.

[0078] The second feature extraction module introduces known classification information in the calculation process of the traditional TF-IDF algorithm, and improves the unsupervised TF-IDF algorithm into a supervised TF-IDF algorithm. For the input words (w1, w2,..., w N ), the improved TF-IDF algorithm is used to calculate the frequency weights of each word in each category, and the bidirectional long short-term memory network is used to further extract features.

[0079] The prediction module is a fully connected layer. It takes the concatenated outputs of the first feature extraction module and the second feature extraction module as input, and then uses the softmax function to obtain the final classification result. The softmax function calculates the probability that the text belongs to a certain category.

[0080] For the specific architecture of the model, please refer to Figure 2 .

[0081] Step S4 is the training of the model. The model is trained by minimizing the model loss function. During the training process, the gradient descent method and the backpropagation method are used to propagate the error to each layer of the model, so as to adjust the parameter values of the model according to the error, and continuously iterate the training until the optimal solution is finally obtained.

[0082] Step S5 is the specific application of the model.

[0083] In one implementation, step S2 includes:

[0084] S2.1: Obtain the sentiment category array according to the divided training set;

[0085] S2.2: Obtain the word library from the training set and the number of times each word appears in each sentiment category;

[0086] S2.3: Traverse the training set, convert the words into the subscripts in the word library, parse each sentence, and obtain the word vector (Token Embeddings), segment vector (Segment Embeddings) and position encoding vector (Position Embeddings) of the sentence.

[0087] In one implementation, in the model constructed in step S3, the processing process of the embedding module includes:

[0088] Use the vector obtained after preprocessing the original text as the input of the BERT pre-training model. The vector obtained after preprocessing the original text includes the word vector, segment vector and position encoding vector of the sentence;

[0089] Use the BERT pre-training model to predict the probability that a word in the vocabulary appears at a corresponding position in a sentence. Specifically: find the top K nearest neighbor word vectors of each word in a sentence, denoted as K T ;

[0090] Respectively obtain the direction representation d (t,k) between each word and the K nearest neighbor word vectors, and obtain the total perturbation r t of the word vector x t :

[0091]

[0092] where \(t\) represents the vector of the \(t\)-th word in the sentence, \(k\) represents the \(k\)-th nearest neighbor word vector, \(|K|\) represents the number of nearest neighbor word vectors, and \(\alpha\) (t,k) represents the weight vector for perturbing word \(t\) to the nearest neighbor word \(k\);

[0093] An adversarial word vector is obtained by adding a perturbation vector to the word vector of the sentence and used as an adversarial sample. Its calculation method is as follows:

[0094]

[0095] where \(x\) t represents the vector of the \(t\)-th word in the sentence, \(r\) t represents the adversarial perturbation, \(T\) represents the number of word vectors in the sentence, and \(X\) +r , is the adversarial sample, and the optimal perturbation direction is obtained by calculating the gradient:

[0096]

[0097]

[0098] where \(X\) +r is used as the input, \(Y\) represents the label, \(\Theta\) represents the model parameters, represents the loss function of a single data, \(\varepsilon\) represents the threshold of the adversarial perturbation, and \(g\) t represents the gradient, and \(g\) represents the concatenation of all \(g\) t ;

[0099] The word vectors and adversarial samples obtained after preprocessing the original text are input into the BERT encoder to obtain word vector representations containing context semantics.

[0100] Specifically, the vector obtained after preprocessing the original text is the input of the BERT pre-trained model. Specifically: the word vector, segment vector, and position encoding vector are added together and then input into the BERT pre-trained model. The embedding module inputs the original word vectors (the word vectors obtained after preprocessing the original text) and the adversarial samples into the BERT encoder to obtain word vector representations containing context semantics, thereby improving the robustness and generalization ability of the deep neural network model.

[0101] In one implementation, in the model constructed in step S3, the processing process of the first feature extraction module includes:

[0102] The output of the embedding module is passed into a bidirectional long short-term memory network model to extract text feature information from both directions of each word feature vector;

[0103] Using the attention mechanism, the hidden state at each step of the bidirectional long short-term memory network model is weighted, different weights are given to the hidden states at each moment, and the weighted sum of the hidden states at all moments is used as the output of the final feature vector.

[0104] Through the bidirectional long short-term memory network (Bi-LSTM) model, text information can be extracted from two directions of each word feature vector, and the sequence information before and after each feature word can be preserved, thus further alleviating the problem that features with a long distance cannot be stored in natural language processing.

[0105] Through Bi-LSTM, the problem of long-term dependence can be solved to a certain extent. However, since Bi-LSTM obtains sequence information step by step, it is difficult for Bi-LSTM to preserve all useful information for long texts. Therefore, the attention mechanism is used after the Bi-LSTM model to weight the hidden state at each step and give different weights to each hidden state. Among them, in the Bi-LSTM model, only the output of the last moment is used for the feature vector. Using the attention mechanism, the weight at each moment will be calculated, and the weighted sum of the hidden states at all moments will be used as the output of the final feature vector.

[0106] In one implementation, in the model constructed in step S3, the processing process of the second feature extraction module includes:

[0107] Calculate the term frequency (TF) value of word w t using the following formula:

[0108]

[0109] where represents the number of occurrences of each word w t in the sentiment label C j , represents the total number of words in the sentiment label C j , and TF(w t , C j ) is the term frequency (TF) value of word w t :

[0110] In the IDF calculation, considering the existing sentiment classification information, a supervised TF-IDF algorithm is implemented. Specifically: calculate the number of sentiment categories n containing word w t , and calculate the IDF value of the word:

[0111]

[0112] where |k| represents the total number of all sentiment categories;

[0113] According to word w tCalculate the TF-IDF value using the term frequency (TF) value and inverse document frequency (IDF) value of the appearance frequency:

[0114] TF-IDF = TF × IDF

[0115] The TF-IDF value is used to represent the frequency weight of each word appearing in each category;

[0116] Input the feature vector represented by the TF-IDF value into a bidirectional long short-term memory network for further feature extraction to obtain the text vector representation.

[0117] Specifically, the second feature extraction module introduces known classification information into the calculation process of the traditional TF-IDF algorithm, improving the unsupervised TF-IDF algorithm into a supervised TF-IDF algorithm. For the input words (w1, w2, …, w N ), use the improved TF-IDF algorithm to calculate the frequency weight of each word appearing in each category, that is, the TF-IDF value. Input the feature vector represented by the TF-IDF value into the bidirectional long short-term memory network for model training to obtain the text vector representation.

[0118] Among them, after separately calculating the TF and IDF values, simultaneously consider the frequency of a word appearing in the sentiment category and the inverse document frequency index. That is, TF-IDF is positively correlated with the TF value of the word's appearance frequency in the sentiment category and negatively correlated with the number n of sentiment categories containing the word, so as to obtain the calculation formula of the TF-IDF value.

[0119] In one implementation, the prediction module specifically uses the softmax function to solve the probability that the text belongs to a certain category, thereby obtaining the final classification result.

[0120] In one implementation, during the training process of S4, the gradient descent method and the backpropagation method are used to propagate the error to each layer of the model, so as to adjust the parameter values of the model according to the error, and continuously iterate the training until the optimal solution is finally obtained, and the cross-entropy loss function is used to calculate the final classification loss The calculation process is expressed as:

[0121]

[0122] Where D represents the size of the training set, C represents the number of categories, q (i,j) represents whether it belongs to the current category, logp (i,j) represents the probability that the feature word i is predicted as category j, represents the loss of text confrontation.

[0123] The beneficial effects produced by the present invention are:

[0124] (1) Aiming at the problem of how to generate interpretable adversarial samples in text adversarial training, the present invention proposes to introduce adversarial training into the embedding layer of the BERT model (AT-BERT), use BERT with a masked language model head to find the neighboring words of the feature words, and standardize the direction of adversarial perturbations to the direction of the neighboring words of the feature words in the corpus, so as to ensure the sentiment classification accuracy of the model and generate interpretable adversarial samples.

[0125] (2) The attention mechanism and the bidirectional long short-term memory network (Bi-LSTM) are introduced into the AT-BERT model to solve the problem of inability to obtain long-distance information in the text sequence, assign different attention degrees to the feature words in the sentence, obtain the hidden feature information in the text, and further optimize the classification model.

[0126] (3) The known classification information in the training set is introduced into the traditional TF-IDF algorithm to implement a supervised TF-IDF algorithm, obtain additional text features from the known classification information, and jointly design a text sentiment classification model with dual-channel feature fusion with the model designed in point (2).

[0127] Embodiment 2

[0128] Based on the same inventive concept, this embodiment provides a text sentiment classification device based on adversarial training and TF-IDF, including:

[0129] A dataset acquisition module for acquiring the original text and dividing the training set from the original text;

[0130] A preprocessing module for preprocessing the training set;

[0131] A model construction module for constructing a text sentiment classification model based on adversarial training and TF-IDF. The model includes an embedding module, a first feature extraction module, a second feature extraction module, and a prediction module. Among them, the embedding module uses adversarial training to generate adversarial text and merges the adversarial text with the input text as the input word vector representation; the first feature extraction module uses a bidirectional long short-term memory network model, takes the output of the embedding module as the input to extract text features, and then uses the attention mechanism to weight the hidden states in the bidirectional long short-term memory network model to give different weights to each hidden state; the second feature extraction module uses an improved TF-IDF algorithm to calculate the frequency weights of each word in each category for the input words, and uses a bidirectional long short-term memory network to further extract features. The prediction module is used to obtain the classification result according to the outputs of the first feature extraction module and the second feature extraction module;

[0132] A training module for training the text sentiment classification model based on the preprocessed training set to obtain a trained text sentiment classification model;

[0133] A classification module for performing text sentiment classification using a trained text sentiment classification model.

[0134] Since the device introduced in the second embodiment of the present invention is the device used for implementing a text sentiment classification method based on adversarial training and TF-IDF in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of the device, so it will not be elaborated here. Any device used in the method of the first embodiment of the present invention falls within the scope of protection of the present invention.

[0135] Embodiment Three

[0136] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed, it implements the method described in Embodiment One.

[0137] Since the computer-readable storage medium introduced in the third embodiment of the present invention is the computer-readable storage medium used for implementing a text sentiment classification method based on adversarial training and TF-IDF in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of the computer-readable storage medium, so it will not be elaborated here. Any computer-readable storage medium used in the method of the first embodiment of the present invention falls within the scope of protection of the present invention.

[0138] Embodiment Four

[0139] Based on the same inventive concept, the present application also provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the above program, it implements the method in Embodiment One.

[0140] Since the computer device introduced in the fourth embodiment of the present invention is the computer device used for implementing a text sentiment classification method based on adversarial training and TF-IDF in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of the computer device, so it will not be elaborated here. Any computer device used in the method of the first embodiment of the present invention falls within the scope of protection of the present invention.

[0141] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0142] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 block or multiple blocks.

[0143] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0144] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A text sentiment classification method based on adversarial training and TF-IDF, characterized in that Including: S1: Obtain the original text and divide the training set from the original text; S2: Preprocess the training set; S3: Construct a text sentiment classification model based on adversarial training and TF-IDF. The model includes an embedding module, a first feature extraction module, a second feature extraction module, and a prediction module. Among them, the embedding module uses adversarial training to generate adversarial text, and combines the adversarial text with the input text as the input word vector representation; the first feature extraction module uses a bidirectional long short-term memory network model, takes the output of the embedding module as the input to extract text features, and then uses an attention mechanism to weight the hidden states in the bidirectional long short-term memory network model, giving different weights to each hidden state; the second feature extraction module uses an improved TF-IDF algorithm to obtain the frequency weights of each word in each category for the input words, and uses a bidirectional long short-term memory network to further extract text features. The prediction module is used to obtain the classification result according to the outputs of the first feature extraction module and the second feature extraction module. The processing process of the embedding module includes: Use the vector obtained after preprocessing the original text as the input of the BERT pre-trained model. The vector obtained after preprocessing the original text includes the word vector, segment vector, and position encoding vector of the sentence; Use the BERT pre-trained model to predict the probability of each word in the vocabulary appearing at its corresponding position in a sentence. Specifically, find the nearest neighbor word vectors for each word in a sentence, denoted as ; Calculate the directional representation between each word and the vector representations of its nearest neighbors respectively, and obtain the adversarial perturbation of the word vector : where $\mathbf{t}$ represents the vector of the $t$-th word in the sentence, $\mathbf{k}$ represents the vector of the $k$-th nearest neighbor word, and $|K|$ represents the number of nearest neighbor word vectors. represents the weight vector for perturbing word $\mathbf{t}$ to the nearest neighbor word $\mathbf{k}$; Add a perturbation vector to the word vector of the sentence to obtain an adversarial word vector as an adversarial sample. Its calculation method is: where represents the vector of the t-th word in the sentence, and T represents the number of word vectors in the sentence, is the adversarial example, and the optimal perturbation direction is obtained by calculating the gradient: Among them As the input Represents the label Represents the model parameter Represents the loss function of a single data Represents the threshold of the adversarial perturbation Represents the gradient Represents all Connection ; Input the word vector and the adversarial sample obtained after preprocessing the original text into the BERT encoder to obtain a word vector representation containing context semantics; S4: Train the text sentiment classification model based on the preprocessed training set to obtain a trained text sentiment classification model; S5: Use the trained text sentiment classification model for text sentiment classification.

2. The text sentiment classification method based on adversarial training and TF-IDF according to claim 1, characterized in that Step S2 includes: S2.1: Obtain the sentiment category array according to the divided training set; S2.2: Obtain the word library from the training set and the number of times each word appears in each sentiment category; S2.3: Traverse the training set, convert the words into the subscripts in the word library, parse each sentence, and obtain the word vector, segment vector, and position encoding vector of the sentence.

3. The text sentiment classification method based on adversarial training and TF-IDF according to claim 1, characterized in that In the model constructed in step S3, the processing process of the first feature extraction module includes: Input the output of the embedding module into a bidirectional long short-term memory network model to extract text feature information from two directions of each word feature vector; Use the attention mechanism to weight the hidden state of each step in the bidirectional long short-term memory network model, give different weights to the hidden state at each moment, and take the weighted sum of the hidden states at all moments as the final feature vector output.

4. The text sentiment classification method based on adversarial training and TF-IDF according to claim 1, wherein In the model constructed in step S3, the processing process of the second feature extraction module includes: Calculate the TF value of the frequency of occurrence of the word according to the following formula: Among them, represents the number of occurrences of each word in the sentiment label and represents the total number of words in the sentiment label The TF value of the occurrence frequency of the word is: Considering the existing sentiment classification information in IDF calculation to implement a supervised TF-IDF algorithm, specifically: calculate the number of sentiment categories containing the word , and calculate the IDF value of the word: where |k| represents the number of all sentiment categories; Calculate the TF-IDF value based on the term frequency (TF) value and inverse document frequency (IDF) value of the word : The TF-IDF value is used to represent the frequency weight of each word in each category; Input the feature vector represented by the TF-IDF value into a bidirectional long short-term memory network for further feature extraction to obtain a text vector representation.

5. The text sentiment classification method based on adversarial training and TF-IDF according to claim 1, wherein The prediction module specifically uses the softmax function to solve the probability that the text belongs to a certain category, so as to obtain the final classification result.

6. The text sentiment classification method based on adversarial training and TF-IDF according to claim 1, characterized in that During the training process of S4, the gradient descent method and the backpropagation method are used to propagate the error to each layer of the model, so as to adjust the parameter values of the model according to the error, and continuously iterate the training until the optimal solution is finally obtained, and the cross-entropy loss function is used to calculate the final classification loss , and the calculation process is expressed as: Among them represents the size of the training set, represents the number of categories, indicates whether it belongs to the current category, represents the feature word is predicted to be the category probability, represents the loss of text adversarial attack.

7. A text sentiment classification device based on adversarial training and TF-IDF, characterized in that, Including: A dataset acquisition module, configured to acquire the original text and divide the training set from the original text; A preprocessing module, configured to preprocess the training set; A model construction module, configured to construct a text sentiment classification model based on adversarial training and TF-IDF. The model includes an embedding module, a first feature extraction module, a second feature extraction module, and a prediction module. Among them, the embedding module generates adversarial text through adversarial training and combines the adversarial text with the input text as the input word vector representation; the first feature extraction module uses a bidirectional long short-term memory network model, takes the output of the embedding module as the input to extract text features, and then uses an attention mechanism to weight the hidden states in the bidirectional long short-term memory network model to give different weights to each hidden state; the second feature extraction module uses an improved TF-IDF algorithm to calculate the frequency weights of each word in each category for the input words, and uses a bidirectional long short-term memory network to further extract features. The prediction module is configured to obtain the classification result according to the outputs of the first feature extraction module and the second feature extraction module; A training module, configured to train the text sentiment classification model based on the preprocessed training set to obtain a trained text sentiment classification model. The processing process of the embedding module includes: Using the vector obtained by preprocessing the original text as the input of the BERT pre-training model. The vector obtained by preprocessing the original text includes the word vector, segment vector, and position encoding vector of the sentence; Use the BERT pre-trained model to predict the probability of each word in the vocabulary appearing at its corresponding position in a sentence. Specifically, find the first nearest neighbor word vectors for each word in a sentence, denoted as ; The direction representations between each word and the vector representations of its nearest neighbor words are obtained respectively, and the adversarial perturbation of the word vector is obtained: where \(t\) represents the vector of the \(t\)-th word in the sentence, \(k\) represents the \(k\)-th nearest neighbor word vector, and \(|K|\) represents the number of nearest neighbor word vectors. represents the weight vector for perturbing word \(t\) to the nearest neighbor word \(k\); Adding a perturbation vector to the word vector of the sentence to obtain an adversarial word vector as an adversarial sample. Its calculation method is: where represents the vector of the t-th word in the sentence, and T represents the number of word vectors in the sentence. is the adversarial example, and the optimal perturbation direction is obtained by calculating the gradient: Among them as the input represents the label represents the model parameter represents the loss function of a single data represents the threshold of the adversarial perturbation represents the gradient represents all connections ; Inputting the word vector and the adversarial sample obtained by preprocessing the original text into the BERT encoder to obtain a word vector representation containing context semantics; A classification module, configured to perform text sentiment classification using the trained text sentiment classification model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method described in any one of claims 1 to 6.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 6.

Images