A BERT short text sentiment analysis method with improved training method

Abstract

The invention discloses a BERT short text sentiment analysis method with an improved training method, which constructs a short text sentiment analysis model, including an input layer, a semantic feature extraction layer, a pooling layer, a fully connected layer and a classification output layer; collects and preprocesses a data set; encodes the input text in the input layer to obtain a word vector representation; adds disturbance to the word vector to obtain an adversarial sample; the semantic feature extraction layer extracts semantic features of the adversarial sample based on the BERT model and outputs a feature vector; after the pooling layer and the fully connected layer, Softmax is used for normalization processing to obtain the final sentiment polarity classification result; the invention adopts an adversarial training method to train the short text sentiment analysis model, solves the problem that sentiment misclassification is caused by polysemy of a Chinese word, new network words, etc., and that the traditional model cannot extract local key information while extracting context information, thereby enhancing the model robustness and alleviating the problems of poor model training efficiency and model performance degradation.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, relates to natural language processing using artificial intelligence, and particularly relates to a BERT short text sentiment analysis method with an improved training method. Background Art

[0002] With the rapid development of information technology and the rise of social networks, more and more netizens are expressing their views and opinions on the Internet. Internet platforms such as Weibo, Facebook, and Twitter generate massive amounts of text comments every day. Mining and analyzing their potential emotional tendencies is of great value in assisting governments, businesses, and other groups in making decisions.

[0003] The core of sentiment analysis technology lies in the construction of sentiment classification models. Traditional sentiment analysis methods include sentiment dictionary-based methods and machine learning methods. Sentiment analysis methods based on sentiment dictionaries refer to matching preprocessed words with words in sentiment dictionaries, and then calculating sentiment scores and judging sentiment polarity based on the degree of word matching. This method mainly relies on the construction of sentiment dictionaries. Due to the rapid development of the Internet at this stage, the speed of information update is accelerated, and new words are constantly generated on the Internet. If the dictionary cannot be updated in time, it will cause misjudgment of sentiment tendencies and lead to deviations in analysis results. Therefore, it is necessary to continuously expand the sentiment dictionary to meet the needs of sentiment analysis. The expansion of the sentiment dictionary requires a lot of time and resources.

[0004] Building a model using machine learning methods refers to a method of training a model with specific data and predicting results based on the model. Commonly used models include Naive Bayes NB, Support Vector Machine SVM, Maximum Entropy ME, etc. Building a sentiment analysis model based on machine learning uses a large amount of labeled or unlabeled corpus, uses statistical machine learning algorithms, extracts features, and finally outputs the sentiment polarity judgment of the text. Although the use of machine learning methods has reduced the workload of manual processing to a certain extent, it consumes a lot of time and energy in manually constructing features, and the model generalization ability is poor.

[0005] Sentiment analysis methods based on deep learning are currently the most widely used. Deep learning can extract deep features of text through text representation, learn text features well, and improve classification accuracy. Regarding text representation, Word2Vec or Glove is used to train word vectors, and then the word vectors are used as inputs to the neural network model. Deep learning methods are used to learn deep semantic features. Commonly used deep learning models include Text-CNN, RNN, and LSTM. However, the word vectors trained by Word2VEc and Glove models are static word vectors. Static word vectors can convert the characteristic words of a sample into vectors of the same dimension. The word vectors trained are fixed, that is, each word has only one corresponding digital vector. Therefore, it is difficult to solve the phenomenon of polysemy of Chinese words and new network words with rich semantic expressions. This makes the traditional word embedding method very unsuitable for the current network comment sentiment classification task.

[0006] BERT supports a dynamic word vector model that can be transferred. During the training phase, the entire BERT model is transferred to generate word vectors for text in specific scenarios. This method effectively solves the problem that traditional static word vectors cannot solve the polysemy problem of Chinese words and the problem of new words constantly appearing on the network.

[0007] Deep learning models are vulnerable to carefully designed input samples, which are called adversarial samples. Adversarial samples refer to adding some perturbations that are imperceptible to the original samples, causing the model to make wrong judgments. Dealing with the model vulnerability caused by unknown adversarial samples and enhancing the robustness of the model has become an important task today.

[0008] In multi-classification problems, the target labels of a class-balanced dataset are evenly distributed. If the number of samples of a certain class of targets is much larger than that of other classes, the dataset can be considered an unbalanced dataset. This imbalance will lead to two problems: low training efficiency and degraded model performance. Summary of the invention

[0009] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a BERT short text sentiment analysis method with an improved training method. The method adopts the generation of adversarial samples for adversarial training to enhance the robustness and generalization ability of the sentiment analysis model, and adopts a deep learning method to automatically extract features from the text to solve the problem that dictionary construction and feature engineering in traditional sentiment analysis models require a lot of manual participation. The Focal loss loss function is used to replace the cross entropy loss function commonly used in classification problems, and further solves the problems of low training efficiency and model performance degradation caused by data imbalance.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is:

[0011] A BERT short text sentiment analysis method with improved training method, including:

[0012] Step 1: construct a short text sentiment analysis model, which includes an input layer, a semantic feature extraction layer, a pooling layer, a fully connected layer and a classification output layer;

[0013] Step 2: Collect a data set, which is a public data set or a data set constructed by collecting short text comment data. When it is a self-constructed data set, each collected short text data is labeled with its sentiment polarity label, and the sentiment polarity includes six emotions: happy, sad, angry, surprise, neutral, and fear.

[0014] Step 3: Preprocess the short text data in the dataset, remove characters that are useless for sentiment analysis, and convert non-simplified Chinese content into simplified Chinese to obtain a cleaned short text dataset, which is convenient for the subsequent construction of a short text sentiment analysis model;

[0015] Step 4: At the input layer, the input simplified Chinese text is first segmented, and then the segmented text is encoded to obtain the word vector representation of the input text. The word vector is obtained by adding the character vector, text vector, and position vector.

[0016] Step 5: Add a perturbation to the word vector to obtain an adversarial sample;

[0017] Step 6: The semantic feature extraction layer extracts semantic features of the adversarial sample based on the BERT model, outputs feature vectors, and obtains feature vector matrix B∈R s*e , where s is the length of the text in words and e is the dimension of the feature vector;

[0018] Step 7: The pooling layer performs pooling processing on the feature vector, reduces the dimension, removes redundant information, compresses the features, simplifies the network complexity, and outputs the pooled feature vector to the fully connected layer;

[0019] Step 8: The fully connected layer extracts semantic features based on the pooled feature vector to capture emotional information, and finally uses the Softmax classification function to normalize the feature vector output by the fully connected layer to obtain the final emotional polarity classification result;

[0020] Step 9: Train the short text sentiment analysis model. The adversarial training process is as follows:

[0021] Step 1, calculate the loss value of the word vector x along the forward propagation of the model, and then back-propagate to obtain the gradient g of the loss function with respect to the input word vector x. fθ () is the neural network function to obtain the predicted value, y is the true sentiment polarity label of the sample, and L() is the loss function;

[0022] Step 2, according to the formula Calculate the disturbance r adv , where ∈ represents the perturbation space;

[0023] Step 3, perturb r adv Add to the word vector x, that is, x+r adv , calculate x+r adv The loss value is propagated forward along the model, and then the loss function is back-propagated to obtain the loss function about x+r adv The gradient g′ of the loss function is continuously iterated to find a gradient g′ that makes the loss function about x+r adv The maximum loss value with the true emotional polarity label y, at this time r adv That is the optimal perturbation;

[0024] Step 4: Add the optimal perturbation to the word vector. That is, when the perturbation is fixed, train the model so that the loss function has the minimum loss value with respect to the true sentiment polarity label y under the input sample. When the loss value of the loss function tends to be stable in two consecutive iterations, end the training process of the model and obtain the short text sentiment analysis model, which can be used to perform short text sentiment analysis.

[0025] In one embodiment, the method of word segmentation and encoding in step 4 is:

[0026] Word segmentation is performed through the WordPiece model, and single words are directly used as the basic unit of text, that is, one character is one word. Each character in the text is converted into a one-dimensional word vector according to the query word vector table. The value of the text vector is automatically learned during the model training process, which is used to characterize the global semantic information of the text and integrate it with the semantic information of single words. A different vector is attached to the characters in different positions to distinguish them. Finally, the word vector, text vector and position vector are added to get the word vector representation of the input text.

[0027] In one embodiment, in step 5, the perturbation added to the word vector is the optimal perturbation.

[0028] In one embodiment, in step 6, the BERT model is constructed using a deep bidirectional Transformer encoder to maximize the use of context information in structure; the Transformer encoder includes word vectors and position encoding, a multi-head attention mechanism, residual connections and layer normalization, and a feedforward network;

[0029] The word vector and position encoding are used to provide the position information of each word in the short text, so that the dependency and temporal relationship of the words in the short text can be identified;

[0030] Using the multi-head self-attention mechanism, the relationship between each word in the short text and the rest of the words in the sentence is calculated, so that each word vector contains the information of all word vectors in the short text;

[0031] The word vector obtained by using the multi-head self-attention mechanism is input into the feed-forward network, which has two layers, the first layer is an activation function ReLU, and the second layer is a linear activation function.

[0032] In one embodiment, all output results of the last layer Transformer of the BERT model, that is, the feature vector matrix B∈R s*e .

[0033] In one embodiment, in step 7, the pooling layer performs a pooling operation on the feature vector using maximum-average pooling. The pooling layer reduces the dimension of the feature vector, removes redundant information, compresses features, and simplifies network complexity to alleviate overfitting, reduce the amount of calculation, and reduce memory consumption.

[0034] In one embodiment, in step 9, the hyperparameters of the short text sentiment analysis model are adjusted by a multiple parameter adjustment method, and the Dropout strategy and L2 regularization are used during parameter adjustment to avoid the overfitting problem of the model.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention improves the training method and uses the FGM algorithm to generate adversarial samples for adversarial training, which greatly enhances the robustness and generalization ability of the model. The BERT pre-training model is used to solve the model misclassification problem caused by polysemy in Chinese and new network words. The use of the pre-training model reduces the time and difficulty of training the deep learning model from scratch. On the basis of the original BERT, the addition of a pooling layer improves the overfitting problem that occurs when the BERT model is directly applied to sentiment analysis tasks. The focal loss function is used to iterate the model, which to a certain extent solves the problems of low training efficiency and model performance degradation caused by imbalanced data categories. Compared with the traditional sentiment analysis model, this sentiment analysis model has the advantages of strong robustness, high accuracy and easy training. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The present invention is a flow chart of the method.

[0038] Figure 2 This is the BERT model structure.

[0039] Figure 3 It is the Transformer encoder structure. DETAILED DESCRIPTION

[0040] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0041] like Figure 1 As shown, the present invention is a BERT short text sentiment analysis method with an improved training method, comprising the following steps:

[0042] Step 1: Build a short text sentiment analysis model, referred to as the BERT-AdvFL model.

[0043] The short text sentiment analysis model of the present invention mainly includes an input layer, a semantic feature extraction layer, a pooling layer, a fully connected layer and a classification output layer.

[0044] Step 2: Collect the dataset.

[0045] The dataset of the present invention can be a public dataset or a dataset constructed by collecting short text data. When it is a self-constructed dataset, each collected short text data is labeled with its emotional polarity label, and the emotional polarity includes six emotions: happy, sad, angry, surprise, neutral, and fear.

[0046] Step 3: Preprocess the short text data in the dataset.

[0047] Taking simplified Chinese as an example, data preprocessing mainly involves removing characters that are useless for sentiment analysis and converting non-simplified Chinese content into simplified Chinese to obtain a cleaned short text dataset, which is convenient for the subsequent construction of a short text sentiment analysis model.

[0048] Step 4: Taking simplified Chinese as an example, for a short text data (i.e. simplified Chinese text), at the input layer, it is first segmented, and then the segmented text is encoded to obtain the word vector representation of the input text. The word vector is obtained by adding the character vector, text vector, and position vector.

[0049] Specifically, the word segmentation and encoding method of the present invention is:

[0050] Word segmentation is performed through the WordPiece model, and single words are directly used as the basic unit of text, that is, one word is one word. Each word in the text is converted into a one-dimensional word vector as the input of the model according to the query word vector table. In addition to the word vector, the model input also includes a text vector and a position vector. The value of the text vector is automatically learned during the model training process, which is used to characterize the global semantic information of the text and integrate it with the semantic information of single words. Since the semantic information carried by words appearing at different positions in the text is different, a different position vector is attached to the words at different positions for distinction. Finally, the word vector, text vector, and position vector are added to obtain the word vector representation of the input text.

[0051] The present invention adopts the context-dependent dynamic byte encoding model WordPiece, which can not only dynamically encode the semantics of a word in different contexts in the input text, but also encode the logical relationship between each sentence in the input text.

[0052] Step 5: Add a perturbation to the word vector to obtain an adversarial example.

[0053] The specific method of obtaining the perturbation is introduced in step 9. In this step, the ultimate goal is to add the so-called optimal perturbation to the word vector.

[0054] Step 6: The semantic feature extraction layer extracts semantic features of adversarial samples based on the BERT pre-trained model to obtain feature vectors, and finally outputs the feature vector matrix B∈R s*e , where s is the length of the text in words and e is the dimension of the feature vector.

[0055] The present invention adopts the BERT pre-trained model of context-dependent sentence vector-level encoding, which can accurately quantify the meaning of the same word in different contexts in a short text, and can also encode the relationship between sentences in the text.

[0056] refer to Figure 2 In this invention, the BERT pre-training model is constructed using a deep bidirectional Transformer encoder. The BERT pre-training model used in this invention is its Base version, which has a total of 12 layers of Transformers. It is a multi-layer bidirectional decoder composed of Transformer encoders as basic units, which maximizes the use of context information in structure. All output results of the last layer of Transformer in the BERT model, that is, the feature vector matrix B∈R s*e .

[0057] refer to Figure 3, Transformer encoder, including word vector and position encoding, multi-head attention mechanism, residual connection and layer normalization and feedforward network. Among them, the word vector and position encoding provide the position information of each word in the short text, so that the dependency and temporal relationship of the word in the short text can be identified; and based on the multi-head self-attention mechanism, the relationship between each word in the short text and the rest of the words in the sentence is calculated, so that each word vector contains the information of all word vectors in the short text; in the short text, the multi-head self-attention mechanism is used to obtain a vector matrix, and each word vector in the vector matrix contains the information of other word vectors because it is processed by the multi-head self-attention mechanism. The obtained word vector is input into the feedforward network to increase the generalization ability. The feedforward network has two layers, the first layer is the activation function ReLU, and the second layer is a linear activation function.

[0058] In order to solve the problem of gradient disappearance in traditional deep learning and accelerate model training, the Transformer encoder also uses residual connection and layer normalization methods. Layer standardization speeds up model training and accelerates model convergence by normalizing the hidden layers in the neural network to a standard normal distribution; residual connection is used to solve the problems of gradient disappearance and network degradation.

[0059] Step 7: The pooling layer performs pooling on the feature vector, reduces the dimension, removes redundant information, compresses the features, simplifies the network complexity, and outputs the pooled feature vector to the fully connected layer.

[0060] In the present invention, the pooling layer uses the maximum-average pooling to perform a pooling operation on the feature vector. The mean and maximum values ​​are calculated along the text length and the embedding dimension respectively, and then the mean and maximum values ​​are cascaded into a vector, and the hidden sequence is converted to a vector. The pooling layer reduces the dimension of the feature vector, removes redundant information, compresses features, and simplifies the network complexity to alleviate overfitting, reduce the amount of calculation, and reduce memory consumption.

[0061] Step 8: The fully connected layer extracts semantic features based on the pooled feature vector to capture emotional information. Finally, the Softmax classification function is used to normalize the feature vector output by the fully connected layer to obtain the final emotional polarity classification result.

[0062] Step 9: Use adversarial training to train the short text sentiment analysis model.

[0063] Adversarial training is a method of defending against adversarial attacks. The idea is to add the generated adversarial samples to the training set, perform data enhancement, and let the model learn the adversarial samples first during training. During the adversarial training process, the samples will be mixed with some small perturbations (the changes are small, but they are likely to cause misclassification), and then the neural network will adapt to this change, so that it is robust to adversarial samples. The essence of adversarial training is to perform two gradient updates in one step. First, do a gradient ascent to find the best perturbation to maximize the loss value; then do a gradient descent to find the best model parameters to minimize the loss value. In the present invention, the model parameters include at least the learning rate, the maximum length of the input text, and the number of training rounds.

[0064] Adversarial training can be summarized as the following maximum-minimization formula:

[0065]

[0066] Among them, the maximization formula The purpose is to find the perturbation that maximizes the loss function, that is, the added perturbation should confuse the neural network as much as possible. x represents the word vector representation of the input sample, r adv represents the perturbation added to x, ∈ represents the perturbation space, f θ () is the neural network function, through which the predicted value is obtained, y is the true emotional polarity label of the sample, L() is the loss function, L(f θ (x+r adv ),y) means superimposing a perturbation r on the word vector x adv , and then through the neural network function f θ (), the loss compared with y.

[0067] The present invention adopts the FGM algorithm to calculate the disturbance. FGM adopts L2 normalization. L2 normalization is to divide the value of each dimension of the gradient by the L2 norm of the gradient. The formula is expressed as in That is, the gradient of the loss function L() with respect to the input word vector x.

[0068] Minimize the formula min θ E (x,y) It means optimizing the neural network so that when the disturbance is fixed, the training model minimizes the loss on the training data, that is, the model has a certain robustness to adapt to the disturbance.

[0069] The adversarial training process of the present invention can be described as:

[0070] Step 1, calculate the loss value of the word vector x along the forward propagation of the model, and then back-propagate to obtain the gradient g of the loss function with respect to the input word vector x;

[0071] Step 2, calculate the disturbance r adv ;

[0072] Step 3, perturb r adv Add to the word vector x, that is, x+r adv , calculate x+r adv The loss value is propagated forward along the model, and then the loss function is back-propagated to obtain the loss function about x+r adv The gradient g′ of the adversarial sample is accumulated on the gradient of the original word vector x because back propagation is performed again after step 1. We keep iterating and find a solution that makes the loss function about x+r adv The maximum loss value with the true emotional polarity label y, at this time r adv That is the optimal perturbation;

[0073] Step 4: Add the optimal perturbation to the word vector. That is, when the perturbation is fixed, train the model so that the loss function has the minimum loss value with respect to the true sentiment polarity label y under the input sample. When the loss value of the loss function tends to be stable in two consecutive iterations, end the training process of the model and obtain the short text sentiment analysis model, which can be used to perform short text sentiment analysis.

[0074] The present invention can find the optimal disturbance by repeating step 2 and step 3 continuously. The optimal disturbance is obtained by the formula It means that through continuous forward calculation and back propagation, a perturbation that maximizes the loss is found. The perturbation at this time is called the optimal perturbation, and this process is called attacking the word vector. This process can be understood as: in the first round, a gradient is calculated to obtain a perturbation, and this perturbation is added to the word vector. This iterative process is continuously performed to find the optimal perturbation that maximizes the loss.

[0075] The present invention uses the focal loss function to calculate the loss value to measure the gap between the true probability distribution of the emotion label and the predicted probability distribution. The calculation formula is as follows:

[0076] FL(p t )=-α(1-p t ) γ log(p t )

[0077] In this loss function, a modulation factor (1-p t ) γ, γ∈[0,5] is the focus parameter. Different values ​​of γ have different effects on the results. When γ=0, FL=CE, which is equal to the traditional cross entropy function. When γ>0, the relative loss value of simple samples (samples with large errors from the true label when predicted) is reduced, and attention is focused on difficult samples and misclassified samples. Therefore, during the training process, only difficult samples (samples with small errors from the true label when predicted) are trained, and simple samples are trained less. α is used as a balancing weight to control the shared weight of positive and negative samples to the total loss and adjust the scaling ratio, α∈[0,1].

[0078] Based on this, Focal loss effectively alleviates the problems of low training efficiency and model performance degradation caused by data imbalance compared to the cross entropy loss function commonly used in classification problems. Regardless of which category has less data, it is easier to make mistakes in the actual training process due to the small number of samples, the category features are not learned enough, the confidence level is also low, and the loss increases accordingly. At the same time, simple samples are gradually discarded during the learning process, so that the difficult samples of each category are left, which can achieve the same training optimization purpose.

[0079] Furthermore, in the adversarial training, the present invention can use multiple parameter adjustments to adjust the hyperparameters of the model, and use the Dropout strategy and L2 regularization during parameter adjustment to avoid the overfitting problem of the model.

[0080] Furthermore, in the adversarial training, the present invention can use the Adam optimizer to update the gradient. Adam is an extension of the gradient descent optimization algorithm and can be used to correct the learning rate and the gradient.

[0081] Furthermore, in the adversarial training, the present invention can adopt the Warmup strategy training. Warmup is a learning rate optimization method. It uses a small learning rate when the model starts training, and then uses a predetermined learning rate after a certain number of training steps. A small learning rate is used when the model is close to the convergence point. Such a learning rate setting can prevent the model from overfitting when it starts training.

[0082] When verifying the performance of the trained model, accuracy, precision, recall rate, and F1 value can be used as indicators.

[0083] In one embodiment of the present invention, taking the sentiment analysis of microblog comments on Chinese websites as an example, in the data preprocessing stage, user IDs, forwarding tags, URLs, @ and other symbols and links that are not related to the sentiment comments are removed; non-simplified Chinese words such as traditional Chinese and English words in the comments are converted into simplified Chinese words, and emoji expressions are converted into text output.

[0084] According to the different sentence structures of Weibo comment texts, they are divided into short-sentence comment texts and long-sentence comment texts. For short-sentence comment texts, the MLM method is used to build a language model. By randomly covering or replacing 15% of any Chinese characters in the comment text, the model is allowed to predict the covered or replaced parts by understanding the context. The replacement methods include: 80% probability of replacement with [MASK], such as I love my hometown Xi'an->I love my hometown Xi[MASK]; 10% probability of being replaced with another character, such as I love my hometown Xi'an->I love my hometown Xijing; 10% probability of keeping the original content unchanged, such as I love my hometown Xi'an->I love my hometown Xi'an. For long-sentence comment texts, in addition to MLM, [CLS] and [SEP] are added to the semantic logic of the comment text to determine the starting position of the upper and lower sentences, and the context-related and context-independent comment texts are input as the input representation layer in a 1:1 ratio, so that the model can understand the relationship between text sentences.

[0085] According to the adversarial training formula and gradient, the perturbation r is calculated and added to the word vector. The final input is the BERT input word vector plus the perturbation.

[0086] The dataset used is the Weibo comment text crawled based on the COVID-19 related topics. After manual annotation, 27,768 training sets, 2,000 validation sets, and 5,000 test sets were obtained. The comparison experiment of the model obtained by training the training set and testing the test set is shown in the table. The evaluation indicators are the accuracy ACC and the comprehensive evaluation indicator F1 value. The following table compares the analysis results of the BERT-adv model of the present invention with the existing BIGRU model and BILSTM model.

[0087]

[0088]

[0089] As can be seen from the table, the accuracy and comprehensive evaluation index of the present invention are much higher than those of existing models. It can be seen that the improved training method of the present invention effectively improves the robustness of the model in the face of malicious adversarial samples, reduces the occurrence of overfitting, and improves the model's prediction ability for unknown samples. Compared with the traditional short text sentiment analysis model, this model effectively solves the sentiment misclassification caused by polysemy of Chinese words, new network words, etc., and solves the problem that traditional models cannot effectively extract local key information while extracting contextual information, thereby enhancing the robustness of the model. At the same time, it alleviates the problems of poor model training efficiency and model performance degradation caused by data category imbalance to a certain extent, and has the advantages of high accuracy and easy training.

[0090] The above disclosure is only a preferred specific embodiment of the present invention, but the embodiments of the present invention are not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A BERT short text sentiment analysis method with an improved training method, characterized in that: include: Step 1: construct a short text sentiment analysis model, which includes an input layer, a semantic feature extraction layer, a pooling layer, a fully connected layer and a classification output layer; Step 2: Collect a data set, which is a public data set or a data set constructed by collecting short text comment data. When it is a self-constructed data set, each collected short text data is labeled with its sentiment polarity label, and the sentiment polarity includes six emotions: happy, sad, angry, surprise, neutral, and fear. Step 3: Preprocess the short text data in the dataset, remove characters that are useless for sentiment analysis, and convert non-simplified Chinese content into simplified Chinese to obtain a cleaned short text dataset, which is convenient for the subsequent construction of a short text sentiment analysis model; Step 4: At the input layer, the input simplified Chinese text is first segmented, and then the segmented text is encoded to obtain the word vector representation of the input text. The word vector is obtained by adding the character vector, text vector, and position vector. Step 5: Add a perturbation to the word vector to obtain an adversarial sample; Step 6: The semantic feature extraction layer extracts semantic features of the adversarial sample based on the BERT model, outputs feature vectors, and obtains feature vector matrix B∈R s*e , where s is the length of the text in words, and e is the dimension of the feature vector; the BERT model is constructed using a deep bidirectional Transformer encoder, which maximizes the use of context information in structure; the Transformer encoder includes word vectors and position encoding, multi-head attention mechanism, residual connection and layer normalization, and feedforward network; The word vector and position encoding are used to provide the position information of each word in the short text, so that the dependency and temporal relationship of the words in the short text can be identified; Using the multi-head self-attention mechanism, the relationship between each word in the short text and the rest of the words in the sentence is calculated, so that each word vector contains the information of all word vectors in the short text; Input the word vector obtained by using the multi-head self-attention mechanism into the feed-forward network, wherein the feed-forward network has two layers, the first layer is an activation function ReLU, and the second layer is a linear activation function; Step 7: The pooling layer performs pooling processing on the feature vector, reduces the dimension, removes redundant information, compresses the features, simplifies the network complexity, and outputs the pooled feature vector to the fully connected layer; Step 8: The fully connected layer extracts semantic features based on the pooled feature vector to capture emotional information, and finally uses the Softmax classification function to normalize the feature vector output by the fully connected layer to obtain the final emotional polarity classification result; Step 9: Train the short text sentiment analysis model. The adversarial training process is as follows: Step 1, calculate the loss value of the word vector x along the forward propagation of the model, and then back-propagate to obtain the gradient g of the loss function with respect to the input word vector x. f θ () is the neural network function to obtain the predicted value, y is the true sentiment polarity label of the sample, and L( ) is the loss function; Step 2, according to the formula Calculate the disturbance r adv , where ∈ represents the perturbation space; Step 3, perturb r adv Add to the word vector x, that is, x+r adv , calculate x+r adv The loss value is propagated forward along the model, and then the loss function is back-propagated to obtain the loss function about x+r adv The gradient g' of the loss function is continuously iterated to find a gradient g' that makes the loss function about x+r adv The maximum loss value with the true emotional polarity label y, at this time r adv That is the optimal perturbation; Step 4: Add the optimal perturbation to the word vector. That is, when the perturbation is fixed, train the model so that the loss function has the minimum loss value with respect to the true sentiment polarity label y under the input sample. When the loss value of the loss function tends to be stable in two consecutive iterations, end the training process of the model and obtain the short text sentiment analysis model, which can be used to perform short text sentiment analysis.

2. The BERT short text sentiment analysis method with improved training mode according to claim 1, characterized in that: In step 4, the word segmentation and encoding method is: Word segmentation is performed through the WordPiece model, and single words are directly used as the basic unit of text, that is, one character is one word. Each character in the text is converted into a one-dimensional word vector according to the query word vector table. The value of the text vector is automatically learned during the model training process, which is used to characterize the global semantic information of the text and integrate it with the semantic information of single words. A different vector is attached to the characters in different positions to distinguish them. Finally, the word vector, text vector and position vector are added to get the word vector representation of the input text.

3. The BERT short text sentiment analysis method with improved training mode according to claim 1 is characterized in that: In step 5, the disturbance added to the word vector is the optimal disturbance.

4. The BERT short text sentiment analysis method with improved training mode according to claim 1, characterized in that: All output results of the last layer Transformer of the BERT model, that is, the feature vector matrix B∈R s*e .

5. The BERT short text sentiment analysis method with improved training mode according to claim 1, characterized in that: In step 7, the pooling layer performs a pooling operation on the feature vector using maximum-average pooling.

6. The BERT short text sentiment analysis method with improved training mode according to claim 1, characterized in that: In step 9, the hyperparameters of the short text sentiment analysis model are adjusted by multiple parameter adjustment methods, and the Dropout strategy and L2 regularization are used during parameter adjustment to avoid the overfitting problem of the model.

Images