Text sentiment classification method integrated with novel adversarial strategy and enhanced feature extraction

Abstract

The invention belongs to the technical field of natural language processing, and particularly relates to a text sentiment classification method integrated with a novel adversarial strategy and enhanced feature extraction, which comprises the following steps of: carrying out preprocessing and word segmentation operation on a text; constructing a model; the embedding learning layer obtains word embedding vectors of a text through BERT and BiLSTM; the enhanced feature extraction layer performs weighted fusion on a local attention mechanism and a TextCNN model; the classification layer performs emotion polarity classification according to the extracted feature vectors to obtain a final classification result; and carrying out data enhancement by adopting a large-batch adversarial training strategy. According to the method, a local attention mechanism and TextCNN weighting are combined, and emotion features of different granularities in a text are fully mined. The local attention mechanism can identify key information in a text and endow the key information with a higher weight, and the TextCNN can capture local features and a context dependency relationship, so that compared with an existing method, the method has more advantages in the aspect of feature extraction, and text representation with more semantic representativeness is generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a text sentiment classification method incorporating a novel adversarial strategy and enhanced feature extraction. Background Art

[0002] In recent years, with the continuous development of deep learning, various excellent models have been proposed, which greatly improved the accuracy of successful sentiment classification.

[0003] Early sentiment analysis mainly relied on pre-built sentiment dictionaries, which contain sentiment-oriented words and their corresponding sentiment polarity labels, and judged the sentiment state of the entire text through the sentiment orientation of these words. In 2016, William L. et al. proposed the SENTPROP framework, which combines domain-specific word embedding with a label propagation algorithm, and uses a small number of seed words to effectively guide the construction of domain-specific sentiment dictionaries. In the same year, Zhiyang Teng et al. proposed a context-sensitive sentiment dictionary method, which uses a weighted sum model and BiLSTM to calculate the weight of each sentiment word and the sentence-level sentiment deviation score. These methods can effectively capture the unstructured features of text and facilitate analysis and understanding. However, methods based on sentiment dictionaries also have many limitations, such as difficulty in identifying new words and Internet terms, over-reliance on dictionary resources, and poor performance in cross-domain and cross-language sentiment analysis.

[0004] With the continuous development of machine learning technology, the rise of deep learning has brought revolutionary changes to sentiment analysis. In 2018, Devlin et al. proposed the landmark pre-trained language model BERT, which pre-trained deep bidirectional representations by jointly considering the contextual information of each layer, verifying the effectiveness of language model pre-training in improving the performance of various natural language processing tasks. Since then, improved models based on BERT have continued to emerge. For example, RoBERTa further optimizes the pre-training process of BERT by introducing dynamic masks, larger training data sets, and larger batch sizes. ALBERT reduces the number of parameters of the model through cross-layer parameter sharing and embedding layer parameter decomposition, while maintaining BERT's bidirectional context understanding ability. ELECTRA uses a generator and a discriminator to perform pre-training by replacing the tag detection task. SpanBERT improves the model's understanding of sentence structure by focusing on question-answering and relation extraction tasks through improved pre-training tasks. DistilBERT reduces the model size through knowledge distillation technology while maintaining the performance of BERT. Although BERT and its variants have achieved outstanding results in the field of natural language processing, their decision-making process lacks transparency, model predictions are difficult to interpret, and their performance is highly dependent on the quality of pre-training data, which limits their application in specific fields. Summary of the invention

[0005] In response to the above technical problems, the present invention provides a text sentiment classification method that incorporates a new adversarial strategy and enhanced feature extraction, aiming to improve the recognition ability and robustness of the sentiment classification model for complex text sentiment information by introducing adversarial training strategies and multi-layer feature extraction networks.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] The text sentiment classification method incorporating a novel adversarial strategy and enhanced feature extraction includes the following steps:

[0008] S1. Preprocess and segment the text;

[0009] S2. Build the model. The model is divided into three layers: embedding learning layer, enhanced feature extraction layer and classification layer;

[0010] S3, the embedding learning layer obtains the word embedding vector of the text through BERT and BiLSTM;

[0011] S4, the enhanced feature extraction layer performs weighted fusion of the local attention mechanism and the TextCNN model to further enhance the feature extraction effect;

[0012] S5, the classification layer classifies the sentiment polarity according to the extracted feature vector to obtain the final classification result;

[0013] S6. During the model training process, a large batch adversarial training strategy is used for data augmentation to improve the robustness and generalization ability of the model. Since Dropout is enabled by default during the fine-tuning process of the BERT pre-trained model, measures need to be taken to make the adversarial strategy and Dropout adapt to each other to ensure the stability and consistency of the training process.

[0014] S7, the embedding layer maps each word or character in the vocabulary into a continuous vector representation in a high-dimensional space, while capturing the semantic relationship between words.

[0015] The method for preprocessing and word segmentation of the text in S1 is: using the BertTokenizer word segmenter for word segmentation, BertTokenizer is built based on the WordPiece algorithm, and the vocabulary size of its basic version is 21128; in BertTokenizer, unrecorded punctuation marks and rare characters will be marked as [UNK], and Chinese text will be split into single Chinese characters for processing.

[0016] The method for obtaining the word embedding vector of the text through BERT in S3 is as follows: after preprocessing and word segmentation operations, the serialized representation T = {X1, X2, ..., X n}, where Xi represents the i-th word in the text; before inputting the data into the BERT model, it needs to be processed in sequence through the tag embedding layer, the fragment embedding layer, and the position embedding layer to obtain the corresponding embedding representation Among them, E i represents the embedding vector of the i-th word in the text, and a represents a specific layer; finally, all the embedding representations are accumulated as the input representation of BERT, that is,

[0017] BERT in ={[CLS],W1,W2,...,W n ,[SEP]}

[0018] Among them, W i represents the cumulative embedding vector of the i-th word; the input is encoded by the Encoder to obtain the word-level output, which is represented by O = {O1, O2, ..., O n}, where O i Represents the representation vector of the i-th word.

[0019] The method for obtaining the word embedding vector of the text through BiLSTM in S3 is as follows: after the input text is encoded by the pre-trained model BERT, a deep bidirectional language representation is generated to obtain the complex relationship between words; the learning layer uses BiLSTM to process sequence data and captures long-distance dependencies through its gating mechanism; the calculation formula of LSTM at a certain moment is as follows:

[0020] f t =σ(W f ·[h t-1 ,o t ]+b f )

[0021] i t =σ(W i ·[h t-1 ,o t ]+b i )

[0022]

[0023] u t =σ(W o ·[h t-1 ,o t ]+b o )

[0024] h t =u t *tanh(C t )

[0025] Among them, h t-1 is the hidden state of the previous time step, o t is the input of the current time step. The first input represents the hidden layer vector output by BERT; i t is the activation value of the input gate, is a candidate memory cell, C t is the memory unit of the current time step, u t is the activation value of the output gate; then, a bidirectional mechanism is added to the LSTM, that is, the input of each time step will be processed by two LSTMs: one for the forward sequence and the other for the reverse sequence; for this, two hidden states will be obtained for each time step t: (positive) and (Reverse); Finally, the forward and reverse hidden states of BiLSTM are concatenated to obtain the final hidden layer vector representation of the model:

[0026]

[0027] The method of S4 weighted fusion of the local attention mechanism and the TextCNN model is:

[0028] In the encoding layer, the input sequence is the output of BERT:

[0029]

[0030] Among them, BAT o Indicates the dimension size of the input sequence, n indicates the sequence length; the output uses the hidden state of the previous BiLSTM forward and backward concatenation That is h t =(h1,h2,...,h l ), h represents the hidden vector of a word, and l represents the length of the sequence;

[0031] In the decoding layer, the hidden state S at each time step i The calculation is as follows:

[0032] z i =σ(W a s i-1 +Y a k i )

[0033] r i =σ(W b s i-1 +Y b k i )

[0034]

[0035] Where W a , W b , W∈R m×q , Y a , Y b , Y∈R q×2q are all weight matrices, m and q correspond to the dimensions of the word vector and the hidden layer respectively; for the initial hidden layer vector s0, the hidden layer vector of the reverse process of the first time step in the encoding layer is used for calculation, and the formula is as follows:

[0036]

[0037] Among them, W s ∈R n×n ;k i Indicates that with i and weight α ij The changed context vector is calculated as follows:

[0038]

[0039] The weight α at each time step t of the calculation decoding layer ijWhen , we need to first determine a position p in the input sequence that is aligned with this moment t , and then set a window size with this position as the center, that is, [p t -D,p t +D], D is an integer. Since the data set is short text, D is set to 4. The alignment method uses the predictive alignment (local-p) method, that is, p is aligned at each time step. t Make a prediction; the calculation formula is as follows:

[0040]

[0041] Among them, W p and v p are weight matrices, p t ∈[0,n]; Then, when calculating the weight vector, only the hidden state in the window in the coding layer is considered. When the range of the window exceeds the range of the input sequence, the excess part is directly discarded; at the same time, Gaussian distribution is used for correction, and the calculation formula is as follows:

[0042]

[0043] represents the hidden state of the encoding layer at the sth time step, s represents the position of the input sequence,

[0044] For the sequence S t =(s1,s2,...,s n ), using a convolution kernel with a window size of d, the width of the convolution kernel is the same as the hidden layer vector dimension q output by the previous model, that is, the convolution kernel ω∈R d×q ; The calculation process is as follows:

[0045] h f (l) = f(W f *x l:l+d-1 +b f )

[0046] Among them, h f (l) is the output of the f-th convolution kernel at position l, x l:l+d-1 is the vector of d words starting from position l in the input sequence, f is the size of the convolution kernel; f(·) is the activation function, and the ReLU function is usually used to introduce nonlinearity. The pooling layer uses maximum pooling to select the maximum value in each convolution feature map, which can reduce the number of parameters and extract the most important features. The calculation formula is as follows:

[0047] h f (l)=max(0,h f (l))

[0048] h f =max l h f (l)

[0049] Among them, h f It is the output of the fth convolution kernel after maximum pooling;

[0050] Use multiple convolution kernels for feature extraction to better utilize the convolution kernel to extract features, C = {C1, C2, ..., C m} is used to represent the combination of m convolution kernels, C m Represents the size of the mth convolution kernel; then calculate m eigenvectors and flatten them into a long vector to get the final result:

[0051] H t =[H1,H2,...,H F ].

[0052] The method for the classification layer in S5 to classify the sentiment polarity according to the extracted feature vector is:

[0053] The classification layer first concatenates the outputs of BiLSTM and TextCNN, and uses the diversity of different feature extraction methods to achieve multi-scale feature fusion, thereby generating a more comprehensive text representation and reducing the risk of overfitting the model on a specific data set; then, the fused features are input into the fully connected layer for further processing and classification; the formula is as follows:

[0054] H out =[h t :H t ]

[0055] H final =Linear(H out )

[0056] Finally, the corresponding tags in the sentiment classification task are obtained, and the loss function used is the cross entropy loss function, which is expressed as follows:

[0057]

[0058] The label of each sample is represented as a vector of length 4 (or 2), where the correct category corresponds to 1 and the other positions are 0; i,j is the jth element in the label vector. If sample i belongs to category j, then y i,j =1, otherwise y i,j =0; It represents the probability that the model predicts that sample i belongs to category j.

[0059] The method for data enhancement using a large batch adversarial training strategy in S6 is:

[0060] The robustness of the model is improved by introducing adversarial perturbations during the training process, generating adversarial samples to train the model and improve its robustness; expressed as follows:

[0061]

[0062] Add an adversarial perturbation δ with a small perturbation amplitude and easy to cause misclassification to the original input sample X, and then make the model resistant to this perturbation by updating the network parameters; where θ represents the parameters of the model, ||δ|| is the norm of the perturbation, and ε is the upper limit of the perturbation, which is used to control the degree of difference between the adversarial sample and the original sample; y is the true label of the original input X, L is the loss function, and f θ is the prediction function of the model;

[0063] Add a perturbation δ obtained by gradient ascent and the gradient of the model parameter θ to the input X of the model to maximize the loss function of the model; the implementation formula of the adversarial perturbation δ is:

[0064] δ new =δ old +α·sign(▽ δ L(f θ (X+δ old ),y)

[0065] Where α is the learning rate. When using PGD to generate adversarial samples for multiple iterations, the parameter gradients of the adversarial training samples are calculated in each iteration and accumulated and saved, instead of updating the model parameters immediately after each step. The acquired gradient information is effectively accumulated, which is equivalent to virtually expanding the calculation effect by K times, thereby further improving the model performance. In addition, the large batch adversarial strategy introduces up to K different norm constraints in training, imposing invariance on K different adversarial perturbations. The implementation formula is as follows:

[0066]

[0067] Finally, after K iterations, the model parameters θ are updated once using the average value of the accumulated gradients, as follows:

[0068]

[0069] Where τ is the learning rate; if the norm of the perturbation exceeds the preset range in a certain step, it needs to be projected back to the allowed norm range to ensure that the perturbation does not exceed the set limit. The formula is as follows:

[0070]

[0071] Here, δ is the calculated perturbation and ε is the maximum allowed norm.

[0072] The method of taking measures in S6 to make the adversarial strategy and Dropout adapt to each other is: at the beginning of adversarial training, the current Dropout mask needs to be initialized and backed up; in the subsequent iteration process, each time an adversarial sample is generated and the model parameters are updated, the same Dropout mask is used in the forward propagation to ensure that the network structure remains consistent in each iteration; in the K-step iteration, each forward and backward propagation uses the backup Dropout mask to ensure that the perturbation calculation and parameter update are performed under the same network structure; after completing the adversarial training, the hyperparameters are adjusted according to the experimental results to ensure that the model achieves the best performance.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] 1. When facing adversarial samples or noise interference, the traditional sentiment classification model is prone to a significant decline in classification effect. This invention introduces an improved adversarial training strategy and uses multi-step perturbation accumulation technology to enable the model to more effectively resist adversarial attacks and improve the stability and robustness of the model in complex scenarios.

[0075] 2. This invention combines the local attention mechanism with TextCNN weighting to fully explore the sentiment features of different granularities in the text. The local attention mechanism can identify the key information in the text and give it a higher weight, while TextCNN can capture local features and contextual dependencies, thus having more advantages in feature extraction than existing methods and generating more semantically representative text representations.

[0076] 3. The present invention adopts an improved adversarial training strategy, using the cumulative gradient method to avoid frequent updates of model parameters, thereby reducing the amount of calculation, and controlling the disturbance amplitude through multi-norm constraints, reducing the generalization error problem caused by the single norm constraint in the traditional PGD method. Compared with the existing PGD method, this strategy can significantly enhance the adversarial defense capability of the model while improving the efficiency of adversarial training.

[0077] 4. Traditional adversarial training methods usually cannot be used simultaneously with Dropout regularization, because the randomness of Dropout will lead to unstable network structure and affect the effectiveness of adversarial perturbation. The present invention fixes the Dropout mask during the training process to make adversarial training compatible with the Dropout randomization mechanism, which can maintain the original regularization effect of the model without affecting the stable generation of adversarial samples.

[0078] 5. By integrating multiple feature extraction models and attention mechanisms, the method can effectively process diverse text inputs in different sentiment classification task scenarios, and through fine control of feature levels, it shows good generalization ability in cross-domain and cross-language tasks. Compared with traditional models, this method can more accurately capture the feature differences under different tasks in multi-task learning, thereby improving the overall performance.

[0079] 6. The present invention adopts multiple model fusion strategies in the feature extraction layer and expands the training data set through a large batch adversarial strategy, so that the model can better learn the feature differences between different emotion categories while reducing overfitting. Compared with the model of a single feature extraction strategy, the text representation generated by this method is more comprehensive, thereby further improving the classification effect of the model in practical applications.

[0080] 7. The present invention uses a multi-level feature extraction mechanism to enable the model to effectively distinguish the sentiment tendencies of the same words in different contexts, solving the problem of improper handling of polysemous words, sentiment transitions and complex contexts in traditional models, thereby improving the overall accuracy of sentiment classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0082] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0083] Figure 1 It is a model structure diagram of the present invention;

[0084] Figure 2 It is the word segmentation flow chart of the present invention;

[0085] Figure 3 It is the overall structural flow chart of the large batch confrontation strategy of the present invention. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0087] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0088] In sentiment analysis tasks, we usually give labels representing sentiment polarity and text sentences to be classified, and use the model to classify sentences into corresponding sentiment polarity categories. Assume that the dataset sample form is (T, S), where T = {t1, t2, ..., t n} represents the set of sentiment polarity labels, and n represents the number of labels. i ,...,s m} represents the set of sentences to be classified, s i Represents the characters of the i-th sentence, and m is the length of the sentence.

[0089] The model structure is as follows Figure 1 As shown in the figure, the whole is divided into three layers: embedding learning layer, enhanced feature extraction layer and classification layer. First, the embedding learning layer obtains the word embedding vector of the text through BERT and BiLSTM. Then, the enhanced feature extraction layer performs weighted fusion of the local attention mechanism and the TextCNN model to further enhance the feature extraction effect. Finally, the classification layer performs sentiment polarity classification based on the extracted feature vector to obtain the final classification result.

[0090] During the model training process, a large batch adversarial training strategy is used for data augmentation to improve the robustness and generalization ability of the model.

[0091] The role of the embedding layer is to map each word or character in the vocabulary into a continuous vector representation in a high-dimensional space while capturing the semantic relationship between words.

[0092] In the text preprocessing stage, the BertTokenizer is used for word segmentation. BertTokenizer is built based on the WordPiece algorithm, and the vocabulary size of its basic version is 21128. WordPiece is a subword-level word segmentation algorithm that finds the set of subwords that can maximize the likelihood of the training set data during the merging process. In BertTokenizer, uncollected punctuation marks, rare characters and other words will be marked as [UNK], while Chinese texts will be split into single Chinese characters for processing. For example:

[0093] Text: Do you know how to pronounce the character "鋆"?

[0094] Tokens: you, know, [UNK], [UNK], [UNK], [UNK], character, how, to, pronounce, [UNK]

[0095] The specific word segmentation process is as Figure 2 shown.

[0096] After preprocessing and word segmentation operations, the serialized representation T = {X1, X2,..., X n} of the input text is obtained, where Xi represents the i-th word segment in the text. Before inputting the data into the BERT model, it needs to be processed through the token embedding layer, segment embedding layer, and position embedding layer in sequence to obtain the corresponding embedding representation where E i represents the embedding vector of the i-th word in the text, and a represents a specific layer. Finally, all the embedding representations are accumulated as the input representation of BERT, that is

[0097] BERT in = {[CLS], W1, W2,..., W n , [SEP]}

[0098] where W i represents the accumulated embedding vector of the i-th word. The input is encoded by the Encoder to obtain the output at the word level, denoted as O = {O1, O2,..., O n}, where O i represents the representation vector of the i-th word.

[0099] After the input text is encoded by the pre-trained model BERT, deep bidirectional language representations are generated, obtaining the complex relationships between words. The learning layer uses BiLSTM to process sequence data and captures long-distance dependencies through its gating mechanism.

[0100] The calculation formula of LSTM at a certain moment is as follows:

[0101] ft =σ(W f ·[h t-1 ,o t ]+b f )

[0102] i t =σ(W i ·[h t-1 ,o t ]+b i )

[0103]

[0104] u t =σ(W o ·[h t-1 ,o t ]+b o )

[0105] h t =u t *tanh(C t )

[0106] Among them, h t-1 is the hidden state of the previous time step, o t is the input of the current time step (the first input represents the hidden layer vector output by BERT), i t is the activation value of the input gate, is a candidate memory cell, C t is the memory unit of the current time step, u t is the activation value of the output gate. Then, this embodiment adds a bidirectional mechanism to the LSTM, that is, the input of each time step will be processed by two LSTMs: one for the forward sequence and the other for the reverse sequence. In this regard, this embodiment will obtain two hidden states for each time step t: (positive) and (Reverse). Finally, this embodiment concatenates the forward and reverse hidden states of BiLSTM to obtain the final hidden layer vector representation of the model:

[0107]

[0108] In order to capture local features more effectively, this embodiment combines the local attention mechanism with TextCNN. At present, most attention mechanisms will find the hidden state most relevant to the current output target from all hidden states of the input sequence at each time step during decoding, and use it together with the output of the previous moment as the input of the decoding layer at the current moment, thereby retaining most of the information of the input sequence. However, the computational cost of considering all hidden states is very high, especially for long sentences, the computational complexity increases significantly, making processing impractical. The local attention mechanism improves this, and only focuses on the local hidden states in the encoding layer instead of all states each time decoding. This method is more efficient and can better enhance the extraction of local features. The specific analysis and calculation are as follows:

[0109] In the encoding layer, the input sequence is the output of BERT:

[0110]

[0111] Among them, BAT o Indicates the dimension of the input sequence, and n indicates the length of the sequence. The output of this embodiment uses the hidden state of the previous BiLSTM forward and backward concatenation. That is h t =(h1,h2,...,h l ), h represents the hidden vector of a word, and l represents the length of the sequence.

[0112] In the decoding layer, the hidden state S at each time step i The calculation is as follows:

[0113] z i =σ(W a s i-1 +Y a k i )

[0114] r i =σ(W b s i-1 +Y b k i )

[0115]

[0116] Where W a , W b , W∈R m×q , Y a , Y b , Y∈R q×2qare all weight matrices, m and q correspond to the dimensions of the word vector and the hidden layer respectively. For the initial hidden layer vector s0, the hidden layer vector of the first time step in the encoding layer is used for calculation, and the formula is as follows:

[0117]

[0118] Among them, W s ∈R n×n . k i Indicates that with i and weight α ij The changed context vector is calculated as follows:

[0119]

[0120] The weight α at each time step t of the calculation decoding layer ij When , we need to first determine a position p in the input sequence that is aligned with this moment t , and then set a window size with this position as the center, that is, [p t -D,p t +D], D is an integer. Since the data set is short text, D is set to 4 in this embodiment. The alignment method in this embodiment adopts the predictive alignment (local-p) method, that is, p is aligned at each time step. t Make a prediction. The calculation formula is as follows:

[0121]

[0122] Among them, W p and v p are weight matrices, p t ∈[0,n]. Next, when calculating the weight vector, this embodiment only considers the hidden state in the window in the coding layer. When the range of the window exceeds the range of the input sequence, the excess part is directly discarded. At the same time, Gaussian distribution is used for correction, and its calculation formula is as follows:

[0123]

[0124] represents the hidden state of the encoding layer at the sth time step, s represents the position of the input sequence,

[0125] The local attention mechanism can identify the key parts of the text and assign higher weights to these important contents while ignoring irrelevant information. TextCNN uses convolution kernels to extract similar feature representations of local phrases containing the same vocabulary in the convolution layer. Its multi-layer structure can gradually learn abstract high-level features from the original text data, so as to better understand the meaning of the same words in different contexts. The weighted combination of the key contents identified by the local attention mechanism and TextCNN can further strengthen these local features and generate a comprehensive representation. While retaining the local features of the text, this representation also integrates global context information, thereby improving the model's ability to understand and express the emotional features of the text.

[0126] For the sequence S t =(s1,s2,...,s n ), using a convolution kernel with a window size of d, the width of the convolution kernel is the same as the hidden layer vector dimension q output by the previous model, that is, the convolution kernel ω∈R d×q The calculation process is as follows:

[0127] h f (l) = f(W f *x l:l+d-1 +b f )

[0128] Among them, h f (l) is the output of the f-th convolution kernel at position l, x l:l+d-1 is the vector of d words starting from position l in the input sequence, and f is the size of the convolution kernel. f(·) is the activation function, and the ReLU function is usually used to introduce nonlinearity. The pooling layer uses maximum pooling to select the maximum value in each convolution feature map, which can reduce the number of parameters and extract the most important features. The calculation formula is as follows:

[0129] h f (l)=max(0,h f (l))

[0130] h f =max l h f (l)

[0131] Among them, h f It is the output of the fth convolution kernel after maximum pooling.

[0132] This embodiment uses multiple convolution kernels to extract features to better utilize the convolution kernels to extract features, C=

[0133] {C1,C2,...,C m} is used to represent the combination of m convolution kernels, C mRepresents the size of the mth convolution kernel. Then calculate m eigenvectors and flatten them into a long vector to get the final result:

[0134] H t =[H1,H2,...,H F ]

[0135] The classification layer first concatenates the outputs of BiLSTM and TextCNN, and uses the diversity of different feature extraction methods to achieve multi-scale feature fusion, thereby generating a more comprehensive text representation and reducing the risk of overfitting the model on a specific data set. Subsequently, the fused features are input into the fully connected layer for further processing and classification. The formula is as follows:

[0136] H out =[h t :H t ]

[0137] H final =Linear(H out )

[0138] Finally, the corresponding mark in the sentiment classification task is obtained. The loss function used in this embodiment is the cross entropy loss function, which is expressed as follows:

[0139]

[0140] The label of each sample is represented as a vector of length 4 (or 2), with the correct category corresponding to the position 1 and the other positions 0. i,j is the jth element in the label vector. If sample i belongs to category j, then y i,j =1, otherwise y i,j =0. It represents the probability that the model predicts that sample i belongs to category j.

[0141] The basic idea of ​​adversarial training is to improve the robustness of the model by introducing adversarial perturbations during the training process.

[0142] The core concept of traditional adversarial training is the Min-Max formula, which is used to describe the optimization problem of the model when facing adversarial perturbations. Adversarial samples are usually generated to train the model and improve its robustness. The specific expression is as follows:

[0143]

[0144] This formula means adding an adversarial perturbation δ with a small perturbation amplitude and prone to misclassification to the original input sample X, and then updating the network parameters to make the model resistant to this perturbation. Among them, θ represents the parameters of the model, ||δ|| is the norm of the perturbation, and ε is the upper limit of the perturbation, which is used to control the degree of difference between the adversarial sample and the original sample. y is the true label of the original input X, L is the loss function, and f θ is the prediction function of the model.

[0145] Most existing adversarial algorithms usually use a one-time calculation method for gradients, and add perturbations to the word embedding layer during multi-step adversarial training, then adjust and optimize the perturbed word embedding space, and then input the generated word vector into the model for training. The projected gradient descent algorithm (PGD) has been innovatively designed to introduce a perturbation radius. When the gradient rises beyond this radius, it is projected back to the original gradient space to ensure that the perturbation amplitude is not too large. Specifically, PGD performs only a small update each time through multiple iterations, and projects the perturbation into a preset range at each iteration, so that the final gradient of the model is the sum of the initial gradient and the gradient generated by the last perturbation. Although this method is simple and efficient, its computational efficiency is low, and each gradient descent requires multiple steps (K steps) of gradient improvement, resulting in high computational cost. To this end, this embodiment proposes an improved PGD algorithm. By calculating the input gradient when the gradient rises, another parameter gradient can be obtained without additional cost. Therefore, when the gradient descends, the input gradient and the parameter gradient are used at the same time to introduce more information in one calculation, thereby accelerating the process of adversarial training. This improved strategy is called a large batch adversarial strategy. The specific analysis is as follows:

[0146] This strategy does not rely on global scalars, but instead embeds adversarial perturbations directly into specific words by calculating the gradient information of each word vector. Based on the gradient of each word vector, subtle changes at the word level can be dynamically considered throughout the model learning process, so as to more accurately capture the detailed differences in the model's learning of text data features. When determining the adversarial attack target, this can be achieved by calculating the relative importance of each word. Usually, the importance is evaluated by deleting words one by one and observing the changes in the model output results. For example, for the sentence "This restaurant has excellent service and delicious food", the large batch adversarial strategy will delete each word in the sentence in turn, and determine which words play a key role in the model's positive sentiment classification, while monitoring the changes in the model's prediction results. If the model's prediction changes from positive sentiment to negative sentiment or uncertain sentiment after deleting "good" or "delicious", it means that these words have a significant impact on the model's judgment and are therefore considered potential attack target words. On this basis, adding subtle perturbations to these key words can construct adversarial samples. From a human perspective, this adversarial sample is very similar to the original sentence, but it can successfully mislead the model and make it make incorrect sentiment classification, thereby revealing the potential vulnerability of the model and effectively improving the robustness of the model. The specific operation is to add a perturbation δ obtained by gradient ascent and the gradient of the model parameter θ to the input X of the model to maximize the loss function of the model. The specific implementation formula of the adversarial perturbation δ is:

[0147] δ new =δ old +α·sign(▽ δ L(f θ (X+δ old ),y)

[0148] α is the learning rate. When using PGD to generate adversarial samples for multiple iterations, the parameter gradients of the adversarial training samples are calculated in each iteration and accumulated and saved, instead of updating the model parameters immediately after each step. In this way, the acquired gradient information can be effectively accumulated, which is equivalent to virtually expanding the calculation effect by K times, thereby further improving the model performance. In addition, the large-batch adversarial strategy introduces up to K different norm constraints in training, imposing invariance on K different adversarial perturbations. In contrast, traditional PGD only uses a single norm constraint, and such a design is prone to greater generalization error. Multi-norm constraints can better balance the performance of the model under different perturbation conditions, thereby effectively reducing the generalization error and improving the stability and robustness of the model. The implementation formula is as follows:

[0149]

[0150] Finally, after K iterations, the average value of the accumulated gradients is used to update the model parameters θ at one time. As follows:

[0151]

[0152] Where τ is the learning rate. This method can virtually expand the original data set to K times with minimal additional overhead, thereby reducing the number of model parameter updates and improving the utilization of gradient information during training, which helps to enhance the robustness of the model. If the norm of the perturbation exceeds the preset range in a certain step, it needs to be projected back to the allowed norm range to ensure that the perturbation does not exceed the set limit. The specific formula is as follows:

[0153]

[0154] Here, δ is the calculated perturbation and ε is the maximum allowed norm.

[0155] Normally, adversarial training is not used with Dropout. Because Dropout, as a regularization method, randomly discards some network connections at each forward propagation, thereby changing the network structure, which may lead to instability in the adversarial perturbation calculation process. However, since the BERT pre-trained model enables Dropout by default during the fine-tuning process, this embodiment needs to take a series of measures to make the adversarial strategy and Dropout adapt to each other, so as to ensure the stability and consistency of the training process.

[0156] First, at the beginning of adversarial training, the current Dropout mask (Dropoutmask) needs to be initialized and backed up. In the subsequent iterations, each time an adversarial sample is generated and the model parameters are updated, the same Dropout mask is used in the forward propagation to ensure that the network structure remains consistent in each iteration. Specifically, in the K-step iteration, each forward and backward propagation uses the backup Dropout mask to ensure that the perturbation calculation and parameter update are performed under the same network structure. After completing the adversarial training, the hyperparameters are adjusted according to the experimental results to ensure that the model achieves the best performance. The overall structural flow chart of the large batch adversarial strategy is shown in the figure. Figure 3 shown.

[0157] Only the preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the protection scope of the present invention.

Claims

1. A text sentiment classification method that incorporates a new adversarial strategy and enhanced feature extraction, characterized in that: The following steps are involved: S1. Preprocess and segment the text; S2. Build the model. The model is divided into three layers: embedding learning layer, enhanced feature extraction layer and classification layer; S3, the embedding learning layer obtains the word embedding vector of the text through BERT and BiLSTM; S4, the enhanced feature extraction layer performs weighted fusion of the local attention mechanism and the TextCNN model to further enhance the feature extraction effect; S5, the classification layer classifies the sentiment polarity according to the extracted feature vector to obtain the final classification result; S6. During the model training process, a large batch adversarial training strategy is used for data augmentation to improve the robustness and generalization ability of the model. Since Dropout is enabled by default during the fine-tuning process of the BERT pre-trained model, measures need to be taken to make the adversarial strategy and Dropout adapt to each other to ensure the stability and consistency of the training process. S7, the embedding layer maps each word or character in the vocabulary into a continuous vector representation in a high-dimensional space, while capturing the semantic relationship between words.

2. The text sentiment classification method incorporating a novel adversarial strategy and enhanced feature extraction according to claim 1, characterized in that: The method for preprocessing and word segmentation of the text in S1 is: using the BertTokenizer word segmenter for word segmentation, BertTokenizer is built based on the WordPiece algorithm, and the vocabulary size of its basic version is 21128; in BertTokenizer, unrecorded punctuation marks and rare characters will be marked as [UNK], and Chinese text will be split into single Chinese characters for processing.

3. The text sentiment classification method incorporating a novel adversarial strategy and enhanced feature extraction according to claim 1, characterized in that: The method for obtaining the word embedding vector of the text through BERT in S3 is as follows: after preprocessing and word segmentation operations, the serialized representation T = {X1, X2, ..., X n }, where Xi represents the i-th word in the text; before inputting the data into the BERT model, it needs to be processed in sequence through the tag embedding layer, the fragment embedding layer, and the position embedding layer to obtain the corresponding embedding representation Among them, E i represents the embedding vector of the i-th word in the text, and a represents a specific layer; finally, all the embedding representations are accumulated as the input representation of BERT, that is, BERT in ={[CLS],W1,W2,...,W n ,[SEP]} Among them, W i represents the cumulative embedding vector of the i-th word; the input is encoded by the Encoder to obtain the word-level output, which is represented by O = {O1, O2, ..., O n }, where O i Represents the representation vector of the i-th word.

4. The text sentiment classification method incorporating a novel adversarial strategy and enhanced feature extraction according to claim 1, characterized in that: The method for obtaining the word embedding vector of the text through BiLSTM in S3 is as follows: after the input text is encoded by the pre-trained model BERT, a deep bidirectional language representation is generated to obtain the complex relationship between words; the learning layer uses BiLSTM to process sequence data and captures long-distance dependencies through its gating mechanism; the calculation formula of LSTM at a certain moment is as follows: f t =σ(W f ·[h t-1 ,o t ]+b f ) I t =σ(W i ·[h t-1 ,o t ]+b i ) you t =σ(W o ·[h t-1 ,o t ]+b o ) h t =u t *tanh(C t ) Among them, h t-1 is the hidden state of the previous time step, o t is the input of the current time step. The first input represents the hidden layer vector output by BERT; i t is the activation value of the input gate, is a candidate memory cell, C t is the memory unit of the current time step, u t is the activation value of the output gate; then, a bidirectional mechanism is added to the LSTM, that is, the input of each time step will be processed by two LSTMs: one for the forward sequence and the other for the reverse sequence; for this, two hidden states will be obtained for each time step t: (positive) and (Reverse); Finally, the forward and reverse hidden states of BiLSTM are concatenated to obtain the final hidden layer vector representation of the model:

5. The text sentiment classification method incorporating a novel adversarial strategy and enhanced feature extraction according to claim 1, characterized in that: The method of S4 weighted fusion of the local attention mechanism and the TextCNN model is: In the encoding layer, the input sequence is the output of BERT: Among them, BAT o Indicates the dimension size of the input sequence, n indicates the sequence length; the output uses the hidden state of the previous BiLSTM forward and backward concatenation That is h t =(h1,h2,...,h l ), h represents the hidden vector of a word, and l represents the length of the sequence; In the decoding layer, the hidden state S at each time step i The calculation is as follows: With i =σ(W a s i-1 +Y a k i ) r i =σ(W b s i-1 +Y b k i ) Where W a , W b , W∈R m×q , Y a , Y b , Y∈R q×2q are all weight matrices, m and q correspond to the dimensions of the word vector and the hidden layer respectively; for the initial hidden layer vector s0, the hidden layer vector of the reverse process of the first time step in the encoding layer is used for calculation, and the formula is as follows: Among them, W s ∈R n×n ;k i Indicates that with i and weight α ij The changed context vector is calculated as follows: The weight α at each time step t of the calculation decoding layer ij When , we need to first determine a position p in the input sequence that is aligned with this moment t , and then set a window size with this position as the center, that is, [p t -D,p t +D], D is an integer. Since the data set is short text, D is set to 4. The alignment method uses the predictive alignment (local-p) method, that is, p is aligned at each time step. t Make a prediction; the calculation formula is as follows: Among them, W p and v p are weight matrices, p t ∈[0,n]; Then, when calculating the weight vector, only the hidden state in the window in the coding layer is considered. When the range of the window exceeds the range of the input sequence, the excess part is directly discarded; at the same time, Gaussian distribution is used for correction, and the calculation formula is as follows: h s represents the hidden state of the encoding layer at the sth time step, s represents the position of the input sequence, For the sequence S t =(s1,s2,...,s n ), using a convolution kernel with a window size of d, the width of the convolution kernel is the same as the hidden layer vector dimension q output by the previous model, that is, the convolution kernel ω∈R d×q ; The calculation process is as follows: h f (l)=f(W f *x l:l+d-1 +b f ) Among them, h f (l) is the output of the f-th convolution kernel at position l, x l:l+d-1 is the vector of d words starting from position l in the input sequence, f is the size of the convolution kernel; f(·) is the activation function, and the ReLU function is usually used to introduce nonlinearity. The pooling layer uses maximum pooling to select the maximum value in each convolution feature map, which can reduce the number of parameters and extract the most important features. The calculation formula is as follows: h f (l)=max(0,h f (l)) h f =max l h f (l) Among them, h f It is the output of the fth convolution kernel after maximum pooling; Use multiple convolution kernels for feature extraction to better utilize the convolution kernel to extract features, C = {C1, C2, ..., C m } is used to represent the combination of m convolution kernels, C m Represents the size of the mth convolution kernel; then calculate m eigenvectors and flatten them into a long vector to get the final result: H t =[H1,H2,...,H F ]。 6. The text sentiment classification method incorporating a novel adversarial strategy and enhanced feature extraction according to claim 1, characterized in that: The method for the classification layer in S5 to classify the sentiment polarity according to the extracted feature vector is: The classification layer first concatenates the outputs of BiLSTM and TextCNN, and uses the diversity of different feature extraction methods to achieve multi-scale feature fusion, thereby generating a more comprehensive text representation and reducing the risk of overfitting the model on a specific data set; then, the fused features are input into the fully connected layer for further processing and classification; the formula is as follows: H out =[h t :H t ] H final =Linear(H out ) Finally, the corresponding tags in the sentiment classification task are obtained, and the loss function used is the cross entropy loss function, which is expressed as follows: The label of each sample is represented as a vector of length 4 (or 2), where the correct category corresponds to 1 and the other positions are 0; i,j is the jth element in the label vector. If sample i belongs to category j, then y i,j =1, otherwise y i,j =0; It represents the probability that the model predicts that sample i belongs to category j.

7. The text sentiment classification method incorporating a novel adversarial strategy and enhanced feature extraction according to claim 1, characterized in that: The method for data enhancement using a large batch adversarial training strategy in S6 is: The robustness of the model is improved by introducing adversarial perturbations during the training process, generating adversarial samples to train the model and improve its robustness; expressed as follows: Add an adversarial perturbation δ with a small perturbation amplitude and easy to cause misclassification to the original input sample X, and then make the model resistant to this perturbation by updating the network parameters; where θ represents the parameters of the model, ||δ|| is the norm of the perturbation, and ε is the upper limit of the perturbation, which is used to control the degree of difference between the adversarial sample and the original sample; y is the true label of the original input X, L is the loss function, and f θ is the prediction function of the model; Add a perturbation δ obtained by gradient ascent and the gradient of the model parameter θ to the input X of the model to maximize the loss function of the model; the implementation formula of the adversarial perturbation δ is: Where α is the learning rate. When using PGD to generate adversarial samples for multiple iterations, the parameter gradients of the adversarial training samples are calculated in each iteration and accumulated and saved, instead of updating the model parameters immediately after each step. The acquired gradient information is effectively accumulated, which is equivalent to virtually expanding the calculation effect by K times, thereby further improving the model performance. In addition, the large batch adversarial strategy introduces up to K different norm constraints in training, imposing invariance on K different adversarial perturbations. The implementation formula is as follows: Finally, after K iterations, the model parameters θ are updated once using the average value of the accumulated gradients, as follows: Where τ is the learning rate; if the norm of the perturbation exceeds the preset range in a certain step, it needs to be projected back to the allowed norm range to ensure that the perturbation does not exceed the set limit. The formula is as follows: Here, δ is the calculated perturbation and ε is the maximum allowed norm.

8. The text sentiment classification method incorporating a novel adversarial strategy and enhanced feature extraction according to claim 1, characterized in that: The method of taking measures in S6 to make the adversarial strategy and Dropout adapt to each other is: at the beginning of adversarial training, the current Dropout mask needs to be initialized and backed up; in the subsequent iteration process, each time an adversarial sample is generated and the model parameters are updated, the same Dropout mask is used in the forward propagation to ensure that the network structure remains consistent in each iteration; in the K-step iteration, each forward and backward propagation uses the backup Dropout mask to ensure that the perturbation calculation and parameter update are performed under the same network structure; after completing the adversarial training, the hyperparameters are adjusted according to the experimental results to ensure that the model achieves the best performance.

Images