An adversarial attack text classification method based on granules and metric learning

Abstract

The application belongs to the technical field of natural language processing, and particularly relates to an adversarial attack text classification method based on granular ball and metric learning, which comprises the following steps: obtaining a text to be classified, inputting the text to be classified into a trained encoding module to obtain semantic vector representation of the text to be classified; mapping the semantic vector representation of the text to be classified to a trained granular ball space to obtain a granular ball center, inputting the semantic vector representation corresponding to the granular ball center into a classifier to obtain a classification result; the application uses granular ball clustering to aggregate the semantic vector representation of the same kind of text into a granular ball to obtain a granular ball space, and then maps the semantic vector representation of the text to be classified to the granular ball space to obtain a corresponding granular ball center, and the influence of an adversarial sample is eliminated by classifying by using the corresponding granular ball center, so that the accuracy of the classifier under an attack condition is improved compared with the prior art.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and particularly relates to an adversarial attack text classification method based on granular sphere and metric learning. BACKGROUND

[0002] In the field of natural language processing (NLP), text classification is a core task and is widely used in information retrieval, sentiment analysis, spam filtering and many other scenarios.

[0003] Traditional text classification methods are usually based on feature extraction and machine learning models, such as Naive Bayes classifier, support vector machine (SVM) and decision tree. However, with the complexity and scale of text data growing, traditional methods often struggle to handle high-dimensional, sparse data and diverse texts. Especially when facing attack samples, traditional classification methods not only may be affected, leading to inaccurate prediction results, but also may weaken the overall understanding ability of the model to the text.

[0004] Currently, there is no classification method that can effectively resist the interference of attackers, so that the classification performance is still poor in the presence of attacks. Therefore, it is urgent to explore new methods to enhance the robustness and accuracy of text classification models when facing attacks. SUMMARY

[0005] To solve the above technical problems of the prior art, the application adopts an adversarial attack text classification method based on granular sphere and metric learning, which comprises: obtaining a text to be classified, inputting the text to be classified into a trained encoding module to obtain a semantic vector representation of the text to be classified; mapping the semantic vector representation of the text to be classified to a trained granular sphere space to obtain a granular sphere center, inputting the semantic vector representation corresponding to the granular sphere center into a classifier to obtain a classification result; the training process of the encoding module, the granular sphere space and the classifier comprises:

[0006] S1, obtaining a text data set, preprocessing the text in the text data set to obtain preprocessed text;

[0007] S2, inputting the preprocessed text into an encoding module for encoding to obtain a semantic vector representation of the text;

[0008] S3, performing supervised clustering on the semantic vector representations of all texts using granular sphere calculation to generate a granular sphere space, the granular sphere space comprising a plurality of granular spheres and corresponding granular sphere centers;

[0009] S4, inputting the semantic vector representation corresponding to the granular sphere center of the granular sphere in which each text is located into a classifier to obtain a classification result of each text;

[0010] S5, constructing a sample pair of each text according to the semantic vector representation of all texts, calculating a loss function value according to the sample pair of each text and the classification result, updating the parameters of the encoding module and the classifier according to the loss function value, and obtaining a finally trained encoding module, granular ball space and classifier when the loss function value is minimum.

[0011] Preprocessing the texts in the text data set includes:

[0012] S11, constructing a regular expression rule, and denoising the texts in the text data set by using the regular expression rule;

[0013] S12, performing word segmentation processing on the denoised texts to obtain preprocessed texts.

[0014] The encoding module encodes the preprocessed texts, including:

[0015] S21, performing deep semantic feature extraction on the preprocessed texts by using the pre-trained Bert model to obtain a feature vector of the texts;

[0016] S22, performing pooling on the feature vector of the texts, and performing normalization processing on the pooled result to obtain a semantic vector representation of the texts.

[0017] The supervised clustering of the semantic vector representations of all texts includes:

[0018] S31, combining the semantic vector representations of all texts into a granular ball, and setting a purity threshold and a quality threshold;

[0019] S32, randomly selecting a granular ball o g from all granular balls, calculating the purity of the granular ball o g , judging whether the purity of the granular ball o g is less than the purity threshold, and if so, performing step S33 on the granular ball o g ; otherwise, calculating the quality of the granular ball o g , and judging whether the quality of the granular ball o g reaches the quality threshold, and if so, performing step S35; otherwise, performing step S34;

[0020] S33, traversing each semantic vector representation corresponding to a category label in the granular ball o g to obtain a label number k, dividing the granular ball o g into k granular balls by using a K-means clustering algorithm, and then performing step S32;

[0021] S34, aggregating granular balls with the same label and without reaching the quality threshold into one granular ball, and then performing step S32;

[0022] S35. Determine if the mass of all particles reaches the mass threshold. If so, combine all particles into a particle space; otherwise, stop particle movement. g The process remains unchanged, and step S32 is performed on the remaining particles.

[0023] The purity of a particle is the proportion of semantic vectors in the particle that are identical to the particle's label, and the label of a particle is the label whose corresponding semantic vector is most frequently represented in the particle.

[0024] The sample pairs for constructing text i include:

[0025] S41. Perform data augmentation on the semantic vector representations of M texts to obtain M augmented texts. Divide the semantic vector representations of the M texts into S spheres according to the labels. s Calculate each ball o s The center of the sphere is S, where S is the number of labels in the text dataset, and s is the number of spheres. s The index;

[0026] S42, Place the ball o that is identical to the text label i. s The semantic vector representations of all texts and their augmented texts are combined into a positive sample space, and balls o that are different from the text i label are included. s The semantic vector representations of all texts constitute a negative sample space;

[0027] S43. Based on the positive sample space, negative sample space, and sphere o s The center constructs sample pairs of text i.

[0028] Each text sample pair is:

[0029]

[0030] in, A set representing sample pairs of text. <t i ,m i ,n i > represents a sample pair of text i, t i m i n i Both are triples, representing sub-sample pairs of sample pairs for text i, x i Let i be the semantic vector representation of text i. Let be the semantic vector representation of text i in the positive sample space. Let be the semantic vector representation of text i in the negative sample space. For the ball o that is the same as the text i label s The semantic vector representation of the center, For the ball o that is different from the text i label s The semantic vector representation of the center, For the enhanced text of text i, M is the number of texts.

[0031] The loss function Loss is:

[0032]

[0033] Where y i is the label of text i, p i is the classification result of text i, L is the metric loss function, and γ is the index of the sample pair.

[0034] The metric loss function is:

[0035] L(γ)=L(t i ,m i ,n i )=θ×d(t i ,m i )-μ×d(t i ,n i )

[0036] Where θ and μ are hyperparameters, and d represents the cosine distance.

[0037] Mapping the semantic vector representation of the text to be classified to the trained granular ball space includes: calculating the metric distance between the semantic vector representation of the text to be classified and the semantic vector representation corresponding to the granular ball center in the trained granular ball space, and selecting multiple granular ball centers with the smallest metric distance.

[0038] The beneficial effects of the present application are:

[0039] 1. The present application uses granular ball clustering to aggregate the semantic vector representations of similar texts into a granular ball, obtains a granular ball space, and then maps the semantic vector representation of the text to be measured to the granular ball space to obtain the corresponding granular ball center. The influence of the adversarial sample is eliminated by using the corresponding granular ball center for classification, which improves the accuracy of the classifier under attack compared with the prior art; 2. While optimizing the classifier, the present application optimizes the semantic vector representation of the text through contrastive learning, and then optimizes the granular ball space, effectively reducing the interference of the adversarial sample on the model, and improving the stability of the model under harsh conditions. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A flowchart of the present application, an adversarial attack text classification method based on granular balls and metric learning;

[0041] Figure 2 A result schematic diagram of the present application, an adversarial attack text classification method based on granular balls and metric learning. DETAILED DESCRIPTION

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] This invention provides an adversarial attack text classification method based on particle spheres and metric learning, such as... Figure 1 As shown, the process includes the following steps: obtaining the text to be classified, inputting the text to be classified into the trained encoding module to obtain the semantic vector representation of the text to be classified; mapping the semantic vector representation of the text to be classified onto the trained particle space to obtain the particle center; inputting the semantic vector representation corresponding to the particle center into the classifier to obtain the classification result.

[0044] Specifically, the mapping method involves calculating the metric distance between the semantic vector representation of the text to be classified and the semantic vector representation corresponding to the center of the particle in the trained particle space. The r centers with the smallest metric distance are selected, where r is a hyperparameter. The value of r is set to different orders of magnitude depending on the dataset, generally between 4 and 300. The semantic vector representations corresponding to the r centers are input into the classifier to obtain the classification results of the r centers. The classification results of the r centers are then voted on, and the classification result with the highest vote rate is taken as the final classification result.

[0045] The training process for the encoding module, particle space, and classifier includes:

[0046] S1. Obtain a text dataset, preprocess the text in the text dataset, and obtain preprocessed text; the text dataset includes M texts and their labels.

[0047] S11. Use web crawling technology and other methods to obtain text datasets;

[0048] The text dataset at this point may contain a large number of illegal characters and redundant symbols, which constitute text noise; therefore, it is necessary to denoise the text dataset.

[0049] S12. Construct regular expression rules and denoise the text according to the regular expression rules;

[0050] Text denoising based on regular expression rules includes: uniformly lowercase text conversion, and replacing web page tags in the text with whitespace characters, such as... This regular expression rule removes duplicate punctuation marks, non-alphanumeric characters, and non-punctuation symbols. It also includes rules such as using ^[0-9] to match numbers, ^[\u4e00-\u9fa5] to match Chinese characters, and ^[A-Za-z] to match English letters.

[0051] S13, performing word segmentation processing on the denoised text;

[0052] Specifically, stop words are removed, and the text representation is further optimized through stem extraction or morphological reduction to enhance semantic consistency and set a topic label for each data according to its semantic features.

[0053] Specifically, different word segmentation tools are selected for different text data sets. If the text data is English text, the NLTK package in the Python library is used for word segmentation; if the text data is Chinese text, the jieba library can be used for word segmentation.

[0054] S2, inputting the preprocessed text into an encoding module for encoding to obtain a semantic vector representation of the text;

[0055] S21, inputting the preprocessed text into a pre-trained Bidirectional Encoder Representations from Transformers (Bert) model. The transformer structure of the Bert model will extract deep semantic features from the text to obtain a feature vector, i.e. a hidden layer representation;

[0056] S22, using a pooling method (average pooling or maximum pooling) to aggregate the feature vector output by the Bert to reduce the feature dimension; then normalizing the pooled feature vector and saving the feature representation, as follows:

[0057] Normalization processing: normalizing the feature vector, usually using L2 normalization to convert the feature vector to a unit vector, reducing the influence of feature differences on subsequent clustering.

[0058] Feature representation saving: saving the normalized feature vector as a semantic vector representation of the text for subsequent processing.

[0059] In one embodiment, large-scale text data is processed in batches, each batch including an appropriate amount of text to avoid memory overflow.

[0060] S3, using granular ball calculation to perform supervised clustering on the semantic vector representations of all texts to generate a granular ball space, which includes multiple granular balls and their corresponding granular ball centers;

[0061] Specifically, step S3 uses a granular ball purity adaptive method to cluster the semantic vector representations of all texts to obtain a sentence set, including:

[0062] S31, combine the semantic vector representations of all texts into a granule, set purity threshold and quality threshold;

[0063] S32, randomly select a granule o from all granules g , calculate the purity of granule o g , determine whether the purity of granule o g is less than the purity threshold, if yes, execute step S33 on granule o g ; otherwise, calculate the quality of granule o g , and determine whether the quality of granule o g reaches the quality threshold, if yes, execute step S35; otherwise, execute step S34;

[0064] S33, traverse each semantic vector representation in granule o g to obtain the number of labels k, divide granule o g into k granules by K-means clustering algorithm, and then execute step S32;

[0065] S34, aggregate granules with the same label and quality not reaching the quality threshold into one granule, and then execute step S32;

[0066] S35, determine whether the quality of all granules reaches the quality threshold, if yes, combine all granules into a granule space; otherwise, granule o g remains unchanged, and execute step S32 on the remaining granules.

[0067] Preferably, the purity threshold is usually set to be above 0.85, and the quality threshold is usually set to be above 2.

[0068] The formulas for calculating the purity, quality and center of a granule are as follows:

[0069]

[0070] l GB =max(l1,l2,…,l n ),

[0071]

[0072] wherein GB is a granule, x j is the semantic vector representation of the jth text in the granule, c GB is the center of the granule, i.e. the average of the semantic vector representations of all texts in the granule, l GB is the label of the granule, which is the label appearing most frequently in the granule, and (l1,l2,…,l n) is the purity of the granule ball, that is, the proportion of semantic vector representations in the granule ball that are the same as the label of the granule ball, label(j) represents the label of the text j. GB ) is the purity of the granule ball, that is, the proportion of semantic vector representations in the granule ball that are the same as the label of the granule ball, label(j) represents the label of the text j.

[0073] S4, the semantic vector corresponding to the granule center of the granule ball where each text is located is respectively represented into a classifier to obtain the classification result of each text.

[0074] For the initial granule text clustering, the vector information of the text completely depends on the text semantic cognition learned by the pre-training of the pre-trained Bert encoder, and there is no specific semantic cognition for the classification task at the moment, so the granule center obtained by the granule self-adaptive method cannot well represent the information contained in its own class, and cannot adapt to the attack method. Therefore, the model is trained in combination with specific classification tasks and attack conditions, and metric learning is used to enable the model to deeply recognize the semantic similar features of the same class of texts and the semantic distinguishing features of different classes of texts. The specific training process includes:

[0075] S41, the semantic vector representations of M texts are respectively subjected to data enhancement to obtain M enhanced texts, and the semantic vector representations of the M texts are divided into S balls according to the labels s , the center of each ball o s is calculated, S is the number of labels of the text data set, and s is the index of the ball o s ; the center of each ball o s is the average value of the semantic vector representations of all texts in the ball.

[0076] S42, the semantic vector representations of all texts in the ball o s same as the label of the text i and its enhanced texts are combined into a positive sample space, and the semantic vector representations of all texts in the ball o s different from the label of the text i form a negative sample space.

[0077] S43, a sample pair of the text i is constructed according to the positive sample space, the negative sample space and the center of the ball o s .

[0078] S5, a sample pair of each text is constructed according to the semantic vector representations of all texts, a loss function value is calculated according to the sample pair of each text and the classification result, and the parameters of the encoding module and the classifier are updated according to the loss function value. When the loss function value is the smallest, the finally trained encoding module, granule space and classifier are obtained.

[0079] Specifically, the calculation formula of the training loss is as follows:

[0080]

[0081] L(t i ,m i ,n i )=θ×d(t i ,m i )-μ×d(t i ,n i )

[0082]

[0083] wherein, denotes a set of text sample pairs, <t i ,m i ,n i > is a sample pair of text i, the triplet the triplet the triplet x i is a semantic vector representation of text i, is a semantic vector representation in the positive sample space of text i, is a semantic vector representation in the negative sample space of text i, is a semantic vector representation of the center of the ball o s with the same label as the text i, is a semantic vector representation of the center of the ball o s with a different label from the text i, is an enhanced text of text i, y i is a label of text i, o i is a classification result of text i, L is a metric loss function, γ is an index of a sample pair, θ and μ are hyperparameters, and d denotes a cosine distance.

[0084] In the training process, the parameters of the model optimized by back propagation and the text semantic vector representation generated by encoding are updated, and the optimized vector is also re-clustered to continuously update the granular ball used for mapping, so that the granular ball adapts to the text.

[0085] As shown in Figure 2 , a normal sample will get an adversarial sample after being attacked by the attack model. The adversarial sample modifies some words based on the original sample, and the overall does not affect the negative sentiment of the sentence, but after the adversarial sample is input into the original language model, the original language model will make a mistake and predict the sample as a positive sentiment, while the model of the present application will defend the disturbance of the adversarial sample and correctly predict the sample as a negative sentiment.

[0086] The above examples further illustrate the objects, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and are not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made to the present application within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An adversarial attack text classification method based on particle spheres and metric learning, characterized in that, include: The text to be classified is obtained and input into the trained encoding module to obtain the semantic vector representation of the text to be classified; the semantic vector representation of the text to be classified is mapped onto the trained particle space to obtain the particle center; the semantic vector representation corresponding to the particle center is input into the classifier to obtain the classification result. The training process for the encoding module, the particle space, and the classifier includes: S1. Obtain the text dataset, preprocess the text in the text dataset, and obtain the preprocessed text; S2. Input the preprocessed text into the encoding module for encoding to obtain the semantic vector representation of the text; S3. Supervised clustering of the semantic vector representations of all texts is performed using particle sphere computation to generate a particle sphere space, which includes multiple particles and their corresponding particle sphere centers. S4. Input the semantic vector corresponding to the center of the sphere containing each text into the classifier to obtain the classification result for each text. S5. Construct sample pairs for each text based on the semantic vector representations of all texts. Calculate the loss function value based on the sample pairs and classification results of each text. Update the parameters of the encoding module and classifier based on the loss function value. When the loss function value is minimized, the final trained encoding module, particle space, and classifier are obtained. Supervised clustering of the semantic vector representations of all texts includes: S31. Combine the semantic vector representations of all texts into a single sphere, and set a purity threshold and a quality threshold. S32. Randomly select one ball from all the balls. g Calculate the particle size. g The purity of the particles is used to determine the purity of the particles. g If the purity is less than the purity threshold, then for the granules... g Execute step S33; otherwise, calculate the particle size. g The quality of the particles and the determination of their quality. g If the quality reaches the quality threshold, proceed to step S35; otherwise, proceed to step S34. S33, Traversing the spheres g Each semantic vector in the dataset represents a corresponding category label, resulting in the number of labels k. The K-means clustering algorithm is then used to separate the spheres into clusters. g Divide into k spheres, then proceed to step S32; S34. Aggregate the particles that do not meet the quality threshold and have the same label into one particle, and then execute step S32. S35. Determine if the mass of all particles reaches the mass threshold. If so, combine all particles into a particle space; otherwise, stop particle movement. g The process remains unchanged, and step S32 is performed on the remaining particles; The purity of a particle is the proportion of semantic vector representations in the particle that are identical to the particle's label; the label of a particle is the label with the most semantic vector representations in the particle; the quality of a particle is the number of semantic vector representations in the particle.

2. The adversarial attack text classification method based on particle spheres and metric learning according to claim 1, characterized in that, Preprocess the text in the text dataset: S11. Construct regular expression rules and use them to denoise the text in the text dataset. S12. Perform word segmentation on the denoised text to obtain the preprocessed text.

3. The adversarial attack text classification method based on particle spheres and metric learning according to claim 1, characterized in that, The encoding module encodes the preprocessed text, including: S21. Use a pre-trained BERT model to perform deep semantic feature extraction on the pre-processed text to obtain the text's feature vector. S22. Pool the feature vectors of the text, and normalize the pooled results to obtain the semantic vector representation of the text.

4. The adversarial attack text classification method based on particle spheres and metric learning according to claim 1, characterized in that, The sample pairs for constructing text i include: S41. Perform data augmentation on the semantic vector representations of M texts to obtain M augmented texts. Divide the semantic vector representations of the M texts into S spheres according to the labels. s Calculate each ball o s The center of the sphere is S, where S is the number of labels in the text dataset, and s is the number of spheres. s The index; S42, Place the ball o that is identical to the text label i. s The semantic vector representations of all texts and their augmented texts are combined into a positive sample space, and balls o that are different from the text i label are included. s The semantic vector representations of all texts constitute a negative sample space; S43. Based on the positive sample space, negative sample space, and sphere o s The center constructs sample pairs of text i.

5. The adversarial attack text classification method based on particle spheres and metric learning according to claim 4, characterized in that, Each text sample pair is: in, A set representing sample pairs of text. <t i ,m i ,n i > represents a sample pair of text i, t i m i n i All are triples, t i m i n i x is a subsample pair of sample pairs of text i. i Let i be the semantic vector representation of text i. Let be the semantic vector representation of text i in the positive sample space. Let be the semantic vector representation of text i in the negative sample space. For the ball o that is the same as the text i label s The semantic vector representation of the center, For the ball o that is different from the text i label s The semantic vector representation of the center, Let M be the enhanced text of text i, and M be the number of texts.

6. The adversarial attack text classification method based on particle spheres and metric learning according to claim 5, characterized in that, The loss function is: Among them, y i p is the label for text i. i Let L be the classification result for text i, L be the loss function, and γ be the index of the sample pair.

7. The adversarial attack text classification method based on particle spheres and metric learning according to claim 6, characterized in that, The loss function is: L(γ)=L(t i ,m i ,n i )=θ×d(t i ,m i )-μ×d(t i ,n i ) Where θ and μ are hyperparameters, and d represents the cosine distance.

8. The adversarial attack text classification method based on particle spheres and metric learning according to claim 1, characterized in that, Mapping the semantic vector representation of the text to be classified to the trained particle space includes: calculating the metric distance between the semantic vector representation of the text to be classified and the semantic vector representation corresponding to the particle center in the trained particle space, and selecting multiple particle centers with the smallest metric distance.

Images