Granular ball and metric learning-based adversarial attack text classification method

Abstract

The invention belongs to the technical field of natural language processing, and particularly relates to an adversarial attack text classification method based on granular ball and metric learning, which comprises the following steps: acquiring a to-be-classified text, and inputting the to-be-classified text into a trained coding module to obtain semantic vector representation of the to-be-classified text; mapping the semantic vector representation of the to-be-classified text to the trained grainball space to obtain a grainball center, and inputting the semantic vector representation corresponding to the grainball center into the classifier to obtain a classification result; according to the method, semantic vector representations of similar texts are aggregated into one granular ball by using granular ball clustering to obtain the granular ball space, then semantic vector representations of to-be-classified texts are mapped into the granular ball space to obtain the corresponding granular ball centers, and the influence of adversarial samples is eliminated by using the corresponding granular ball centers for classification, so that compared with the prior art, the method has the advantages that the classification efficiency is improved, and the classification efficiency is improved. And the accuracy of the classifier under the attack condition of the model is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to an adversarial attack text classification method based on granular sphere and metric learning. Background Art

[0002] In the field of natural language processing (NLP), text classification is a core task and is widely used in many scenarios such as information retrieval, sentiment analysis, and spam filtering.

[0003] Traditional text classification methods are usually based on feature extraction and machine learning models, such as Naive Bayes classifier, Support Vector Machine (SVM), and decision tree. However, with the growth of complexity and scale of text data, traditional methods often appear to be unable to cope with high-dimensional, sparse data and diverse text. Especially when facing attack samples, traditional classification methods may not only be affected, resulting in inaccurate prediction results, but also weaken the model's overall understanding of the text.

[0004] At present, there is no classification method that can effectively resist the interference of attackers, so the classification performance in the presence of attacks is still poor. Therefore, it is urgent to explore new methods to enhance the robustness and accuracy of text classification models in the face of attacks. Summary of the invention

[0005] In order to solve the above-mentioned prior art problems, the present invention adopts a text classification method for adversarial attack based on granular sphere and metric learning, comprising: obtaining a text to be classified, inputting the text to be classified into a trained encoding module, and obtaining a semantic vector representation of the text to be classified; mapping the semantic vector representation of the text to be classified into a trained granular sphere space, obtaining a granular sphere center, and inputting the semantic vector representation corresponding to the granular sphere center into a classifier to obtain a classification result; the training process of the encoding module, the granular sphere space and the classifier comprises:

[0006] S1. Obtain a text data set, preprocess the text in the text data set, and obtain the preprocessed text;

[0007] S2, input the preprocessed text into the encoding module for encoding to obtain the semantic vector representation of the text;

[0008] S3, using granular calculation to perform supervised clustering on the semantic vector representations of all texts to generate a granular space, which includes multiple granular balls and their corresponding granular ball centers;

[0009] S4, inputting the semantic vectors corresponding to the center of the sphere where each text is located into the classifier to obtain the classification result of each text;

[0010] S5. Construct sample pairs for each text based on the semantic vector representation of all texts, calculate the loss function value based on the sample pairs and classification results of each text, update the parameters of the encoding module and classifier based on the loss function value, and obtain the final trained encoding module, granular space and classifier when the loss function value is minimized.

[0011] Preprocess the text in the text dataset:

[0012] S11, constructing regular expression rules, and using the regular expression rules to denoise the text in the text data set;

[0013] S12, performing word segmentation processing on the denoised text to obtain a preprocessed text.

[0014] The encoding module encodes the preprocessed text including:

[0015] S21, using the pre-trained Bert model to perform deep semantic feature extraction on the pre-processed text to obtain a feature vector of the text;

[0016] S22. Pool the feature vectors of the text, and normalize the pooled results to obtain a semantic vector representation of the text.

[0017] Supervised clustering of the semantic vector representations of all texts includes:

[0018] S31, combining the semantic vector representations of all texts into a sphere, and setting a purity threshold and a quality threshold;

[0019] S32. Randomly select a ball o from all the balls g , calculate the particle sphere o g The purity of the particles is determined by g Is the purity less than the purity threshold? If so, g Execute step S33; otherwise, calculate the particle o g The quality of the ball g Whether the quality reaches the quality threshold, if so, execute step S35; otherwise, execute step S34;

[0020] S33, traverse the particle ball o g Each semantic vector in represents the corresponding category label, and the number of labels k is obtained. The particles o are clustered by K-means clustering algorithm. g Divide into k spheres, and then execute step S32;

[0021] S34, aggregate the spheres whose qualities do not reach the quality threshold and have the same label into one sphere, and then execute step S32;

[0022] S35, determine whether the mass of all the spheres reaches the mass threshold, if so, combine all the spheres into a sphere space; otherwise, the spheres o g The remaining balls remain unchanged and step S32 is executed for the remaining balls.

[0023] The purity of a ball is the proportion of semantic vectors in the ball that are the same as the ball label, and the ball label is the label with the most corresponding semantic vectors in the ball.

[0024] The sample pairs for constructing text i include:

[0025] S41. Perform data enhancement on the semantic vector representations of M texts to obtain M enhanced texts, and divide the semantic vector representations of M texts into S balls according to the labels. s , calculate each ball o s The center of the ball o is S, the number of labels in the text dataset, and s is s The index of

[0026] S42, the ball o with the same label as the text i s The semantic vector representations of all texts in and their enhanced texts are combined into a positive sample space, and the ball o with different labels from text i is s A negative sample space composed of the semantic vector representations of all texts in;

[0027] S43, according to the positive sample space, the negative sample space and the sphere o s The center of constructs the sample pair of text i.

[0028] The sample pairs for each text are:

[0029]

[0030] in, A collection of sample pairs representing text, <t i ,m i ,n i > is a sample pair of text i, t i 、m i 、n i are all triplets, which are subsample pairs of sample pairs of text i, x i is the semantic vector representation of text i, is the semantic vector representation of the positive sample space of text i, is the semantic vector representation in the negative sample space of text i, For the same ball o as the text i label s The semantic vector representation of the center of For a ball o that is different from the text i label s The semantic vector representation of the center of is the enhanced text of text i, and M is the number of texts.

[0031] The loss function Loss is:

[0032]

[0033] Among them, y i is the label of text i, p i is the classification result of text i, L is the metric loss function, and γ is the index of the sample pair.

[0034] The metric loss function is:

[0035] L(γ)=L(t i ,m i ,n i )=θ×d(t i ,m i )-μ×d(t i ,n i )

[0036] Among them, θ and μ are hyperparameters, and d represents the cosine distance.

[0037] Mapping the semantic vector representation of the text to be classified to the trained sphere space includes: calculating the metric distance between the semantic vector representation of the text to be classified and the semantic vector representation corresponding to the sphere center in the trained sphere space, and selecting multiple sphere centers with the smallest metric distance.

[0038] Beneficial effects of the present invention:

[0039] 1. The present invention uses sphere clustering to aggregate the semantic vector representations of similar texts into a sphere to obtain a sphere space, and then maps the semantic vector representation of the text to be tested into the sphere space to obtain the corresponding sphere center. The influence of adversarial samples is eliminated by using the corresponding sphere center for classification. Compared with the prior art, the accuracy of the classifier of the model under attack is improved; 2. While optimizing the classifier, the present invention optimizes the semantic vector representation of the text through comparative learning, and then optimizes the sphere space, which effectively reduces the interference of adversarial samples on the model and improves the stability of the model under harsh conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a text classification method for adversarial attack based on granular sphere and metric learning according to the present invention;

[0041] Figure 2 This is a schematic diagram of the results of a text classification method for adversarial attacks based on granular spheres and metric learning according to the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The present invention provides a text classification method for adversarial attack based on granular sphere and metric learning, such as Figure 1 As shown, the method includes the following steps: obtaining the text to be classified, inputting the text to be classified into a trained encoding module, and obtaining a semantic vector representation of the text to be classified; mapping the semantic vector representation of the text to be classified into a trained sphere space, obtaining the sphere center, and inputting the semantic vector representation corresponding to the sphere center into a classifier to obtain a classification result.

[0044] Specifically, the mapping method is to calculate the metric distance between the semantic vector representation of the text to be classified and the semantic vector representation corresponding to the sphere center in the trained sphere space, and select r sphere centers with the smallest metric distance. r is a hyperparameter, and different magnitudes of r values ​​are set according to different data sets, generally set to 4-300; the semantic vector representations corresponding to the r sphere centers are input into the classifier respectively to obtain the classification results of the r sphere centers, and the classification results of the r sphere centers are voted, and the classification result with the highest vote rate is taken as the final classification result.

[0045] The training process of the encoding module, granular space and classifier includes:

[0046] S1. Obtain a text dataset, preprocess the text in the text dataset to obtain preprocessed text; the text dataset includes M texts and their labels;

[0047] S11. Use crawler technology and other methods to obtain text data sets;

[0048] At this time, the text dataset may include a large number of illegal characters and redundant symbols, which belong to text noise; therefore, the text dataset needs to be denoised;

[0049] S12, constructing a regular expression rule, and denoising the text according to the regular expression rule;

[0050] Text denoising based on regular expression rules includes: lowercase text, replacing web page tags in text with empty characters, such as , remove repeated punctuation marks, remove non-alphanumeric and non-punctuation symbols; this regular expression rule includes: use ^[0-9] to match numbers, use ^[\u4e00-\u9fa5] to match Chinese characters, use ^[A-Za-z] to match English letters, etc.

[0051] S13, performing word segmentation processing on the denoised text;

[0052] Specifically, it includes: removing stop words, further optimizing text representation through stem extraction or word form restoration to enhance semantic consistency, and setting topic labels for each data according to its semantic features.

[0053] Specifically, different word segmentation tools are used for different text data sets. If the text data is English text, the NLTK package in the Python library is used for word segmentation; if the text data is Chinese text, the jieba library can be used for word segmentation.

[0054] S2, input the preprocessed text into the encoding module for encoding to obtain the semantic vector representation of the text;

[0055] S21. Input the preprocessed text into the pre-trained Bidirectional Encoder Representations from Transformers (Bert) model. The transformer structure of the Bert model will perform deep semantic feature extraction on the text to obtain the feature vector of the text, i.e., the hidden layer representation;

[0056] S22. Use a pooling method (average pooling or maximum pooling) to aggregate the feature vectors output by Bert to reduce the feature dimension; then normalize the pooled feature vectors and save the feature representation, as follows:

[0057] Normalization: Normalize the feature vector, usually using L2 normalization, to convert the feature vector into a unit vector to reduce the impact of feature differences on subsequent clustering.

[0058] Feature representation preservation: The normalized feature vector is saved as the semantic vector representation of the text for subsequent processing.

[0059] In one embodiment, large-scale text data is processed in batches, each batch including an appropriate amount of text to avoid memory overflow.

[0060] S3, using granular calculation to perform supervised clustering on the semantic vector representations of all texts to generate a granular space, which includes multiple granular balls and their corresponding granular ball centers;

[0061] Specifically, step S3 uses the granular purity adaptive method to cluster the semantic vector representations of all texts to obtain a sentence set, including:

[0062] S31, combining the semantic vector representations of all texts into a sphere, and setting a purity threshold and a quality threshold;

[0063] S32. Randomly select a ball o from all the balls g , calculate the particle sphere o g The purity of the particles is determined by g Is the purity less than the purity threshold? If so, g Execute step S33; otherwise, calculate the particle o g The quality of the ball g Whether the quality reaches the quality threshold, if so, execute step S35; otherwise, execute step S34;

[0064] S33, traverse the particle ball o g Each semantic vector in represents the corresponding category label, and the number of labels k is obtained. The particles o are clustered by K-means clustering algorithm. g Divide into k spheres, and then execute step S32;

[0065] S34, aggregate the spheres whose qualities do not reach the quality threshold and have the same label into one sphere, and then execute step S32;

[0066] S35, determine whether the mass of all the spheres reaches the mass threshold, if so, combine all the spheres into a sphere space; otherwise, the spheres o g The remaining balls remain unchanged and step S32 is executed for the remaining balls.

[0067] Preferably, the purity threshold is usually set to be above 0.85, and the quality threshold is usually set to be above 2.

[0068] The formulas for calculating the purity, mass and center of the pellet are as follows:

[0069]

[0070] l GB =max(l1,l2,…,l n ),

[0071]

[0072] Among them, GB is a sphere, x j is the semantic vector representation of the jth text in the sphere, c GB is the center of the sphere, i.e., the average value of the semantic vector representation of all texts in the sphere, l GB is the label of the ball. The label of the ball is the label that appears most frequently in the ball. (l1,l2,…,l n) is the label corresponding to the semantic vector representation of the text, D is the set of all texts j in the sphere, n is the number of semantic vector representations of the text in the sphere, the mass of the sphere is the number n of semantic vector representations in the sphere, p GB is the purity of the ball, that is, the proportion of semantic vectors in the ball that are the same as the ball label, and label(j) represents the label of text j.

[0073] S4, inputting the semantic vectors corresponding to the center of the sphere where each text is located into the classifier to obtain the classification result of each text;

[0074] For the initial sphere text clustering, the vector information of the text depends entirely on the text semantic cognition learned by the pre-trained Bert encoder pre-training. There is no semantic cognition specific to the classification task yet. Therefore, the sphere center obtained by the sphere adaptive method cannot well represent the information contained in its own category, nor can it adapt to the attack method. Therefore, the present invention trains the model in combination with specific classification tasks and attack conditions, and uses metric learning to enable the model to deeply recognize the semantic similarity features of similar texts and the semantic distinction features of heterogeneous texts. The specific training process includes:

[0075] S41. Perform data enhancement on the semantic vector representations of M texts to obtain M enhanced texts, and divide the semantic vector representations of M texts into S balls according to the labels. s , calculate each ball o s The center of the ball o is S, the number of labels in the text dataset, and s is s The index of each ball o s The center of is the average value of the semantic vector representation of all texts in the ball;

[0076] S42, the ball o with the same label as the text i s The semantic vector representations of all texts in and their enhanced texts are combined into a positive sample space, and the ball o with different labels from text i is s A negative sample space composed of the semantic vector representations of all texts in;

[0077] S43, according to the positive sample space, the negative sample space and the sphere o s The center of constructs the sample pair of text i.

[0078] S5. Construct sample pairs for each text based on the semantic vector representation of all texts, calculate the loss function value based on the sample pairs and classification results of each text, update the parameters of the encoding module and classifier based on the loss function value, and obtain the final trained encoding module, granular space and classifier when the loss function value is minimized.

[0079] Specifically, the training loss is calculated as follows:

[0080]

[0081] L(t i ,m i ,n i )=θ×d(t i ,m i )-μ×d(t i ,n i )

[0082]

[0083] in, represents a set of text sample pairs, <t i ,m i ,n i > is a sample pair, triple of text i Triple Triple x i is the semantic vector representation of text i, is the semantic vector representation of the positive sample space of text i, is the semantic vector representation in the negative sample space of text i, For the same ball o as the text i label s The semantic vector representation of the center of For a ball o that is different from the text i label s The semantic vector representation of the center of is the enhanced text of text i, y i is the label of text i, o i is the classification result of text i, L is the metric loss function, γ is the index of the sample pair, θ and μ are hyperparameters, and d represents the cosine distance.

[0084] During the training process, the parameters of the model optimized by reverse parameter transfer and the text semantic vector representation generated by encoding will be used. At the same time, the optimized vectors will be re-clustered by sphere adaptive clustering to continuously iterate and update the spheres used for mapping, so as to achieve the effect of spheres adapting to the text.

[0085] like Figure 2 As shown in the figure, after a normal sample is attacked by the attack model, an adversarial sample will be obtained. The adversarial sample modifies some words on the basis of the original sample, and does not affect the negative emotion of the sentence as a whole. However, after the adversarial sample is input into the original language model, the original language model will make a wrong prediction and predict the sample as a positive emotion. The model of the present invention will defend against the disturbance of the adversarial sample and correctly predict the sample as a negative emotion.

[0086] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A text classification method for adversarial attacks based on granular sphere and metric learning, characterized in that: include: Obtain the text to be classified, input the text to be classified into the trained encoding module, and obtain the semantic vector representation of the text to be classified; map the semantic vector representation of the text to be classified to the trained sphere space, obtain the sphere center, and input the semantic vector representation corresponding to the sphere center into the classifier to obtain the classification result; The training process of the encoding module, granular space and classifier includes: S1. Obtain a text data set, preprocess the text in the text data set, and obtain the preprocessed text; S2, input the preprocessed text into the encoding module for encoding to obtain the semantic vector representation of the text; S3, using granular calculation to perform supervised clustering on the semantic vector representations of all texts to generate a granular space, which includes multiple granular balls and their corresponding granular ball centers; S4, inputting the semantic vectors corresponding to the center of the sphere where each text is located into the classifier to obtain the classification result of each text; S5. Construct sample pairs for each text based on the semantic vector representation of all texts, calculate the loss function value based on the sample pairs and classification results of each text, update the parameters of the encoding module and classifier based on the loss function value, and obtain the final trained encoding module, granular space and classifier when the loss function value is minimized.

2. The method for adversarial attack text classification based on granular sphere and metric learning according to claim 1, characterized in that: Preprocess the text in the text dataset: S11, constructing regular expression rules, and using the regular expression rules to denoise the text in the text data set; S12, performing word segmentation processing on the denoised text to obtain a preprocessed text.

3. The method for adversarial attack text classification based on granular sphere and metric learning according to claim 1, characterized in that: The encoding module encodes the preprocessed text including: S21, using the pre-trained Bert model to perform deep semantic feature extraction on the pre-processed text to obtain a feature vector of the text; S22. Pool the feature vectors of the text, and normalize the pooled results to obtain a semantic vector representation of the text.

4. The method for text classification against attacks based on granular sphere and metric learning according to claim 1, characterized in that: Supervised clustering of the semantic vector representations of all texts includes: S31, combining the semantic vector representations of all texts into a sphere, and setting a purity threshold and a quality threshold; S32. Randomly select a ball o from all the balls g , calculate the particle sphere o g The purity of the particles is determined by g Is the purity less than the purity threshold? If so, g Execute step S33; otherwise, calculate the particle o g The quality of the ball g Whether the quality reaches the quality threshold, if so, execute step S35; otherwise, execute step S34; S33, traverse the particle ball o g Each semantic vector in represents the corresponding category label, and the number of labels k is obtained. The particles o are clustered by K-means clustering algorithm. g Divide into k spheres, and then execute step S32; S34, aggregate the spheres whose qualities do not reach the quality threshold and have the same label into one sphere, and then execute step S32; S35, determine whether the mass of all the spheres reaches the mass threshold, if so, combine all the spheres into a sphere space; otherwise, the spheres o g The remaining balls remain unchanged and step S32 is executed for the remaining balls.

5. The method for text classification against attacks based on granular sphere and metric learning according to claim 4, characterized in that: The purity of a ball is the proportion of semantic vectors in the ball that are the same as the ball label, and the ball label is the label with the most semantic vectors in the ball.

6. The method for text classification against adversarial attacks based on granular sphere and metric learning according to claim 1, characterized in that: The sample pairs for constructing text i include: S41. Perform data enhancement on the semantic vector representations of M texts to obtain M enhanced texts, and divide the semantic vector representations of M texts into S balls according to the labels. s , calculate each ball o s The center of the ball o is S, the number of labels in the text dataset, and s is s The index of S42, the ball o with the same label as the text i s The semantic vector representations of all texts in and their enhanced texts are combined into a positive sample space, and the ball o with different labels from text i is s A negative sample space composed of the semantic vector representations of all texts in; S43, according to the positive sample space, the negative sample space and the sphere o s The center of constructs the sample pair of text i.

7. The method for text classification against adversarial attacks based on granular sphere and metric learning according to claim 6, characterized in that: The sample pairs for each text are: in, A collection of sample pairs representing text, <t i ,m i ,n i > is a sample pair of text i, t i 、m i 、n i are all triplets, which are subsample pairs of sample pairs of text i, x i is the semantic vector representation of text i, is the semantic vector representation of the positive sample space of text i, is the semantic vector representation in the negative sample space of text i, The same ball as the text i label o s The semantic vector representation of the center of For a ball o that is different from the text i label s The semantic vector representation of the center of is the enhanced text of text i, and M is the number of texts.

8. The method for text classification against attacks based on granular sphere and metric learning according to claim 7, characterized in that: The loss function Loss is: Among them, y i is the label of text i, p i is the classification result of text i, L is the metric loss function, and γ is the index of the sample pair.

9. The method for text classification against attacks based on granular sphere and metric learning according to claim 8, characterized in that: The metric loss function is: L(γ)=L(t i ,m i ,n i )=θ×d(t i ,m i )-μ×d(t i ,n i ) Among them, θ and μ are hyperparameters, and d represents the cosine distance.

10. The method for text classification against adversarial attacks based on granular sphere and metric learning according to claim 1, characterized in that: Mapping the semantic vector representation of the text to be classified to the trained sphere space includes: calculating the metric distance between the semantic vector representation of the text to be classified and the semantic vector representation corresponding to the sphere center in the trained sphere space, and selecting multiple sphere centers with the smallest metric distance.

Images