A text adversarial sample detection method based on automaton interpretation

Abstract

The application discloses a kind of text adversarial sample detection methods based on automaton interpretation, belong to text processing technical field, including constructing original classification model;According to probability learning algorithm, abstract probability automaton is generated from original network model;Adversarial sample is generated based on test data set, is added to test set, constitutes the mixed data set to be screened;The mixed data set to be screened is input into original network model, and then the running track of sample on probability automaton is obtained, the running information of sample on automaton is brought into index calculation formula, and the sample is screened according to index result.The application calculates the information that sample runs on the proxy model-probability automaton of recurrent neural network model, measures the suspicious degree of sample being adversarial sample by suspiciousness index, and can effectively screen out adversarial sample in sample set by sorting sample suspiciousness index.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text processing, and more particularly to a text adversarial sample detection method based on automaton interpretation. BACKGROUND

[0002] Due to the excellent performance of neural network models in solving many real-world problems, such as sentiment analysis and machine translation, they are becoming more and more popular. At the same time, neural networks are proven to be vulnerable to problems such as adversarial attacks and embedded backdoors. The erroneous behavior produced by RNN-driven systems can lead to misinterpretation, so the testing and optimization of RNN-driven systems have become an urgent and challenging task.

[0003] RNN-driven systems are built on a data-driven programming paradigm, which requires a large amount of data and manual labeling for model training and evaluation. Collecting a high-quality dataset for building an RNN-driven system requires a large amount of manpower for labeling, which makes the process both expensive and time-consuming. It is inefficient and impractical to use data collected from use scenarios directly for model testing and optimization. Because in a large amount of data, only a small number of cases can trigger the potential errors of the system, and these cases are particularly critical for testing RNN-driven systems. In addition, due to the nature of RNN models, optimizing them through retraining of huge data often requires a large amount of time and computing resources.

[0004] There have been many works on test sample screening for RNN systems, and good results have been achieved. For example, DeepStellar adapted five coverage criteria from Deep Gauge to test and analyze RNN models, which is first converted into a discrete-time Markov chain (DTMC) as an abstraction. TestRNN proposes a series of neuron coverage metrics for LSTM networks and develops a coverage-guided fuzzing method for deep learning models with LSTM network structure. Then, some random mutations are designed to enhance the coverage age to generate test cases. RNN-Test defines the hidden state coverage as the ratio of reaching the maximum value of all hidden states during the test process. DeepState analyzes the change rate and trend of the hidden state in the RNN model, and relies on the relationship between different cases to effectively select tests in a large-scale unlabeled dataset.

[0005] However, these works are all based on natural noise samples, which are naturally aggregated at the classification edge. Adversarial samples generated using attack methods can be distributed at the classification edge or the classification center according to the purpose of the attacker. Adversarial samples distributed at the classification edge can be screened out by the method of screening natural noise samples, while it is more difficult to screen out adversarial samples distributed at the classification center, and there is currently no work in this regard.

[0006] Meanwhile, in the field of image classification, adversarial samples are intentionally synthesized images that look almost identical to the original images but can mislead the classifier to provide incorrect prediction outputs. As for the text field, practical systems such as spam detection, harmful text detection, and malware detection have deployed deep learning models on a large scale, and security is particularly important for these systems. Compared with the image field, the defense research of adversarial attacks in the text field is far from enough.

[0007] Therefore, it is an urgent problem for those skilled in the art to provide a method capable of detecting adversarial samples of text data. SUMMARY

[0008] Therefore, the present application provides a text adversarial sample detection method based on automaton interpretation, which provides a detection method for text adversarial sample defense of deep learning model. The method first extracts a interpretable probabilistic automaton model from an RNN language model as an equivalent proxy model of the RNN, and realizes the screening of test samples through the state transition of the probabilistic automaton on the test set.

[0009] In order to achieve the above purpose, the present application provides the following technical scheme:

[0010] A text adversarial sample detection method based on automaton interpretation, comprising the following steps:

[0011] Step 1: obtaining a text sample and constructing a text sample test set;

[0012] Step 2: constructing a recurrent neural text network classification model, and training an original classification network model through the text sample test set;

[0013] Step 3: generating a probabilistic automaton from the original classification network model according to a probabilistic learning algorithm;

[0014] Step 4: generating adversarial samples of the original classification network model based on the text sample test set, adding the adversarial samples to the text sample test set to form a mixed data set to be screened;

[0015] Step 5: inputting the mixed data set to be screened into the original classification network model, obtaining the running track of the text sample of the mixed data set on the probabilistic automaton, and obtaining the corresponding index result according to the running track; screening the text sample in the mixed data set to be screened according to the index result to obtain a target adversarial sample.

[0016] Preferably, the step 3 specifically comprises:

[0017] Step 3.1: re-input the text sample test set into the trained original classification network model, record the hidden state of the text sample, and the vector of the hidden state is ;

[0018] Step 3.2: K-means clustering operation is performed on the hidden state to obtain the abstract trajectory of the text sample;

[0019] Step 3.3: constructing a probability prefix tree according to the abstract trajectory of the text sample;

[0020] Step 3.4: calculating the distance between each pair of hidden states on the probability prefix tree, and merging the hidden states with a distance lower than a preset threshold to obtain a probability automaton.

[0021] Preferably, the abstract trajectory of the text sample is obtained, specifically including:

[0022] The classes of the hidden state clustering are numbered and processed to construct an abstract alphabet, and the abstract trajectory of each text sample is formed according to the numbering of the abstract alphabet, that is, C(s1), C(s1), … C(s n ), wherein s0 is a hypothetical initial state, is a clustering function:

[0023] .

[0024] Preferably, the step 2.3 specifically includes:

[0025] The abstract trajectory of each text sample is constructed as a tree with s0 as the initial tree root and the abstract alphabet as the path;

[0026] The total path number and the path number of each abstract letter emitted by each hidden state are counted to obtain the probability of each path and obtain the probability prefix tree.

[0027] Preferably, the step 4 specifically includes:

[0028] Step 4.1: selecting multiple adversarial sample generation methods from TextAttack to generate a large number of adversarial texts based on the text sample test set;

[0029] Step 4.2: removing adversarial samples with a confidence lower than a preset value in the generation process, and adding adversarial samples with a confidence not lower than the preset value to the text sample test set to form a mixed data set to be screened.

[0030] Preferably, the step 5 specifically includes:

[0031] Step 5.1: input the mixed data set to be screened into the original classification network model to obtain an abstract trajectory of the mixed data set to be screened;

[0032] Step 5.2: display the abstract trajectory in the probabilistic automaton, and calculate three screening index values of the text sample of the mixed data set to be screened through the running information of the abstract trajectory in the probabilistic automaton;

[0033] Step 5.3: set a weight for each index, and combine the three screening indexes of the sample into one index data;

[0034] Step 5.4: compare the index value of each text sample with a threshold value, and the one less than the threshold value is the target adversarial sample.

[0035] Preferably, the three screening indexes are path average probability, state jump rate and error step accumulation rate.

[0036] Preferably, the path average probability is defined as:

[0037]

[0038] wherein state_pro i is the probability of each step jump of the abstract trajectory of the text sample, and N is the sequence length of the abstract trajectory of the text sample.

[0039] Preferably, the state jump rate is defined as:

[0040]

[0041]

[0042] wherein state_trace is the state set of the first i-1 steps of the abstract trajectory of the text sample on the probabilistic automaton, statei is the state of the i step jump, and N is the sequence length of the abstract trajectory of the text sample.

[0043] Preferably, the error step accumulation rate is defined as:

[0044]

[0045]

[0046] wherein state pro is the probability of the i step of the abstract trajectory of the text sample on the probabilistic automaton, and state_trace i is the probability sequence of the probabilistic automaton from the current state to the next state.

[0047] Compared with the prior art, the technical scheme has the following beneficial effects,

[0048] 1. The prior art scheme does not have a detection scheme for the adversarial samples of the recurrent neural network, and the test work for the RNN is only to use the test scheme suitable for the DNN model after modification, without considering the characteristics of the RNN model. The present application can detect the adversarial samples on the basis of the interpretation work of the RNN model.

[0049] 2. The present application has universality and can be used for all recurrent neural network models.

[0050] 3. The present application calculates the information of the sample running on the proxy model-probabilistic automaton of the recurrent neural network model, calculates the path average probability, state jump rate and suspicious path accumulation rate of the sample on the probabilistic automaton as the suspicious degree index of the sample to measure the suspicious degree of the sample being an adversarial sample, and sorts the suspicious degree index of the sample to effectively screen out the adversarial samples in the sample set. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical schemes in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0052] Figure 1 Flow chart of the RNN test example screening process based on the automaton interpretation in the present application;

[0053] Figure 2 Flowchart of the probabilistic automaton extraction in the present application;

[0054] Figure 3 Flowchart of the adversarial sample detection in the present application. DETAILED DESCRIPTION

[0055] The technical schemes in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0056] For the explanation of RNN, the work in recent years mainly focuses on the direction of extracting proxy models from it. Researchers do not directly understand and reason the neural network, but aim to extract simpler models from the neural network. Ideally, the simpler model will accurately approximate the neural network, and it will be simple enough that it can be understood by humans. In addition, such models can accept automatic system analysis techniques, such as model-based testing, model checking and runtime monitoring. We use this method in the test case screening of RNN system, and propose a text adversarial sample detection method based on automata explanation. The method first extracts a probabilistic automaton model that can be explained from the RNN language model as an equivalent proxy model of the RNN, and realizes the screening of test cases through the state transition of the automaton on the test set.

[0057] The flow of a text adversarial sample detection method based on automata explanation provided by the embodiment of the application is shown in Figure 1 As shown in the figure, there are mainly four steps: 1. Construct different kinds of recurrent neural text network classification models, and train to obtain the original classification network model. 2. According to the probability learning algorithm, abstract a probabilistic automaton from the original network model. 3. Based on the existing toolkit, generate adversarial samples based on the test data set, add them to the test set, and form a mixed data set to be screened. 4. Input the test set to be screened into the original network, and then obtain the running track of the sample on the probabilistic automaton. The running information of the sample on the automaton is brought into the index calculation formula, and the sample is screened according to the index result.

[0058] The embodiment of the application discloses a text adversarial sample detection method based on automata explanation, which comprises the following steps:

[0059] Step 1: Obtain text samples and construct a text sample test set.

[0060] Step 2: Construct a recurrent neural text network classification model, and train to obtain an original classification network model through a text sample test set.

[0061] Text data is an abstract expression of human semantics and cannot be recognized and processed by a computer. The processing of text data is relatively complex and needs to be encoded into a numerical input that can be processed. First, the data is cleaned. In order to have more extensive representation, we select three text data sets, namely AGNEWS news data set, IDMP movie review sentiment classification data set and SNIPS intent recognition data set. Based on the text data, different kinds of recurrent neural network classification models are constructed, and the three text data are input into different recurrent neural network models for training to obtain the original classification network model. The specific steps are as follows:

[0062] Step 2.1: Clean the text data, remove single characters and numbers, convert all to lowercase, split the text into individual words, and then input the processed text corpus into the word encoding model Word2Vec to obtain word vector encoding that can be processed by a computer. Each sample is encoded using word groups to form a vector;

[0063] Step 2.2: Input the data into three network models based on different basic units, calculate the loss function, update the model parameters, and repeat this step for a total of 20 epochs. The three basic units include SRNN, LSTM, and GRU.

[0064] Step 2.3: Map the output vector to the data category to obtain the category probability, and sort the probability to obtain the classification result.

[0065] Step 3: Generate a probabilistic automaton from the original classification network model based on the probability learning algorithm.

[0066] Since RNN is difficult to understand, some researchers have tried to extract an approximate proxy model from the complex recurrent neural network for testing. We use a probabilistic automaton as a proxy model.

[0067] Step 4: Generate adversarial samples for the original classification network model based on the text sample test set. Add the adversarial samples to the text sample test set to form a mixed data set to be screened.

[0068] There are many methods for generating adversarial samples of text data, and our method is not limited to certain specific adversarial methods. Therefore, we use TextAttack, a natural language processing model adversarial attack framework. Based on the adversarial algorithm in it, we generate a batch of adversarial samples with high confidence.

[0069] Step 5: Input the mixed data set to be screened into the original classification network model to obtain the running trajectory of the text sample in the probabilistic automaton, and then input the running trajectory into the index calculation formula to obtain the index result of the text sample. According to the index result, the text samples in the mixed data set to be screened are screened to obtain the target adversarial sample.

[0070] In order to screen out the target sample, three indicators can be set according to the characteristics of the probabilistic automaton to obtain the running information of the sample in the automaton. By reading the running information, the target sample is obtained.

[0071] In another embodiment, as shown in Figure 2 the specific steps of step 3 are:

[0072] Step 3.1: re-input the text sample test set into the trained original classification network model, record the hidden state of the text sample, and the vector of the hidden state is ;

[0073] Step 3.2: K-means clustering operation is performed on the hidden state to obtain the abstract trajectory of the text sample;

[0074] Step 3.3: construct a probability prefix tree according to the abstract trajectory of the text sample;

[0075] Step 3.4: calculate the distance between each pair of hidden states on the probability prefix tree, and merge the hidden states with a distance lower than a preset threshold to obtain a probability automaton.

[0076] The distance between each pair of states on the probability prefix tree is set, and the states with a distance lower than the threshold are merged. This operation is repeated. The distance between each pair of states is measured by the similarity of the future probability distribution. The probability difference of all paths of two states in the probability prefix tree is compared, and whether the difference is within a certain range is calculated.

[0077] The path probability is defined as the product of the probabilities of single-step paths. For example, the probability of the path π = 〈σ1,σ2, σ k 〉 from state A to state B is defined as:

[0078] P(A,π) = P(A,A·σ1)·P(A·σ1,σ2) P(A·σ1 σk-1,σk);

[0079] Two states are defined as similar in future probability if the probability difference of the same path from the two states is less than a threshold:

[0080]

[0081] Two states A and B that meet the above conditions can be merged.

[0082] Preferably, the abstract trajectory of the text sample is obtained, which specifically includes:

[0083] The classes of the clustered hidden states are numbered, an abstract alphabet is constructed, and the abstract trajectory of each text sample is formed according to the numbering of the abstract alphabet, i.e. C(s1), C(s2), … C(s n ), where s0 is a hypothetical initial state, is a clustering function:

[0084] .

[0085] Preferably, step 3.3 specifically comprises:

[0086] An abstract trajectory of each text sample is constructed into a tree with abstract alphabet as path, with s0 as the initial tree root;

[0087] The total path number and the path number of each abstract letter emitted by each hidden state are counted to obtain the probability of each path, and a probability prefix tree is obtained.

[0088] Preferably, step 4 specifically comprises:

[0089] Step 4.1: Select multiple adversarial sample generation methods from TextAttack, and generate a large number of adversarial texts based on the text sample test set;

[0090] Step 4.2: Remove adversarial samples with a confidence lower than a preset value during generation, and add adversarial samples with a confidence not lower than the preset value to the text sample test set to form a mixed data set to be screened. The preset value can be set to 0.8.

[0091] Preferably, as shown in Figure 3 Step 5 specifically comprises:

[0092] Step 5.1: input the mixed data set to be screened into the original classification network model to obtain the abstract trajectory of the mixed data set to be screened;

[0093] Step 5.2: display the abstract trajectory of the mixed data set to be screened in the probabilistic automaton, and calculate the three screening index values of the text samples in the mixed data set to be screened through the running information in the probabilistic automaton;

[0094] Step 5.3: set a weight for each index, and combine the three screening indexes of the sample into one index data;

[0095] Step 5.4: compare the index value of each text sample with a threshold value, and the one less than the threshold value is the target adversarial sample.

[0096] Preferably, the three screening indexes are path average probability, state jump rate and error step accumulation rate.

[0097] Preferably, the path average probability is defined as:

[0098]

[0099] Wherein, state_pro i is the probability of each jump of the abstract trajectory of the text sample, and N is the sequence length of the abstract trajectory of the text sample.

[0100] Preferably, the state jump rate is defined as:

[0101]

[0102]

[0103] wherein state trace is the state set of the text sample abstract trace at i-1 step on the probabilistic automaton, statei is the state of the i step jump, and N is the length of the abstract trace sequence of the text sample.

[0104] Preferably, the error step accumulation rate is defined as:

[0105]

[0106]

[0107] wherein state pro is the probability of the i step on the abstract trace probabilistic automaton of the text sample, state trace i is the probability sequence of the probabilistic automaton from the current state to the next state.

[0108] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the apparatus disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0109] The above description of disclosed embodiments enables those skilled in the art to carry out or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text adversarial example detection method based on automata interpretation, characterized in that, Includes the following steps: Step 1: Obtain text samples and construct a text sample test set; Step 2: Construct a recurrent neural text classification model, and obtain the original classification network model by training it on the text sample test set; Step 3: Generate a probabilistic automaton from the original classification network model using a probabilistic learning algorithm; Step 4: Generate adversarial examples of the original classification network model based on the text sample test set, and add the adversarial examples to the text sample test set to form a mixed dataset to be screened; Step 5: Input the mixed dataset to be screened into the original classification network model, obtain the running trajectory of the text samples of the mixed dataset on the probabilistic automaton, and obtain the corresponding index results of the text samples based on the running trajectory; screen the text samples in the mixed dataset to be screened based on the index results to obtain the target adversarial samples; Step 5 specifically includes: Step 5.1: Input the mixed dataset to be filtered into the original classification network model to obtain the corresponding abstract trajectory; Step 5.2: Display the abstract trajectory in the probabilistic automaton, and calculate the three screening index values ​​of the text samples in the mixed dataset to be screened based on the running information of the abstract trajectory in the probabilistic automaton; Step 5.3: Assign weights to each indicator and merge the three screening indicators of the sample into one indicator data; Step 5.4: Compare the index value of each text sample with the threshold; those less than the threshold are the target adversarial samples. The three screening indicators are the average path probability, state transition rate, and cumulative error rate, respectively. The average probability of the path is defined as: Among them, state_pro i is the probability of each step jump in the abstract trajectory of the text sample, and N is the length of the abstract trajectory sequence of the text sample; The state transition rate is defined as: Where `state_trace` is the set of states on the probabilistic automaton for the first i-1 steps of the abstract trajectory of the text sample. i The state is the i-th step jump, and N is the length of the abstract trajectory sequence of the text sample; The cumulative error rate is defined as: Where state pro It is the probability at the i-th step on the abstract trajectory probabilistic automaton of the text sample, state_trace i It is the probability sequence of a probabilistic automaton moving from the current state to the next state.

2. The text adversarial example detection method based on automata interpretation according to claim 1, characterized in that, Step 3 specifically includes: Step 3.1: Re-feed the text sample test set into the pre-trained original classification network model, and record the hidden states of the text samples. The vector of the hidden states is... ; Step 3.2: Perform K-means clustering on the hidden state to obtain the abstract trajectory of the text sample; Step 3.3: Construct a probability prefix tree based on the abstract trajectory of the text sample; Step 3.4: Calculate the distance between each pair of hidden states on the probability prefix tree, and merge the hidden states whose distance is lower than a preset threshold to obtain a probabilistic automaton.

3. The text adversarial example detection method based on automata interpretation according to claim 2, characterized in that, The abstract trajectory of obtaining the text sample specifically includes: The categories of the hidden state clusters are numbered, an abstract alphabet is constructed, and an abstract trajectory for each text sample is formed according to the numbering of the abstract alphabet. Where s0 is a hypothetical initial state, and C(·) is a clustering function: 。 4. The text adversarial example detection method based on automata interpretation according to claim 3, characterized in that, Step 3.3 specifically includes: The abstract trajectory of each text sample is used to construct a tree with the abstract alphabet as the path, with s0 as the initial tree root. Count the total number of paths emitted by each hidden state and the number of paths for each abstract letter, obtain the probability of each path, and obtain a probability prefix tree.

5. The text adversarial example detection method based on automata interpretation according to claim 1, characterized in that, Step 4 specifically includes: Step 4.1: Select multiple adversarial sample generation methods from TextAttack, and generate a large number of adversarial texts based on the text sample test set; Step 4.2: Remove adversarial samples with confidence levels below the preset value during the generation process, and add adversarial samples with confidence levels above the preset value to the text sample test set to form a mixed dataset to be screened.

Images