Adversarial example detection methods based on model weight variation and confidence distance

Abstract

This invention discloses an adversarial example detection method based on model weight mutation and confidence distance, comprising: selecting an image dataset and dividing it into a training set and a test set; inputting the image dataset into a corresponding neural network model for training until a preset accuracy is achieved; modifying the model weights of each trained neural network model to perform model weight mutation, obtaining mutated models, and filtering them according to classification accuracy; iterating through the mutated models of benign samples and their corresponding adversarial samples in the training set, calculating the confidence distance, and obtaining the confidence distance matrix of adversarial samples and the confidence distance matrix of benign samples respectively; constructing a binary classifier, concatenating the confidence distance matrix of adversarial samples and the confidence distance matrix of benign samples as a binary classification dataset, using the binary classification dataset to train and optimize the binary classifier until a preset accuracy is achieved; and using the optimized binary classifier to distinguish between adversarial samples and benign samples.

Description

Technical Field

[0001] This invention relates to the field of data security, and in particular to an adversarial example detection method based on model weight variation and confidence distance. Background Technology

[0002] The field of adversarial attacks in deep learning is becoming increasingly widespread, and the dangers of adversarial examples and poisoned samples are becoming increasingly prominent. Utilizing data stream processing in deep learning can enable dimensionality reduction attacks, and deceptively realistic adversarial examples can lead to false positives or false negatives in AI-driven recognition systems. For example, attackers have created adversarial examples by attaching tiny stickers as noise to a "stop" sign. To the human eye, these stickers are indistinguishable from the original "stop" sign, but a deep learning-based recognition system will identify them as a "speed limit" sign or other signs, thus attacking autonomous vehicles. Research has also found that injecting a small number or even a single poisoned sample (e.g., a note pasted on a "stop" sign) into the training dataset can alter the deep learning model's recognition of image content. In this case, the note or a similar symbol can become a backdoor that triggers AI-powered autonomous vehicles to identify the image as a "stop" sign, causing potential harm. Therefore, researching defense and detection techniques against adversarial examples to improve the robustness and security of models against unknown threats is of great significance for improving the reliability of applications and achieving safe and reliable AI algorithms.

[0003] Adversarial attacks targeting deep learning are a hot research topic in artificial intelligence security. An adversarial attack is defined as a malicious attack in which, during the model testing phase, an attacker adds carefully designed micro-perturbations to the original data to obtain adversarial examples, causing the deep learning model to completely fail and misclassify with high confidence. Based on whether the model structure of the target is known, attacks are divided into white-box attacks and black-box attacks; based on the attacker's original expectation, attacks are divided into targeted attacks and untargeted attacks; and based on the type of adversarial example, attacks are divided into virtual digital space attacks and real physical space attacks. Attackers generate specific adversarial examples, causing machine learning to incorrectly classify samples that appear very different to humans as samples that the attacker intends to imitate. Generating high-quality adversarial examples is key to deception attacks; related techniques include methods based on direct gradients (L-BFGS, FGSM, BIM, JSMA, DeepFool, C&W attacks), methods based on gradient estimation (ZOO), methods based on adversarial transformation (ATN), methods based on generative networks (UPSET), and methods based on differential evolution (One Pixel Attack), etc.

[0004] Based on their effectiveness, adversarial defense methods can be categorized into detection-only defense and full defense. Unlike adversarial defense, which attempts to correctly classify adversarial examples, adversarial detection detects and selects adversarial examples based on the differences between them and normal examples. Adversarial example detection methods are mainly divided into empirical statistical detection, image preprocessing and reconstruction-based detection, and detection network-based detection. Hendrycks and Gimpel found that the late principal components of adversarial examples typically have a larger variance than benign late principal components. Using this difference, they calculated the difference threshold between benign and adversarial examples to achieve adversarial detection. Liang et al. treated image perturbations as noise and used scalar quantization and spatial smoothing filters to detect adversarial examples of different pixel scales. Cohen et al. combined the k-nearest neighbor algorithm with influence functions to propose Nearest Neighbor Influence Functions (NNIF) for adversarial example detection. Feinman et al. proposed using kernel density and Bayesian uncertainty estimation for adversarial detection. Kernel density estimation (KDE) was used to identify whether data points were far from manifold-like structures, while Bayesian uncertainty estimation (BUE) detected whether data points were close to the low-confidence region where KDE was ineffective. By examining the output of each convolutional layer within the original model, it is determined whether the input sample is adversarial. Gong et al. distinguished between benign and adversarial samples by training a binary classifier network. The binary classifier is a network completely separate from the main classifier; it does not generate adversarial samples for the detector but instead generates adversarial samples for the pre-trained classifier, adding these adversarial samples to the original training data to train the binary classifier. Grosse et al. built upon this by adding an adversarial class enhancement classifier network, training a new model with an additional class based on a pre-trained model using the benign and adversarial samples generated by that model.

[0005] Although existing detection methods can achieve good results, they still face the following challenges:

[0006] (1) Regardless of whether it is based on empirical statistics or image preprocessing, current advanced adversarial example detection techniques suffer from high algorithm complexity, requiring additional model parameters and exhibiting a high degree of dependence on adversarial examples generated by different attack methods. Therefore, designing lightweight adversarial example detection techniques to reduce dependence on adversarial examples is one of the key technical challenges that needs to be addressed.

[0007] (2) The continuous game between offense and defense and the black box nature of artificial intelligence mean that most existing defenses are based on empirical design and cannot defend against a variety of unknown attacks, thus limiting the detection range.

[0008] (3) Current adversarial example detection methods are mainly based on the adversarial examples themselves, while ignoring the impact of changes within the model. There is still a need to study interpretable techniques for attack and defense, to understand the causes of attacks and the feasibility of defenses, in order to guide adversarial example detection. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention proposes an adversarial example detection method based on model weight mutation and confidence distance. This method involves mutating a neural network model, calculating the confidence distribution by observing changes in the output of a series of neural network models, and training a binary classifier based on the differences in confidence distance, thereby achieving high-precision and high-efficiency detection of adversarial examples.

[0010] To achieve the above objectives, the technical solution of the present invention is as follows: A first aspect of the present invention provides an adversarial example detection method based on model weight variation and confidence distance, the method comprising the following steps:

[0011] 1) Select an image dataset, divide it into training and test sets according to class, and perform one-hot encoding preprocessing on the class table of all images in the image dataset;

[0012] 2) Input the image dataset selected in step 1) into the neural network model for training until the preset accuracy is achieved;

[0013] 3) Modify the model weights of each neural network model trained in step 2) to perform model weight mutation, obtain mutated models, and filter the mutated models according to their classification accuracy;

[0014] 4) After traversing and filtering the benign samples and their corresponding adversarial samples in the training set, calculate the confidence distance and concatenate them to obtain the adversarial sample confidence distance matrix and the benign sample confidence distance matrix respectively.

[0015] 5) Construct a binary classifier by concatenating the confidence distance matrix of adversarial samples and the confidence distance matrix of benign samples as a binary classification dataset. Use the binary classification dataset to train and optimize the binary classifier until the preset accuracy is achieved. Use the optimized binary classifier to distinguish between adversarial samples and benign samples.

[0016] A second aspect of the present invention provides an adversarial example detection device based on model weight variation and confidence distance, comprising one or more processors for the aforementioned adversarial example detection method based on model weight variation and confidence distance.

[0017] A first aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used for the above-described adversarial example detection method based on model weight variation and confidence distance.

[0018] Compared with existing technologies, the advantages of this invention are as follows: This invention addresses the problems of limited detection range, reliance on prior knowledge of adversarial attacks, and poor interpretability in existing adversarial example detection methods by proposing a new adversarial example detection method based on model weight variation and confidence distance. Experimental results on real deep learning models demonstrate that this method has good applicability, can effectively detect unknown adversarial examples, has low space complexity, and maintains the classification accuracy of benign examples. Attached Figure Description

[0019] Figure 1 This is a block diagram of an adversarial example detection method based on model weight variation and confidence distance in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the overall framework of an adversarial example detection method based on model weight variation and confidence distance.

[0021] Figure 3 This is a schematic diagram of an adversarial example detection device based on model weight variation and confidence distance in an embodiment of the present invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0023] Reference Figures 1-2 An adversarial example detection method based on model weight variation and confidence distance includes the following steps:

[0024] 1) Select an image dataset, divide it into training and test sets according to classes, and perform one-hot encoding preprocessing on the class tables of all images in the image dataset. The specific process is as follows:

[0025] 1.1) Selection of image datasets: In this embodiment of the invention, the image datasets are selected from MNIST, CIFAR-10 and GTSRB datasets.

[0026] The MNIST dataset consists of 70,000 grayscale images (28×28 pixels) containing handwritten digits 0-9. It includes 60,000 training images and 10,000 test images. The CIFAR-10 dataset consists of 60,000 color images (32×32 pixels) across 10 classes, with 6,000 images per class. The training set contains 50,000 images, and the test set contains 10,000 images. The GTSRB dataset includes over 50,000 48×48 traffic signal images across 43 classes. Image samples and their corresponding class labels are stored, and the sample set is denoted as X = {x1, x2, ..., x...}.m Each image is tagged with a class symbol y.

[0027] 1.2) Divide the initial image dataset into training and testing sets according to classes, and perform one-hot encoding preprocessing on the class tables of all images in the initial image dataset.

[0028] For example, for the GTSRB dataset, in this instance, 80% of the images from each class in the GTSRB dataset are randomly selected as the training set, and the remaining images are used as the test set. One-hot encoding is performed on the class labels y of all datasets to facilitate subsequent training.

[0029] 2) Input the image dataset selected in step 1 into the neural network model for training until the preset accuracy is achieved, as follows:

[0030] In this example, different image datasets require different neural network models for training. For instance, the MNIST dataset is input into the LeNet-5 model for training, the CIFAR-10 dataset is input into the VGG19 model for training, and the GTSRB dataset is input into the ResNet20 model for training. The input layer size of the classifier model is the same as the image size, [H, W, C], and the output layer size is [H×W×C, 1], where H is the image height, W is the width, and C is the number of input channels.

[0031] The training process for a neural network model includes: A deep neural network (DNN) model can be represented as: in Let Y represent the input of the deep neural network model, and y∈Y represent the output of the deep neural network model. The loss function of the deep neural network model is calculated using the following formula during training:

[0032]

[0033] in, Let θ represent the loss function of the deep neural network model, y be the true class label of sample x, θ be the model parameters, C be the total number of classes to which the sample belongs, and i be the index value of the sample. After training, if the classification accuracy of the deep neural network model reaches 95% or higher, the model and training parameters are saved; otherwise, training continues.

[0034] 3) Modify the model weights of each neural network model trained in step 2) to perform model weight mutation, obtaining mutated models, and then filter the mutated models according to their classification accuracy. The specific steps are as follows:

[0035] 3.1) For a neural network model trained in step 2), given a clean training set... The loss function of formula (1) can be rewritten as:

[0036]

[0037] where the neural network model parameter θ satisfies

[0038] 3.2) Define the model weight change δ, and the calculation formula is as follows:

[0039] δ = -H -1 g + η (3)

[0040] where H is the Hessian matrix of the loss function of the model on the clean data set, that is g is the first derivative of the loss function, that is η(0, 1) is a random value that satisfies the standard normal distribution.

[0041] 3.3) Modify the model weights of the neural network model trained in step 2) for model weight mutation: Take a neural network model trained in step 2) as the seed model M s , perform n model weight mutation operations on it. In this example, n takes 10, and n mutant models M m = {M1, M2,..., M n [[ID=X]]} are obtained.

[0042] The process of model weight mutation includes: modifying the neural network model parameter to θ + δ, and modifying the loss function to [[ID=X]] [[ID=X]] [[ID=X]]

[0043] In particular, in order to ensure the classification accuracy of the model for benign samples, the change amount of the weight also needs to satisfy the following formula:

[0044] <000D118>where ||·||2 represents the vector two-norm, represents the change amount of the loss function, and o(1) represents a higher-order infinitesimal.

[0046] 3.4) Screen the mutant models according to the classification accuracy: Calculate the classification accuracies of the mutant models M m = {M1, M2,..., M n} respectively. If the classification accuracy is lower than 90%, then discard the mutant model. Arrange the mutant models in descending order according to the classification accuracy, and take the top k mutant models for subsequent use, where k < n. If the accuracies of all mutant models do not meet the requirements, continue to mutate the model until k mutant model sequences that meet the classification accuracy are obtained. In this example, k takes 5. Note: There are some tags in the original text (such as , ,

[0043] ) that seem to be incomplete or incorrect in the provided text. I have translated based on the best understanding, but it's possible there are errors in the original content. Also, the "X" marked lines are assumed to be placeholders that might need further clarification in the original context.

[0047] 4) After traversing and filtering the mutated models of benign samples and their corresponding adversarial samples, calculate the confidence distance and concatenate them to obtain the confidence distance matrix of adversarial samples and the confidence distance matrix of benign samples respectively.

[0048] The confidence distance is defined as follows:

[0049]

[0050] Where d(p) is the probability distance, v(p) is the probability variance, and p' is the p after descending order. This represents the average value in the p-sequence.

[0051] Given a set of model inputs X = {x1, x2, ...} and a set of variant models M m ={M1, M2, ..., M k Input a sample into the j-th mutation model (j≤k), calculate the output confidence of the mutation model, iterate through all k mutation models, calculate the confidence distance, and concatenate the obtained values ​​to obtain the confidence sequence. Where C represents the total number of categories.

[0052] Adversarial examples are generated using FGSM. The initial benign sample x and its corresponding adversarial sample are input into the filtered mutation model to obtain the confidence distance matrix of the adversarial sample and the confidence distance matrix of the benign sample, respectively.

[0053] 5) Construct a binary classifier by concatenating the confidence distance matrix of adversarial samples and the confidence distance matrix of benign samples as a binary classification dataset. Use the binary classification dataset to train and optimize the binary classifier until the preset accuracy is achieved. Use the optimized binary classifier to distinguish between adversarial samples and benign samples.

[0054] Step 5) specifically includes the following sub-steps:

[0055] 5.1) Building a binary classifier: Since the confidence distance matrix is ​​a simple two-dimensional matrix, a simple fully connected network is sufficient for classification. Three fully connected layers are stacked, with ReLU activation function chosen for the first two layers and softmax activation function for the last layer.

[0056] 5.2) Label the class labels of the benign sample confidence distance matrix as 0 and the adversarial sample confidence distance matrix as 1. Concatenate the adversarial sample confidence distance matrix and the benign sample confidence distance matrix to construct a binary classification dataset, and then split the dataset in an 8:2 ratio. Then, perform one-hot encoding on the class labels.

[0057] Specifically, for the MNIST dataset, 200 adversarial examples and 100 benign examples were used. For the CIFAR-10 dataset, 200 benign examples and 200 FGSM adversarial examples were used. For the GTSRB dataset, 400 FGSM adversarial examples and 15 benign examples were used.

[0058] 5.3) The binary classifier is trained using the split dataset. The hyperparameters for training the binary classifier are set as follows: cross-entropy as the loss function, SGD as the optimizer, and batch size of 5. For the CIFAR-10 dataset, the epoch is set to 10; for the GTSRB dataset, the epoch is set to 15; and for the MNIST dataset, the epoch is set to 5.

[0059] 5.4) Test the binary classifier for optimization.

[0060] Input the confidence distance matrix of the test set into the trained binary classifier. If the classification accuracy is insufficient, modify the number of training epochs or add a fully connected layer to the binary classifier and retrain it.

[0061] The decision scores of benign samples are input into the binary classifier. If the classification accuracy is insufficient, the number of benign samples in the training set is increased, and the binary classifier is retrained to ensure that the detector can accurately distinguish adversarial samples while maintaining its classification accuracy for benign samples.

[0062] 5.5) Use an optimized binary classifier to distinguish between adversarial samples and benign samples.

[0063] Corresponding to the aforementioned embodiments of the adversarial example detection method based on model weight variation and confidence distance, the present invention also provides embodiments of an adversarial example detection device based on model weight variation and confidence distance.

[0064] See Figure 3 The present invention provides an adversarial example detection device based on model weight variation and confidence distance, comprising one or more processors for implementing the adversarial example detection method based on model weight variation and confidence distance in the above embodiments.

[0065] The embodiments of the adversarial example detection device based on model weight variation and confidence distance of this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any data processing-capable device that houses the adversarial example detection device based on model weight variation and confidence distance according to the present invention. (Except for...) Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0066] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0067] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0068] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the adversarial example detection method based on model weight variation and confidence distance described in the above embodiments.

[0069] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0070] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. An adversarial example detection method based on model weight variation and confidence distance, characterized in that, The method includes the following steps: 1) Select an image dataset, divide it into training and test sets according to class, and perform one-hot encoding preprocessing on the class table of all images in the image dataset; 2) Input the image dataset selected in step 1) into the neural network model for training until the preset accuracy is achieved; 3) Modify the model weights of each neural network model trained in step 2) to perform model weight mutation, obtain mutated models, and filter the mutated models according to their classification accuracy; 4) After traversing and filtering the benign samples and their corresponding adversarial samples in the training set, calculate the confidence distance and concatenate them to obtain the adversarial sample confidence distance matrix and the benign sample confidence distance matrix respectively. 5) Construct a binary classifier by concatenating the confidence distance matrix of adversarial examples and the confidence distance matrix of benign examples as a binary classification dataset. Use the binary classification dataset to train and optimize the binary classifier until the preset accuracy is achieved. Use the optimized binary classifier to distinguish between adversarial examples and benign examples. The process of modifying the model weights of each neural network model trained in step 2) to obtain the mutated model includes: Modify the neural network model parameters to θ+δ, and modify the loss function as follows: Specifically, for a neural network model trained in step 2), given a clean training set... The loss function of the neural network model is changed to: Wherein, the neural network model parameters θ satisfy Define the change in model weights δ as: δ=-H -1 g+n Where H represents the neural network model on a clean dataset. The Hessian matrix of the loss function, i.e. g is the first derivative of the loss function, i.e. η~N(0,1) are random values ​​that satisfy a standard normal distribution; For each neural network model trained in step 2), the model weights are modified to perform model weight mutation: take one of the neural network models trained in step 2) as the seed model M. s Perform n model weight mutation operations on it to obtain n mutated models M. m ={M1,M2,...,M n }; Among them, the change in model weights δ must also satisfy: Where ||·||2 represents the 2-norm of the vector, denoted by , o(1) represents the change in the loss function.

2. The adversarial example detection method based on model weight variation and confidence distance according to claim 1, characterized in that, The image dataset is selected from datasets such as MNIST, CIFAR-10, and GTSRB; step 2) of inputting the image dataset selected in step 1 into the corresponding neural network model for training includes: inputting the MNIST dataset into the LeNet-5 model for training, inputting the CIFAR-10 dataset into the VGG19 model for training, and inputting the GTSRB dataset into the ResNet20 model for training.

3. The adversarial example detection method based on model weight variation and confidence distance according to claim 1, characterized in that, The formula for calculating the loss function of the neural network model in step 2) is as follows: in, Let represent the loss function of the deep neural network model, y be the true class label of sample x, θ be the model parameters, C be the total number of classes to which the sample belongs, and i be the index value of the sample.

4. The adversarial example detection method based on model weight variation and confidence distance according to claim 1, characterized in that, The binary classifier constructed in step 5) is a three-layer fully connected network, with ReLU as the activation function for the first two layers and softmax as the activation function for the last layer.

5. The adversarial example detection method based on model weight variation and confidence distance according to claim 1, characterized in that, Step 5) involves concatenating the confidence distance matrix of adversarial samples and the confidence distance matrix of benign samples to form a binary classification dataset. The class labels of the confidence distance matrix of benign samples are marked as 0, and the class labels of the confidence distance matrix of adversarial samples are marked as 1. The confidence distance matrix of adversarial samples and the confidence distance matrix of benign samples are concatenated to construct a binary classification dataset.

6. The adversarial example detection method based on model weight variation and confidence distance according to claim 1, characterized in that, The process of optimizing the binary classifier in step 5) includes: The confidence distance matrix is ​​input into the trained binary classifier for testing. If the classification accuracy does not reach the preset accuracy, the training rounds are modified, or the structure of the fully connected layer in the binary classifier is increased, and the binary classifier is retrained. The decision scores of benign samples are input into the binary classifier. If the classification accuracy does not reach the preset accuracy, the number of benign samples in the training set is increased, and the binary classifier is retrained.

7. An adversarial example detection device based on model weight variation and confidence distance, characterized in that, It includes one or more processors for implementing the adversarial example detection method based on model weight variation and confidence distance as described in any one of claims 1-6.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program is used to implement the adversarial example detection method based on model weight variation and confidence distance as described in any one of claims 1-6.

Images