A neural network backdoor sample and backdoor model detection method

Abstract

The application claims a neural network backdoor sample and backdoor model detection method, specifically comprising: performing data enhancement on the obtained original data set, respectively training to obtain an original model and an enhanced model, extracting features of the original data on the original model and features of the corresponding enhanced data on the enhanced model, calculating the feature stability score of the data, and distinguishing and separating the sample as a benign sample or a backdoor sample according to the feature stability score. Based on the benign sample and the backdoor sample, a shadow model is trained, an energy function idea is introduced to optimize the query noise used to extract the shadow model features, the shadow model features are extracted using the optimized query noise, a meta-classifier is trained, and whether the target model is a backdoor model is detected. The application can better separate the backdoor sample and the benign sample; the energy function idea is introduced to reduce the number of shadow models required for training the meta-classifier, meeting the low-cost demand of users.

Description

Technical Field

[0001] This invention relates to the field of neural network backdoor attacks and defenses, and particularly to a method for detecting backdoor samples and backdoor models. Background Technology

[0002] With the rapid development of artificial intelligence, neural networks (NNs) have become one of the most widely used technologies in the field, achieving remarkable results in areas such as facial recognition, autonomous driving, medical diagnosis, and cybersecurity. For example, facial recognition is widely used as a method of identity authentication for unlocking mobile phones and making mobile payments; autonomous driving technology has also made remarkable progress; in the medical field, deep learning can use big data analysis to assist doctors in disease detection and diagnosis; and in the security field, deep learning models are also widely used in spam and malware detection.

[0003] However, while neural networks bring convenience, they also face numerous security threats. Among them, backdoor attacks, due to their high stealth and destructiveness, pose a serious threat to the reliability of neural networks. Backdoor attacks can be categorized into data poisoning and model poisoning. Data poisoning involves attackers injecting triggers into clean samples during the data collection phase to create backdoor samples. These backdoor samples, along with the clean samples, constitute the poisoned training dataset. Users training models using this poisoned training set will have their models implanted with the backdoor, thus achieving a backdoor attack. Model poisoning involves attackers directly implanting backdoors into the model and then distributing the poisoned model to the public internet. When users deploy their models using the poisoned model released by the attacker, they become vulnerable to attack. Currently, neural networks face the threat of backdoor attacks, necessitating effective defensive measures to ensure the availability and reliability of neural networks.

[0004] Current mainstream backdoor defense methods mainly include: 1) input preprocessing (such as input cleansing and trigger detection); 2) model pruning and fine-tuning; and 3) meta-learning-based model detection. However, these methods have significant shortcomings: input preprocessing methods are difficult to cover unknown trigger forms; model repair methods may impair the original performance; and meta-learning-based methods (such as Neural Cleanse and ABS) usually require a large number of shadow models for training, which is costly and difficult to deploy in resource-constrained scenarios.

[0005] Therefore, this invention proposes a method for detecting backdoor samples and backdoor models in neural networks.

[0006] A search revealed that application publication number CN120995452A belongs to the field of backdoor sample detection technology. The method described includes a black-box backdoor detection method based on adaptive feature injection, comprising: acquiring a sample image to be detected and a pre-trained model as input; preparing a dataset containing the original test image and clean samples; extracting features from the image samples using a deep convolutional neural network; selecting two clean samples with the largest feature contrast as control samples with the most significant feature differences; fusing the two control samples with the sample image to be detected; inputting the fused sample image into the detection model; obtaining the prediction output results of the fused sample and the original image; comparing the semantic consistency of the two prediction results in the previous step to determine whether the original sample carries a potential backdoor trigger; and continuing the above discrimination process for the next sample to be detected until all samples have been detected.

[0007] Unlike the methods described above, this invention focuses on the stability difference of sample features under the original model and the enhanced model. By quantifying this difference, a Feature Stability Score (FSS) is constructed to effectively separate backdoor samples from benign samples. Simultaneously, in the backdoor model detection stage, the concept of an energy function is introduced to optimize query noise selection, significantly reducing the number of shadow models required. This reduces computational overhead while maintaining detection accuracy, making it more suitable for lightweight deployment. Summary of the Invention

[0008] This invention aims to solve the problems of the prior art. It proposes a method for detecting backdoor samples and backdoor models in neural networks. The technical solution of this invention is as follows:

[0009] A method for detecting backdoor samples and backdoor models in neural networks, comprising the following steps:

[0010] S1: Perform data augmentation on the original sample data, and train the original model and the augmented model separately;

[0011] S2: Extract the features of the original data on the original model and the features of the augmented data corresponding to the original data on the augmented model, and calculate the feature stability score (FSS) of the data.

[0012] S3: Based on the feature stability score obtained in S2, determine whether the sample with this score is a benign sample or a backdoor sample;

[0013] S4: Based on benign samples and backdoor samples, train the shadow model, introduce the method of treating combinatorial optimization problems as free energy minimization problems in statistical physics from the concept of energy function, model the optimization process of query noise used to extract shadow model features as a combinatorial optimization problem, and solve it using the FEM method in energy function;

[0014] S5: Use the optimized query noise obtained in S4 to extract shadow model features and train the meta-classifier;

[0015] S6: Use the query noise from S4 to extract features from the model to be detected, and input them into the meta-classifier of S5 to detect whether it is a backdoor model.

[0016] Furthermore, in step S1, data augmentation is performed on the original sample data, and the original model and the augmented model are trained separately, specifically including:

[0017] S11: For the original dataset Perform data augmentation to obtain the corresponding augmented datasets. ;

[0018] S12: Using the original dataset and augmented datasets The original models were trained separately. and corresponding enhancement models .

[0019] Furthermore, in step S2, the features of the original data on the original model and the features of the corresponding augmented data on the augmented model are extracted, and the feature stability score of the data is calculated. Specifically, it includes:

[0020] S21: Transfer the original data Input to the original model obtained in step S12 In the process, the features of the last hidden layer of the sample on the original model are extracted;

[0021] S22: Enhance data The augmented model obtained by inputting into S12 In the process, the features of the last hidden layer of the augmented sample in the augmented model are extracted;

[0022] S23: Define a feature stability score based on the difference in feature changes between the original model and the enhanced model in the intermediate layer. A higher stability score indicates greater feature variation, making it more likely to be a backdoor sample. (The calculation of the sample's stability score is not directly related to the stability score.) .

[0023] Furthermore, in step S3, the characteristic stability score obtained in S2 is used to determine whether a sample with this score is a benign sample or a backdoor sample, specifically including:

[0024] S31: Obtained from S2 Calculate the feature stability score for each sample and construct... Distribution histogram;

[0025] S32: Use the peak-valley detection method to find the valley between the two peaks of the histogram as a threshold. ;

[0026] S33: Based on the threshold The feature stability score is greater than If the result is positive, it is a backdoor sample; otherwise, it is a benign sample.

[0027] Furthermore, in step S4, a shadow model is trained based on benign samples and backdoor samples. The query noise used to extract features from the shadow model is optimized using the energy function concept, specifically including:

[0028] S41: Based on the backdoor samples and benign samples obtained from S3, train the backdoor shadow model and the benign shadow model respectively;

[0029] S42: Initialize query noise and extract initial features to pre-train the initial meta-classifier.

[0030] S43: Model "query noise optimization" as a combinatorial optimization problem, that is, select which noises from the candidate noise pool as the final query noise so that the extracted shadow model features are most representative, and use the energy function FEM method to solve for the optimal query noise.

[0031] Furthermore, in step S5, the optimized query noise obtained in S4 is used to extract shadow model features and train a meta-classifier, specifically including:

[0032] S51: Input the final query noise into the shadow model and extract the features of each shadow model;

[0033] S52: Train the final meta-classifier based on the features.

[0034] Furthermore, in step S6, features are extracted from the query noise of the model to be detected using S4, and then input into the meta-classifier of S5 to detect whether it is a backdoor model. Specifically, this includes:

[0035] S61: Input the optimal query noise from S4 into the model to be detected, and extract the features of the model to be detected;

[0036] S62: Input the features of the model to be detected into the final meta-classifier in S5 to obtain the classification result.

[0037] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a neural network backdoor sample and backdoor model detection method as described in any one of the claims.

[0038] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the neural network backdoor sample and backdoor model detection method as described in any one of the claims.

[0039] A computer program product includes a computer program that, when executed by a processor, implements the neural network backdoor sample and backdoor model detection method as described in any one of the claims.

[0040] The advantages and beneficial effects of this invention are as follows:

[0041] 1. In step S1 of this invention, perturbation is performed through data augmentation. Compared with methods that require careful design of perturbation, it has higher versatility. Furthermore, traditional data augmentation techniques are mainly used to improve the robustness or generalization ability of models, rather than for security detection. Combining data augmentation techniques with feature stability analysis for backdoor detection is novel and innovative in terms of method.

[0042] 2. In step S2 of this invention, the stability of sample features is calculated by comparing the features between the original model and the enhanced model. Traditional backdoor detection methods mainly focus on single-model analysis, abnormal input-output pair detection, or abnormal pattern recognition of model parameters. The method proposed in this invention, which calculates the feature stability score (FSS) by comparing the feature changes of the original model and the enhanced model for data with the same semantics but different representations, breaks through the traditional single-model analysis framework.

[0043] 3. In step S4 of this invention, the Energy Function (FEM) method is introduced to model the query noise optimization problem as a combinatorial optimization problem. Utilizing the FEM method's ability to efficiently solve this problem, the efficiency of query noise optimization is improved, making the shadow model feature extraction more representative. This indirectly enhances the classification performance of the meta-classifier and reduces the cost of training shadow models for the meta-classifier. The FEM method was originally a physics-inspired algorithm proposed to solve combinatorial optimization problems. Applying it to query noise optimization in neural network backdoor detection is an innovative interdisciplinary approach, representing a groundbreaking attempt to introduce a theoretical framework from statistical physics into the field of deep learning security. Attached Figure Description

[0044] Figure 1 This is a flowchart of the preferred embodiment provided by the present invention;

[0045] Figure 2 A framework diagram of a backdoor sample detection method;

[0046] Figure 3 A framework diagram of a meta-classifier backdoor model detection method; Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0048] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0049] 1. To address the difficulty in effectively separating backdoor samples from benign samples, this paper proposes a backdoor sample detection method based on feature stability analysis.

[0050] 2. To address the issue of excessively high costs associated with training shadow models for meta-classifiers, this paper proposes a backdoor model detection method for meta-classifiers based on the energy function FEM.

[0051] like Figure 1 As shown, the present invention provides a method for detecting backdoor samples and backdoor models in neural networks, characterized by comprising the following steps:

[0052] S1: Perform data augmentation on the original sample data, and train the original model and the augmented model separately;

[0053] S2: Extract the features of the original data on the original model and the features of the corresponding augmented data on the augmented model, and calculate the feature stability score of the data. ;

[0054] S3: Based on the feature stability score obtained in S2, determine whether the sample with this score is a benign sample or a backdoor sample;

[0055] S4: Based on benign samples and backdoor samples, train the shadow model and use the energy function idea to optimize the query noise used to extract shadow model features;

[0056] S5: Use the optimized query noise obtained in S4 to extract shadow model features and train the meta-classifier;

[0057] S6: Use the query noise from S4 to extract features from the model to be detected, and input them into the meta-classifier of S5 to detect whether it is a backdoor model.

[0058] In this embodiment, step S1 involves data augmentation of the original sample data and training the original model and the augmented model separately, specifically including the following steps:

[0059] S11: For the original dataset Perform data augmentation (rotation, translation, scaling, etc.). The corresponding augmented datasets were obtained respectively.

[0060] ;

[0061] Because multiple augmentation methods are used, multiple augmentation datasets will be obtained, such as rotation augmentation datasets, translation augmentation datasets, scaling augmentation datasets, etc.

[0062] S12: Using the original dataset and augmented datasets The original models were trained separately. and corresponding enhancement models There are also corresponding augmentation models, such as rotation augmentation models, translation augmentation models, and scaling augmentation models;

[0063] In this embodiment, in step S2, the features of the original data on the original model and the features of the augmented data corresponding to the original data on the augmented model are extracted, and the feature stability score of the data is calculated. Specifically, it includes the following steps:

[0064] S21: Transfer the original data The original model obtained by inputting into S12 In the process, the features of the last hidden layer of the sample on the original model are extracted. ;

[0065] in This is the feature vector of the last hidden layer of the model.

[0066] S22: Enhance data The augmented model obtained by inputting into S12 In the process, features from the last hidden layer of the augmented sample in the augmentation model are extracted. :

[0067]

[0068] The above formula indicates that the sample Multiple augmented samples can be used to obtain their augmented features on the corresponding augmented model.

[0069] S23: Define a feature stability score based on the difference in feature changes between the original model and the enhanced model in the intermediate layer. (Feature Stability Score) A higher stability score indicates greater feature variation, making it more likely to be a backdoor sample; known enhancement methods have For each sample Its characteristic stability score Defined as:

[0070] ;

[0071] The average feature difference between the original sample and each of the corresponding multiple augmented samples is calculated as the stability score, where... The cosine similarity between the original features and the enhanced features is:

[0072] ;

[0073] Considering the directionality of feature vectors, cosine similarity is used as a similarity measure.

[0074] In this embodiment, step S3, which determines whether a sample with a certain feature stability score obtained in S2 is a benign sample or a backdoor sample, specifically includes the following steps:

[0075] S31: Obtained from S2 Calculate the feature stability score for each sample and construct... Distribution histogram;

[0076] S32: Use the peak-valley detection method to find the valley between the two peaks of the histogram as the threshold. ;

[0077] S33: Based on the threshold The feature stability score is greater than The sample that is not a backdoor sample is considered a benign sample, based on a threshold. Separate backdoor samples

[0078]

[0079] The process from S1 to S3 is as follows: Figure 2 As shown.

[0080] In this embodiment, step S4 involves training a shadow model based on benign samples and backdoor samples, and optimizing the query noise used to extract shadow model features using the energy function concept. Specifically, this includes the following steps:

[0081] S41: Based on the backdoor samples and benign samples obtained from S3, train the backdoor shadow model and the benign shadow model respectively;

[0082] After obtaining backdoor and benign samples, a set of shadow models is trained. Use benign samples to train a benign shadow model set Training a backdoor shadow model set using backdoor samples Different initial parameters are used during training to increase model diversity.

[0083] S42: Initialize query noise And extract initial features to pre-train an initial meta-classifier. ;

[0084] Extracting features from the shadow model requires selecting a set of query noise. The output vector of the query noise in the shadow model represents the feature representation of the shadow model. The feature representation of the shadow model is then used as training samples to train the meta-classifier. Therefore, a candidate noise pool needs to be initialized. The input is used to extract initial features from the shadow model, and a two-layer MLP is pre-trained as the initial meta-classifier. Initial meta-classifier Its role is to serve as a proxy energy function for subsequent energy function methods.

[0085] S43: Model "query noise optimization" as a combinatorial optimization problem, that is, select which noises from the candidate noise pool as the final query noise so that the extracted shadow model features are most representative, and use the energy function FEM method to solve for the optimal query noise.

[0086] The Energy Function (FEM) method treats combinatorial optimization problems as free energy minimization problems in statistical physics. Combining automatic differentiation and gradient optimization techniques from machine learning, it can efficiently handle combinatorial optimization problems. Furthermore, it introduces a temperature annealing strategy to avoid getting trapped in local minima and a multi-copy mechanism to simultaneously optimize the free energy of multiple copies, improving search diversity and success rate. In this embodiment, "query noise optimization" is modeled as a combinatorial optimization problem. The candidate noise pool is binarized, with selected noise represented by 1 and unselected noise by 0. The number of selected noise samples is k, and the variable represents whether a particular noise sample is selected (binary). Where m is the number of noises in the candidate noise pool of S42, the optimization objective is to select an optimal query noise set that maximizes the separability of features between the backdoor shadow model and the benign shadow model. This is a combinatorial optimization problem, which can be solved using the energy function FEM method. In this embodiment, the separability measure is the initial meta-classifier. Cross-entropy loss Its formula is:

[0087] ];

[0088] in It is a real label (0 or 1). This is the probability that the model predicts the class as positive. Therefore, the energy function is defined as:

[0089] ;

[0090] Free energy is:

[0091] ;

[0092] in ;

[0093] The gradient of the free energy F is automatically differentiated and calculated. The Adam optimizer is used to optimize noise selection. Temperature annealing cycle is used to avoid local optima. Multiple replicas are set up to optimize and select the replica with the lowest energy. Finally, the optimized query noise set is obtained. .

[0094] In this embodiment, in step S5, the optimized query noise obtained in S4 is used. Extracting shadow model features and training a meta-classifier involves the following steps:

[0095] S51: Optimal query noise Input into shadow model Extract features from each shadow model:

[0096] ;

[0097] S52: Train the final meta-classifier based on features .

[0098] In this embodiment, in step S6, the model to be detected... Features are extracted from the query noise in S4 and input into the meta-classifier in S5 to detect whether it is a backdoor model. The specific steps include the following:

[0099] S61: Model to be detected Input optimal query noise Extract features from the model to be detected;

[0100] S62: Input the features of the model to be detected into the final meta-classifier in S5. The classification results are obtained.

[0101] The process from S4 to S6 is as follows: Figure 3 As shown.

[0102] This invention addresses the difficulty in effectively separating backdoor samples from benign samples by proposing a backdoor sample detection method based on feature stability analysis. It analyzes the stability differences of features in the original and augmented models based on the perturbations inherent in data augmentation, calculating the sample's feature stability score to determine whether a sample is a backdoor sample. Compared to other methods, this paper uses data augmentation for perturbation, eliminating the need for separately designed perturbation methods and offering greater versatility. By comparing the features between the original and augmented models, the stability of sample features is calculated for appropriate discrimination, ensuring both the accuracy of separating backdoor and benign samples and the detection rate of backdoor samples. Furthermore, this paper proposes a meta-classifier backdoor model detection method based on the energy function (FEM). To reduce the cost of training meta-classifiers, it introduces the energy function concept, modeling query noise selection as a combinatorial optimization problem and using the FEM method for optimization. This improves the feature representativeness of shadow models, indirectly enhancing the classification performance of the meta-classifier and significantly reducing the number of shadow models required for meta-classifier training. This approach reduces costs while maintaining detection accuracy, making it more suitable for lightweight deployment.

[0103] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.

[0104] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0105] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0106] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A method for detecting backdoor samples and backdoor models in neural networks, characterized in that, Includes the following steps: S1: Perform data augmentation on the original sample data, and train the original model and the augmented model separately; S2: Extract the features of the original data on the original model and the features of the augmented data corresponding to the original data on the augmented model, and calculate the feature stability score (FSS) of the data. S3: Based on the feature stability score obtained in S2, determine whether the sample with this score is a benign sample or a backdoor sample; S4: Based on benign samples and backdoor samples, train the shadow model, introduce the method of treating combinatorial optimization problems as free energy minimization problems in statistical physics from the concept of energy function, model the optimization process of query noise used to extract shadow model features as a combinatorial optimization problem, and solve it using the FEM method in energy function; S5: Use the optimized query noise obtained in S4 to extract shadow model features and train the meta-classifier; S6: Use the query noise from S4 to extract features from the model to be detected, and input them into the meta-classifier of S5 to detect whether it is a backdoor model.

2. The method for detecting neural network backdoor samples and backdoor models according to claim 1, characterized in that, In step S1, data augmentation is performed on the original sample data, and the original model and the augmented model are trained separately, specifically including: S11: For the original dataset Perform data augmentation to obtain the corresponding augmented datasets. ; S12: Using the original dataset and augmented datasets The original models were trained separately. and corresponding enhancement models .

3. The method for detecting backdoor samples and backdoor models in a neural network according to claim 1, characterized in that, In step S2, the features of the original data on the original model and the features of the corresponding augmented data on the augmented model are extracted, and the feature stability score of the data is calculated. Specifically, it includes: S21: Transfer the original data Input to the original model obtained in step S12 In the process, the features of the last hidden layer of the sample on the original model are extracted; S22: Enhance data The augmented model obtained by inputting into S12 In the process, the features of the last hidden layer of the augmented sample in the augmented model are extracted; S23: Define a feature stability score based on the difference in feature changes between the original model and the enhanced model in the intermediate layer. A higher stability score indicates greater feature variation, making it more likely to be a backdoor sample. (The calculation of the sample's stability score is not directly related to the stability score.) .

4. The method for detecting backdoor samples and backdoor models in a neural network according to claim 1, characterized in that, In step S3, the sample with the characteristic stability score obtained in S2 is determined to be a benign sample or a backdoor sample, specifically including: S31: Obtained from S2 Calculate the feature stability score for each sample and construct... Distribution histogram; S32: Use the peak-valley detection method to find the valley between the two peaks of the histogram as the threshold. ; S33: Based on the threshold The feature stability score is greater than If the result is positive, it is a backdoor sample; otherwise, it is a benign sample.

5. The method for detecting backdoor samples and backdoor models in a neural network according to claim 1, characterized in that, In step S4, a shadow model is trained based on benign samples and backdoor samples. The query noise used to extract features from the shadow model is optimized using the energy function concept. Specifically, this includes: S41: Based on the backdoor samples and benign samples obtained from S3, train the backdoor shadow model and the benign shadow model respectively; S42: Initialize query noise and extract initial features to pre-train an initial meta-classifier; S43: Model "query noise optimization" as a combinatorial optimization problem, that is, select which noises from the candidate noise pool as the final query noise so that the extracted shadow model features are most representative, and use the energy function FEM method to solve for the optimal query noise.

6. The method for detecting backdoor samples and backdoor models in a neural network according to claim 1, characterized in that, In step S5, the optimized query noise obtained in S4 is used to extract shadow model features and train a meta-classifier, specifically including: S51: Input the final query noise into the shadow model and extract the features of each shadow model; S52: Train the final meta-classifier based on the features.

7. The method for detecting backdoor samples and backdoor models in a neural network according to claim 1, characterized in that, In step S6, features are extracted from the model to be detected using the query noise from S4, and then input into the meta-classifier in S5 to detect whether it is a backdoor model. Specifically, this includes: S61: Input the optimal query noise from S4 into the model to be detected, and extract the features of the model to be detected; S62: Input the features of the model to be detected into the final meta-classifier in S5 to obtain the classification result.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the neural network backdoor sample and backdoor model detection method as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the neural network backdoor sample and backdoor model detection method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the neural network backdoor sample and backdoor model detection method as described in any one of claims 1 to 7.

Images