Black box model testing method based on adversarial samples
By training an alternative model to forget the target label features of targeted attacks, targeted adversarial examples are generated and their success rate on a black-box model is tested. This solves the problem of low transferability of adversarial examples in existing technologies and improves the security testing effect of black-box models.
Patent Information
- Application Number
- CN202410996403.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-12
AI Technical Summary
In existing technologies for black-box model security testing, the adversarial examples generated using alternative models have low transferability, resulting in poor test results.
By training an alternative model to forget the target label features of targeted attacks, targeted adversarial samples that can be successfully predicted as the target label are generated. The model is then tested on the target black-box model to calculate its success rate in resisting adversarial samples.
It improves the transferability of adversarial examples, enhances model testing performance in black-box scenarios, and significantly strengthens the security testing performance of black-box models.
Smart Images

Figure CN118941898A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence security technology, and specifically discloses a black box model testing method based on adversarial samples. Background Art
[0002] Adversarial samples refer to small (difficult to distinguish by the human eye) but meaningful adversarial perturbations on the input data, which can deceive the model and cause the model to give an incorrect output with high confidence.
[0003] Adversarial samples can be divided into non-targeted and targeted. Non-targeted adversarial samples do not target a specific target category, but try to make the model make any type of incorrect prediction. Targeted adversarial samples have a specific target category, hoping that the model will misclassify it as that target category. Existing application models can be divided into white-box models and black-box models. White-box models, that is, users have full access to the target model, including its structure, parameters, and training data, and can use this information to generate effective adversarial samples against the target model. Black-box models, that is, users can only obtain information about the target model through limited interactions, and cannot directly access its internal structure or parameters. Most intelligent models in real-world applications are black-box models, and adversarial testing of black-box models is more difficult.
[0004] Existing black-box model security testing methods are mainly divided into two categories: directly modifying perturbations through query results or using alternative models to generate adversarial samples. The former can be further divided into gradient estimation-based (OPT and HSJA), optimization-based (BO), and geometry-based (GEODA and surfree), but such methods often require a large number of queries. The latter generates adversarial samples on alternative models based on the transferability of adversarial samples, but does not always get the expected results on the target black-box model. Whether it is a single model or a model integration (combining the gradient information of multiple models, such as DIM and TI-DIM), it is difficult to meet actual needs.
[0005] The present invention mainly addresses the problems of low transferability when using alternative models to generate adversarial samples, easy failure of adversarial samples, and poor effect when using the generated adversarial samples to perform security testing of black-box scenario models. Summary of the invention
[0006] The purpose of the present invention is to provide a method for security testing of black box models to solve the problem of low adversarial sample transferability caused by differences in model structure and parameters, and to improve the technical problem of model testing effect in black box scenarios. The present invention includes the following steps:
[0007] Step 1: Determine the target black box model to be tested, then use the training set of the target black box model to train the alternative model S, and calculate the prediction accuracy of the alternative model S for the target label tl;
[0008] Step 2: Use the training set of the target black box model to train the substitute model S again, so that the substitute model S forgets the characteristics of the target label tl of the targeted attack, and obtains the forgetting model S'. The objective function used in the training process is:
[0009]
[0010] Among them, θ S′ is the parameter of the forgetting model S', x i is the i-th sample, f(x i ,θ S′ ) is the sample x in the forgetting model S′ i The predicted distribution of y i is the true label of the i-th sample;
[0011] Step 3: Use the adversarial sample generation method to generate a directional adversarial sample x that can be successfully predicted as the target label tl on the forgotten model S' after forgetting adv , where the objective function used is:
[0012]
[0013] in, To add the adversarial sample after perturbation, For the sample in the forgetting model S′ The predicted distribution of
[0014] Step 4: Use the generated adversarial samples to test the target black box model and calculate the success rate of the target black box model in resisting the adversarial samples.
[0015] Furthermore, step 1 also includes improving the prediction accuracy by image transformation or reducing the learning rate when the method is about to converge.
[0016] Furthermore, the image transformation in step 1 includes cropping, adding noise or color transformation.
[0017] Furthermore, the feature of the target label is forgotten in step 2, and the implementation method is as follows:
[0018] Step 2.1, generate a data loader using the training set and image transformation used in the training process in step 1;
[0019] Step 2.2: Calculate the loss function L new , the loss function L newUsed to update the parameters of the forgetting model S' during the iterative training of the forgetting model S', when the number of training times reaches the specified threshold N, or the loss function L new When the difference between two consecutive iterative training processes is less than the threshold M, the training is stopped and the forgetting model S' is obtained, and the loss function L new for:
[0020]
[0021] Among them, θ t is the parameter of the forgetting model S' after the tth training, f(x i ,θ t ) is the sample x in the training process i The predicted distribution of L(y i ,f(x i ,θ t )) is the forgetting model S' for sample x during training i The predicted loss of
[0022] Step 2.3: Calculate the accuracy of the target label on the test set during training And the accuracy of other labels
[0023]
[0024] in, is the true label y i The total number of samples that are not the target label tl and are predicted correctly on the forgetting model S', is the true label y i The total number of samples that are not target labels tl, is the true label y i is the total number of samples with target label tl and predicted correctly on the forgetting model S', is the true label y i is the total number of samples of target label tl;
[0025] Step 2.4: Select a smaller learning rate and a more stable optimizer according to the actual situation, start training, and finally obtain the forgotten model S' after forgetting. The parameter update process of the forgotten model S' is:
[0026]
[0027] Among them, θ t+1 is the parameter of the forgetting model S' after the t+1th training, α is the learning rate, represents the gradient operator with respect to the parameter θ, and Optimizer is the optimizer;
[0028] Step 2.5: Observe the prediction results of the forgetting model S' for other labels during the training process to ensure that the prediction accuracy does not drop by more than the value N1, that is:
[0029]
[0030] Step 2.6: Observe the prediction results of the forgetting model S' for the target label tl during the training process. When the prediction accuracy reaches the value N2, stop the training, that is:
[0031]
[0032] Furthermore, step 3 specifically includes:
[0033] Step 3.1: Use the forgetting model S' trained in step 2 and the adversarial sample generation method to perform multiple rounds of iterations, and finally iterate to generate an adversarial sample x that can be successfully predicted as the target label tl by the forgetting model S'. adv , the iterative process of adversarial samples is:
[0034]
[0035] in, is the adversarial sample generated in the tth iteration, Clip x,∈ is a clipping function used to limit the adversarial sample to the range of x±∈, ∈ is a manually set parameter used to limit the perturbation amplitude of the adversarial sample, sign is a function that returns the gradient direction, and L is a loss function used to calculate the target label tl and The difference between the output values, Indicates the calculation of gradient;
[0036] Step 3.2: If no directed adversarial sample x is successfully generated or generated adv If the number is small, try more iterations or retrain the forgetting model S' to select the one with higher prediction accuracy for the target label tl. The forgetting model is used until a large number of adversarial samples x are generated. adv .
[0037] Furthermore, step 4 specifically includes: generating the adversarial sample x adv Used for testing on the target black box model. If the target black box model will be adversarial sample x adv If the prediction is the target label tl, it fails, otherwise it succeeds. Finally, the target black box model is statistically tested to resist the adversarial sample x. adv The success rate is:
[0038]
[0039] in, is the i-th adversarial sample generated for testing, θ TB are the parameters of the target black box model, Used to calculate adversarial samples The output result after input into the target black box model, To take the label with the highest predicted probability among all output results, ∑ i 1 is the total number of adversarial samples generated for testing.
[0040] Compared with the prior art, the present invention has the following positive effects:
[0041] The present invention solves the problem of adversarial sample failure caused by model structure and parameter differences by improving the mobility of generated adversarial samples, thereby improving the model testing effect in black box scenarios. The present invention can significantly improve the mobility of adversarial samples, thereby improving the black box model testing effect, and the present invention generates directional adversarial samples from the perspective of changing the alternative model, which can be organically combined with other existing adversarial sample generation methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a schematic diagram of the overall process of the present invention;
[0043] Figure 2 It is a characteristic flow diagram of forgetting target labels of the present invention. DETAILED DESCRIPTION
[0044] In order to better understand the purpose, technical solution and function of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings, but the present invention can be implemented in a variety of different ways as defined and covered by the claims. The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention, and the exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.
[0045] like Figure 1 As shown, a specific embodiment of the present invention includes the following steps:
[0046] Step 1, first determine the target black-box model TB (Target Black-box model) to be tested, and then use the training set tr (training set) of the target black-box model TB to train the substitution model S (Substitution model), that is, TB and S use the same training set tr. The prediction accuracy of the substitution model S for the target label tl (target label) is statistically analyzed, and the prediction accuracy is improved as much as possible by methods such as image transformation (such as cropping, adding noise, color transformation, etc.), or by reducing the learning rate when it is about to converge. Image transformation and adjustment of the learning rate are common operations in the model training process. The default parameters can be used, and the parameters can also be manually set as needed, which does not affect the specific implementation of the present invention.
[0047] Step 2: Use the training set tr of the target black box model TB to train the substitution model S again. However, the default objective function is not used in this training process. Instead, the following objective function is used so that the substitution model S can forget the characteristics of the target label tl of the targeted attack. After retraining, the substitution model S obtains the forgotten model S' (Substitutionforgot model). Forgetting the characteristics of the target label tl of the targeted attack minimizes the probability of the target label, making it difficult for the forgotten model S' to predict the input as the target label under any circumstances, thereby achieving deception of the forgotten model S'. The objective function during retraining is:
[0048]
[0049] This objective function is used to guide the training process of the forgetting model S'. The objective function requires the forgetting model S' to predict the true label of samples with non-target labels as their true labels, and to predict any non-tl label for samples with target labels tl. Where θ represents the model parameters, x i represents the i-th sample, f(x i ,θ S′ ) represents the sample x on the model S′ i The predicted distribution of y i represents the true label of the i-th sample.
[0050] Step 3: Use the existing common adversarial sample generation methods (such as iterative fast gradient sign algorithm I-FGSM (Iterative-Fast Gradient Sign Method), basic iterative method BIM (Basic Iterative Method), etc.) and use the following objective function to generate a directional adversarial sample x that can be successfully predicted as the target label tl on the forgetting model S' advSince the target label is forgotten by the forgetting model S' in step 2, the generated adversarial sample x adv The process often requires multiple rounds of iterations. The objective function for generating adversarial samples is:
[0051]
[0052] This objective function is used to guide the adversarial sample x adv The objective function requires that the sample whose true label is not tl, after adding disturbance, the prediction result on the forgetting model S' becomes the target label tl. Represents the adversarial sample after adding perturbations.
[0053] Step 4: Use the generated adversarial samples to test the target black box model TB. Calculate the success rate of the target black box model TB against adversarial samples, that is, the success rate at which the target black box model TB will not be deceived by the adversarial samples. The higher the success rate, the stronger the security of the target black box model TB.
[0054] Regarding the specific process of forgetting the characteristics of the target label in step 2, such as Figure 2 As shown, the implementation is as follows:
[0055] Step 2.1: Generate a data loader DL (Data Loader) using the training set tr and image transformation (such as cropping, adding noise, color transformation, etc.) used in the training process in step 1.
[0056] Step 2.2: To distinguish the label y predicted by the forgetting model S' during training i The loss function L is defined as follows for the cases where the target label tl is the same or different. new , used to update the parameters of the forgetting model S' during the iterative training of the forgetting model S', when the number of training times reaches the specified threshold N, or the loss function L new When the difference between two adjacent iterative training processes is less than the threshold M, the training is stopped and the forgetting model S' is obtained. The thresholds N and M are empirical values, generally N is about 1000 and M is about 0.00001. The directional forgetting of the target label tl is achieved by updating the reverse gradient descent. Based on the loss function L new , so that the forgetting model S' during the training process, when y i = tl, reverse gradient descent is used to update the parameters of the forgetting model S', and in y i ≠tl, the parameters of the forgetting model S' are updated using regular gradient descent.
[0057]
[0058] L newis the loss function that meets the above requirements. t is the parameter of the forgetting model S' after the tth training, f(x i ,θ t ) indicates that during the training process, i The predicted distribution of L(y i ,f(x i ,θ t )) indicates that the forgetting model S' for sample x during training i prediction loss.
[0059] Step 2.3: Calculate the accuracy of the target label on the test set during training And the accuracy of other labels
[0060]
[0061] in, Represents the true label y i The total number of samples that are not the target label tl and are predicted correctly on the forgetting model S', Represents the true label y i The total number of samples that are not the target label tl. Represents the true label y i is the total number of samples with target label tl and predicted correctly on the forgetting model S', Represents the true label y i is the total number of samples with target label yl.
[0062] Step 2.4, select a smaller learning rate and a more stable optimizer according to the actual situation. The learning rate and optimizer are both common parameters in the model training process. You can use the default parameter settings or manually set the parameters as needed. Here, by setting a smaller learning rate and a more stable optimizer, the prediction accuracy of the target label can be slowly reduced, avoiding a rapid decline that causes an unstable training process. After selecting the learning rate and optimizer, start model training, and finally obtain the forgotten model S' after forgetting. The model parameter update process is as follows:
[0063]
[0064] Among them, θ t+1 is the model parameter after the t+1th training, α is the learning rate, represents the gradient operator with respect to the parameter θ, and Optimizer is the optimizer.
[0065] Step 2.5. During the training process, observe the prediction results of the forgetting model S' for other labels to ensure that the prediction accuracy does not drop by more than the value N1. N1 is a threshold parameter determined based on experience. If it exceeds the threshold, it is considered that the prediction accuracy has dropped too much and the model is not usable enough. If there is a sharp drop in accuracy, reduce the learning rate to make the model converge more stably; or adjust the ratio of the target label and the remaining labels in the loss function, increase the attention to the remaining labels, and make the model more inclined to learn the remaining labels. It is generally believed that N1 can be 3%, because when a smaller drop occurs, it may be a normal fluctuation in the training process. When a larger drop occurs, the accuracy will generally continue to drop significantly. That is, the requirements:
[0066]
[0067] Step 2.6: Observe the prediction results of the forgetting model S' for the target label tl during the training process. When the prediction accuracy reaches the value N2, the training can be stopped. N2 is a threshold parameter determined based on experience. It is generally believed that N2 can be 50%, because too small a value will make it difficult to generate a directed adversarial sample of the target label tl.
[0068]
[0069] Regarding the specific process of generating adversarial samples in step 3, its implementation is as follows:
[0070] Step 3.1: Based on the forgetting model S' trained in step 2, use the existing commonly used adversarial sample generation methods (such as iterative fast gradient sign algorithm I-FGSM, basic iterative method BIM, etc.) to generate adversarial samples x adv The generation process requires multiple rounds of iterations, and the final iteration generates an adversarial sample x that can be successfully predicted as the target label tl by the forgetting model S' adv The iterative process of adversarial samples is as follows:
[0071]
[0072] in, is the adversarial sample generated in the tth iteration, Clip x,∈ is a clipping function used to limit the adversarial sample to the range of x±∈, ∈ is a manually set parameter used to limit the perturbation amplitude of the adversarial sample, sign is a function that returns the gradient direction, and L is a loss function used to calculate the target label tl and The difference between the output values, Represents the calculation of gradient.
[0073] Step 3.2: If no directed adversarial sample x is successfully generated or generated advIf the number is small, try more iterations or retrain the forgetting model S' to select the one with higher prediction accuracy for the target label tl. The forgetting model is used until a large number of adversarial samples x are generated. adv , if the number is generally more than 100, a threshold can be set manually to judge the generated directed adversarial sample x adv Whether the quantity meets the conditions.
[0074] Regarding the security testing process of the target model in step 4, it is implemented as follows:
[0075] Step 4.1: Generate the adversarial sample x adv Used for testing on the target black box model TB. If the target black box model TB will adversarial sample x adv If the prediction is the target label tl, it is a failure, otherwise it is a success.
[0076] Finally, the target black box model TB is statistically tested to defend against adversarial samples x adv Success Rate.
[0077]
[0078] in, is the i-th adversarial sample generated for testing, θ TB Represents the parameters of the target black box model TB (any model is essentially a set of parameters arranged in a specific structure, here θ TB is the parameter of the target black box model TB, and also represents the target black box model TB itself), Used to calculate adversarial samples The output result after inputting into the target black box model TB, Indicates taking the label with the highest predicted probability among all output results, ∑ i 1 is the total number of adversarial samples generated for testing. The success rate can be obtained by calculating the ratio of the number of adversarial samples that successfully deceive the target black box model TB to the total number of adversarial samples.
[0079] Step 4.2: The higher the success rate, the stronger the security of the model; the lower the success rate, the weaker the security of the model.
[0080] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A black box model testing method based on adversarial samples, characterized in that: The steps include: Step 1: Determine the target black box model to be tested, then use the training set of the target black box model to train the alternative model S, and calculate the prediction accuracy of the alternative model S for the target label tl; Step 2: Use the training set of the target black box model to train the substitute model S again, so that the substitute model S forgets the characteristics of the target label tl of the targeted attack, and obtains the forgetting model S'. The objective function used in the training process is: Among them, θ S′ is the parameter of the forgetting model S', x i is the i-th sample, f(x i ,θ S′ ) is the sample x in the forgetting model S′ i The predicted distribution of y i is the true label of the i-th sample; Step 3: Use the adversarial sample generation method to generate a directional adversarial sample x that can be successfully predicted as the target label tl on the forgotten model S' after forgetting adv , where the objective function used is: in, To add the adversarial sample after perturbation, For the sample in the forgetting model S′ The predicted distribution of Step 4: Use the generated adversarial samples to test the target black box model and calculate the success rate of the target black box model in resisting the adversarial samples.
2. A black box model testing method based on adversarial samples according to claim 1, characterized in that: Step 1 also includes improving the prediction accuracy by image transformation or reducing the learning rate when convergence is imminent.
3. A black box model testing method based on adversarial samples according to claim 2, characterized in that: The image transformation in step 1 includes cropping, adding noise or color transformation.
4. A black box model testing method based on adversarial samples according to any one of claims 1 to 3, characterized in that: The feature of the target label is forgotten in step 2. This is achieved by: Step 2.1, generate a data loader using the training set and image transformation used in the training process in step 1; Step 2.2: Calculate the loss function L new , the loss function L new Used to update the parameters of the forgetting model S′ during the iterative training of the forgetting model S′, when the number of training times reaches the specified threshold N, or the loss function L new When the difference between two consecutive iterative training processes is less than the threshold M, the training is stopped and the forgetting model S′ is obtained, and the loss function L new for: Among them, θ t is the parameter of the forgetting model S′ after the tth training, f(x i , θt) is the number of samples x in the training process i The predicted distribution of L(y i ,f(x i ,θ t )) is the forgetting model S′ for sample x during training i The predicted loss of Step 2.3: Calculate the accuracy of the target label on the test set during training And the accuracy of other labels in, is the total number of samples whose true label yi is not the target label tl and is predicted correctly on the forgetting model S′, is the total number of samples whose true label yi is not the target label tl, is the total number of samples whose true label yi is the target label tl and predicted correctly on the forgetting model S′, is the total number of samples with the true label yi and the target label tl; Step 2.4: Select a smaller learning rate and a more stable optimizer according to the actual situation, start training, and finally obtain the forgotten model S′ after forgetting. The parameter update process of the forgotten model S′ is: Among them, θ t+1 is the parameter of the forgetting model S′ after the t+1th training, α is the learning rate, represents the gradient operator with respect to the parameter θ, and Optimizer is the optimizer; Step 2.5: Observe the prediction results of the forgetting model S′ for other labels during the training process to ensure that the prediction accuracy does not drop by more than the value N1, that is: Step 2.6: Observe the prediction results of the forgetting model S′ for the target label tl during the training process. When the prediction accuracy reaches the value N2, stop the training, that is:
5. The black box model testing method based on adversarial samples according to claim 4 is characterized in that: Step 3 specifically includes: Step 3.1: Use the forgetting model S′ trained in step 2 and the adversarial sample generation method to perform multiple rounds of iterations, and finally iterate to generate an adversarial sample x that can be successfully predicted as the target label tl by the forgetting model S′. adv , the iterative process of adversarial samples is: in, is the adversarial sample generated in the tth iteration, Clip x,∈ is a clipping function used to limit the adversarial sample to the range of x±∈, ∈ is a manually set parameter used to limit the perturbation amplitude of the adversarial sample, sign is a function that returns the gradient direction, and L is a loss function used to calculate the target label tl and The difference between the output values, Indicates the calculation of gradient; Step 3.2: If no directed adversarial sample x is successfully generated or generated adv If the number is small, try more iterations or retrain the forgetting model S′ to select the one with higher prediction accuracy for the target label tl. The forgetting model is used until a large number of adversarial samples x are generated. adv .
6. A black box model testing method based on adversarial samples according to claim 5, characterized in that: Step 4 specifically includes: generating adversarial samples x adv Used for testing on the target black box model. If the target black box model will be adversarial sample x adv If the prediction is the target label tl, it fails, otherwise it succeeds. Finally, the target black box model is statistically tested to resist the adversarial sample x. adv The success rate is: in, is the i-th adversarial sample generated for testing, θ TB are the parameters of the target black box model, Used to calculate adversarial samples The output result after input into the target black box model, To take the label with the highest predicted probability among all output results, ∑ i 1 is the total number of adversarial samples generated for testing.
