Anti-robust distillation defense method and system based on clean teacher, and terminal

Abstract

The invention discloses a robust distillation defense method, system and terminal based on a clean teacher, and the method comprises the steps: determining a student model and a preset teacher model which is a clean teacher model; obtaining a preset sample, generating an adversarial sample based on the preset sample, and determining a learning target of the student model, the preset sample being a clean sample; based on the preset teacher model, the student model is guided to complete learning of a preset sample and an adversarial sample, and an optimized student model completing the learning target is obtained; optimizing a learning target of the student model based on a self-learning distillation mechanism; and based on the preset teacher model and the optimized student model, guiding the student model to complete learning of a preset sample and an adversarial sample, and obtaining a target model of a learning target after completion of optimization. The clean sample identification accuracy and robustness of the student model can be improved at the same time, and the knowledge system of the student model is expanded.

Description

Technical Field

[0001] The present invention relates to the field of adversarial defense technology, and in particular to an adversarial robust distillation defense method, system, and terminal based on a clean teacher. Background Art

[0002] Deep Neural Networks (DNNs), an extension of traditional artificial neural networks, are powerful tools for solving complex real-world learning problems such as image classification, facial recognition, and natural language processing. Adversarial training is a practical method for improving the robustness of DNNs against adversarial forces. However, studies have shown that adversarial training can cause models to over-focus on interference during training, affecting their accuracy on raw data (clean samples).

[0003] Small models are highly favored in resource-constrained end devices due to their efficient computing performance and low resource usage. However, the presence of adversarial examples severely impacts their practical deployment. Currently, most researchers use Adversarial Robust Distillation (ARD) technology, a technique used in adversarial defense, to improve the robustness of small models. However, these mainstream adversarial robust distillation methods can affect the recognition rate of small models on clean samples, resulting in significant challenges in their practical deployment. Summary of the Invention

[0004] The main purpose of the present invention is to provide an adversarial robust distillation defense method, system and terminal based on a clean teacher, aiming to solve the problem in the prior art that the adversarial robust distillation defense method affects the recognition rate of clean samples of small models.

[0005] To achieve the above objectives, the present invention provides an adversarial robust distillation defense method based on a clean teacher, the method comprising the following steps:

[0006] Determine a student model and a preset teacher model, wherein the preset teacher model is a clean teacher model;

[0007] Obtaining a preset sample, generating an adversarial sample based on the preset sample, and determining a learning objective of the student model, wherein the preset sample is a clean sample;

[0008] Based on the preset teacher model, guide the student model to complete the learning of preset samples and adversarial samples, and obtain an optimized student model that achieves the learning goal;

[0009] Optimizing the learning objective based on a self-learning distillation mechanism;

[0010] Based on the preset teacher model and the optimized student model, the student model is guided to complete the learning of preset samples and adversarial samples, and a target model that completes the optimized learning goal is obtained.

[0011] Optionally, generating an adversarial sample based on the preset sample specifically comprises: attacking the preset sample to generate an adversarial sample corresponding to the preset sample;

[0012] The attacking the preset sample includes: using an optimization-based adversarial attack method to add disturbance to the preset sample under a norm constraint, wherein the amplitude of the disturbance does not exceed a preset disturbance amplitude threshold.

[0013] Optionally, determining the learning objectives of the student model includes:

[0014] Determining a knowledge distillation contrast loss function for measuring the difference in output probabilities between the preset teacher model and the student model;

[0015] A learning objective of the student model is determined based on the knowledge distillation contrast loss function.

[0016] Optionally, the learning objectives include at least a first learning objective and a second learning objective, and determining the learning objectives of the student model based on the knowledge distillation contrast loss function includes:

[0017] Determining a first adjustable parameter of the preset sample and a second adjustable parameter of the adversarial sample;

[0018] Inputting the preset sample into the preset teacher model to obtain a first output;

[0019] Inputting the preset sample and the adversarial sample into the student model to obtain a second output based on the preset sample and a third output based on the adversarial sample;

[0020] Obtaining a first learning objective based on the first output, the second output, and the first adjustable parameter;

[0021] A second learning objective is obtained based on the first output, the third output, and the second adjustable parameter.

[0022] Optionally, optimizing the learning objective based on the self-learning distillation mechanism includes:

[0023] Re-determining the first adjustable parameter and the second adjustable parameter, and adding a third adjustable parameter based on a self-learning distillation mechanism;

[0024] Obtaining optimized first learning objectives and second learning objectives based on the re-determined first adjustable parameter and second adjustable parameter;

[0025] obtaining a third learning objective based on the second output, the third output, and the third adjustable parameter;

[0026] The optimized learning objectives at least include: an optimized first learning objective, an optimized second learning objective, and the third learning objective.

[0027] Optionally, guiding the student model to learn preset samples and adversarial samples based on the preset teacher model includes:

[0028] Inputting the preset sample into the preset teacher model to obtain a first output;

[0029] generating a first soft label based on the first output;

[0030] Based on the first soft label, the student model is guided to complete learning of preset samples and adversarial samples.

[0031] Optionally, guiding the student model to complete learning of preset samples and adversarial samples based on the preset teacher model and the optimized student model includes:

[0032] Inputting the adversarial example into the optimized student model to obtain a third output;

[0033] generating a second soft label based on the third output;

[0034] Based on the first soft label and the second soft label, the student model is guided to complete learning of the preset samples and the adversarial samples.

[0035] In addition, to achieve the above objectives, the present invention also provides a clean teacher-based adversarial robust distillation defense system, wherein the clean teacher-based adversarial robust distillation defense system includes:

[0036] A model determination module is used to determine a student model and a preset teacher model, wherein the preset teacher model is a clean teacher model;

[0037] A sample acquisition module is used to acquire a preset sample and generate an adversarial sample based on the preset sample, where the preset sample is a clean sample;

[0038] A goal determination module, used to determine the learning goal of the student model;

[0039] A target optimization module, configured to optimize the learning target based on a self-learning distillation mechanism;

[0040] A model training module is used to guide the student model to complete the learning of preset samples and adversarial samples based on the preset teacher model, and obtain an optimized student model that achieves the learning objectives; based on the preset teacher model and the optimized student model, guide the student model to complete the learning of preset samples and adversarial samples, and obtain a target model that achieves the optimized learning objectives.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a clean teacher-based adversarial robust distillation defense program stored on the memory and runnable on the processor, and when the clean teacher-based adversarial robust distillation defense program is executed by the processor, the steps of the clean teacher-based adversarial robust distillation defense method as described in any one of the above items are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a clean teacher-based adversarial robust distillation defense program, and when the clean teacher-based adversarial robust distillation defense program is executed by a processor, the steps of the clean teacher-based adversarial robust distillation defense method as described above are implemented.

[0043] The present invention provides a clean teacher-based adversarial robust distillation defense method, comprising the following steps: obtaining clean samples and generating adversarial samples based on the clean samples; determining a student model and a clean teacher model, and determining a learning objective for the student model; guiding the student model to complete learning of clean samples and adversarial samples based on the clean teacher model, thereby obtaining an optimized student model that achieves the learning objective; optimizing the learning objective based on a self-learning distillation mechanism; guiding the student model to complete learning of clean samples and adversarial samples based on the clean teacher model and the optimized student model, thereby obtaining a target model that achieves the optimized learning objective. The clean teacher-based adversarial robust distillation method of the present invention improves the clean sample recognition ability and robustness of the student model by learning the soft labels of the clean teacher model in the early stages of training; introducing a self-learning distillation mechanism to guide the learning of the student model in the middle and late stages of training can further improve the robustness of the student model; and using the knowledge distillation contrast loss function (KDCL) to enable the student model to learn the feature relationships between individual samples in a batch of samples, thereby more comprehensively expanding the knowledge system of the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a preferred embodiment of the clean teacher-based adversarial robust distillation defense method of the present invention;

[0045] Figure 2 Schematic diagram of the framework structure of the clean teacher-based robust distillation defense method of the present invention;

[0046] Figure 3 This is a graph showing the change in recognition rate of clean samples by the student model and teacher model of the existing robust distillation method.

[0047] Figure 4 This is a graph showing the change in recognition rate of adversarial samples for the student model and teacher model of the clean teacher-based adversarial robust distillation defense method of the present invention;

[0048] Figure 5 Schematic diagram of the learning process of a student model for a single sample in the clean teacher-based robust distillation defense method of the present invention;

[0049] Figure 6 This is a comparison chart of the KL divergence and KDCL divergence performance of the student model in the single sample learning process in the clean teacher-based robust distillation defense method of the present invention;

[0050] Figure 7 2 is a structural diagram of a preferred embodiment of the clean teacher-based adversarial robust distillation defense system of the present invention;

[0051] Figure 8 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] In the existing art, mainstream adversarial robust distillation methods, such as the soft-label-based adversarial robust distillation method (RSLAD) and the contrastive relation denoising robust distillation method (CRDND), all use soft labels generated by a robust teacher model to guide the student model's learning process on clean samples. However, since the robust teacher model needs to learn not only the characteristics of normal samples but also the characteristics of adversarial samples during training, its performance on clean samples is limited. This limitation significantly reduces the reliability of the robust teacher model when guiding the student model to learn clean samples, which in turn affects the student model's performance on clean samples.

[0054] The recognition rates of clean samples by the student model and teacher model of the existing robust distillation method are as follows: Figure 3 As shown, Figure 3① is the student model's error rate when identifying clean samples, and ② is the proportion of clean samples misidentified by both the teacher and student models. Experimental data analysis analyzed the changes in the student and teacher model's error rates for clean samples during training. The experimental results show that although the student model's error rate for clean samples gradually decreases, the majority of clean samples misclassified by the student model are also misclassified by the teacher model. This experimental result indicates that the student model's low recognition rate for clean samples is primarily due to the teacher model's insufficient performance on clean samples.

[0055] Therefore, although the adversarial robust distillation method (ARD) has made some progress in improving the robustness of small models, due to its poor performance on clean samples, small models still face significant challenges in practical deployment.

[0056] In response to one or more of the above problems, the present invention determines a student model and a preset teacher model, wherein the preset teacher model is a clean teacher model; obtains preset samples, generates adversarial samples based on the preset samples, and determines the learning objectives of the student model, wherein the preset samples are clean samples; based on the preset teacher model, guides the student model to complete the learning of preset samples and adversarial samples, and obtains an optimized student model that completes the learning objectives; based on a self-learning distillation mechanism, optimizes the learning objectives; based on the preset teacher model and the optimized student model, guides the student model to complete the learning of preset samples and adversarial samples, and obtains a target model that completes the optimized learning objectives; through the clean teacher model and the self-learning distillation mechanism, the clean sample accuracy and robustness of the student model can be improved at the same time, and the knowledge distillation comparison loss function used can also comprehensively expand the knowledge system of the model.

[0057] The method described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, Figure 2 for Figure 1 The framework structure corresponding to the method, the clean teacher-based adversarial robust distillation defense method includes the following steps:

[0058] Step S10: Determine the student model and the preset teacher model.

[0059] The student model is the target model used for training. The existing technology uses a robust teacher model to guide the student model, and there is a situation where the student model has a low recognition rate for clean samples. In order to improve the accuracy of the student model on clean samples, the present invention introduces a clean teacher model, and uses the clean teacher model as a preset teacher model to guide the learning process of the student model. The clean teacher model is a teacher model trained with clean samples to guide the student model. Its core goal is to pass its excellent performance on clean samples to the student model. Although the clean teacher model has weak robustness and is difficult to identify adversarial samples generated for itself, it can still effectively identify adversarial samples generated by other models through white-box attacks.

[0060] like Figure 4 As shown in the figure, this is the change graph of the recognition rate of the student model and the teacher model for adversarial samples. Figure 4 ③ is the accuracy of the student model in identifying adversarial samples, and ④ is the accuracy of the teacher model in identifying adversarial samples. Figure 4 As can be seen from the results, the robustness of the student model gradually improves under the guidance of the clean teacher model. Furthermore, the clean teacher model demonstrates high recognition rates for adversarial examples generated by the student model in the early stages of training. This demonstrates that relying solely on the soft labels of the clean teacher model in the early stages of training can simultaneously improve the student model's clean sample accuracy and robustness. Compared to traditional robust teacher models, the use of a clean teacher model not only effectively addresses the limitations of traditional adversarial robust distillation methods but also significantly reduces training resource costs.

[0061] Step S20: Obtain a preset sample, generate an adversarial sample based on the preset sample, and determine the learning goal of the student model.

[0062] Step S21: Obtain a preset sample, and generate an adversarial sample based on the preset sample.

[0063] In the present invention, the preset samples are clean samples. It should be noted that clean samples are original, undisturbed real data that conforms to natural distribution characteristics. Clean samples are not limited to raw images captured by cameras or other devices, but can also be unprocessed signals collected by sensors, acquired natural language text fragments, etc. Clean samples also need to correspond to the model used and can serve as input data for the corresponding model. In this application, for ease of understanding, raw images containing different content are selected as clean samples.

[0064] Furthermore, the method of generating adversarial samples based on preset samples is: attacking the preset samples to generate adversarial samples corresponding to the preset samples. The attack on the preset samples specifically includes: using the optimized adversarial attack (CW attack) method, pUnder the norm constraint, disturbance is added to the preset sample, and the amplitude of the disturbance does not exceed the preset disturbance amplitude threshold ∈.

[0065] Adversarial samples refer to samples generated by artificially adding subtle perturbations, the amplitude of which is usually lower than the human perception threshold (such as pixel-level modification, text synonym replacement, etc.). Adversarial samples can cause the model to give an erroneous output with high confidence. In this application, CW attacks are used to generate adversarial samples. This attack method has better attack effects and can adjust the confidence level by itself. It can successfully attack many defense methods. In the existing distillation defense classification model, CW attacks can still be highly effective. CW attacks are generally target attacks (TargetAttack) or non-target attacks. A successful attack requires the following two conditions to be met:

[0066] (1) The gap between the adversarial sample and the corresponding clean sample is as small as possible;

[0067] (2) Adversarial examples make the probability of model misclassification as high as possible.

[0068] In this application, the target formula for generating adversarial samples using the CW attack will be further explained in conjunction with the model. The attack method is not limited to the CW attack algorithm, and the fast gradient sign attack (FGSM attack) algorithm and the iterative attack (PGD attack) algorithm can also be used.

[0069] It should be noted that there is no strict order between the selection of clean samples in step S10 and the determination of the student model and clean teacher model in step S20. The student model and clean teacher model corresponding to the clean samples can be selected after the clean samples are selected; or the clean samples that can be input into the student model and clean teacher model can be selected after the student model and clean teacher model are selected. In this application, after the student model and clean teacher model are determined, the clean samples to be used are taken from the data set D where the clean samples are located each time.

[0070] Furthermore, based on the determined student model and clean teacher model, the formula for generating adversarial samples by attacking clean samples with CW is as follows:

[0071] wherex i ′=argmaxCW(S(x i ′,θ),y);

[0072] And ‖x i ′-x i ‖≤∈;

[0073] Among them, x i is a clean sample, x i ′ is an adversarial sample, S(x i′,θ) represents the student model’s response to input x under parameter θ i ′’s predicted output, y is the sample x i The true label of , ∈ is the preset threshold of the perturbation amplitude.

[0074] It should be noted that the generated adversarial sample x i 'needs to satisfy L p Norm constraint.

[0075] Step S22: Determine the knowledge distillation contrast loss function.

[0076] In most previous adversarial robust distillation methods, relative entropy is usually used as the loss function. Relative entropy is also called KL divergence (Kullback-Leibler divergence) or information divergence. The corresponding formula is as follows:

[0077]

[0078] Among them, x i is a sample in the data set, T(x i ) indicates that the teacher model is in sample x i The output on S(x i ) represents the output of the student model under the same input.

[0079] KL divergence is mainly used to measure the difference between the probability distribution of the output of the teacher model and the student model. Its core goal is to make the output of the student model fit the teacher model as much as possible for the same input sample x. i The output of the teacher model is used to guide the student model to imitate the behavior of the teacher model, thereby improving the performance of the student model. However, during the small-batch sample training process, the KL divergence only focuses on the output difference between the teacher model and the student model for a single sample, and fails to fully utilize the correlation information between samples in the same batch.

[0080] Therefore, in this application, the knowledge distillation contrast loss function KDCL is introduced, which is used to measure the output difference between the clean teacher model and the student model. KDCL aims to encourage the model to learn the intrinsic connection between samples in the same batch, and enhance the model performance by explicitly modeling the similarities and differences between samples. In terms of classification loss, the goal of KDCL is to ensure that the output of the student model for clean samples is not only highly consistent with the corresponding output of the teacher model, but also clearly distinguishable from the output of other samples in the same batch of the teacher model, so as to better capture the correlation characteristics between samples and enhance the learning effect of the student model on the teacher model.

[0081] The knowledge distillation contrast loss function KDCL uses different formulas based on the samples input to the student model. In KDCL, the learning formula of the student model for a single clean sample is as follows:

[0082]

[0083] Where N represents the batch size of the dataset D, x i is a clean sample in dataset D, x k For the data set x i Different clean samples, T(x i ) indicates that the teacher model is in the clean sample x i The output on T(x k ) indicates that the teacher model is in the clean sample x k The output on S(x i ) represents the clean sample x of the student model i The output on .

[0084] In KDCL, the learning formula of the student model for a single adversarial example is as follows:

[0085]

[0086] Among them, most of the parameter descriptions are consistent with the learning formula of KDCL for a single clean sample, x i ′ is based on the clean sample x i Generated adversarial examples, S(x i ′) represents the student model adversarial sample x i Output on ′.

[0087] like Figure 5 As shown, this is the learning process of the student model for a single sample. Figure 6 It is used to characterize the relationship between the output of the student model and the teacher model in KL divergence and the relationship between the output of the student model and the clean teacher model in the knowledge distillation contrast loss function KDCL in this application. Figure 5 、 Figure 6 As can be seen from the content, for a mini-batch's first input, KDCL not only ensures that its output S1 on the student model fits the clean teacher model's output T1, but also keeps S1 away from the clean teacher model's output T2 on the second input, thereby further broadening the student model's "field of view" and improving its performance. In other words, the knowledge distillation contrast loss function KDCL encourages the student model to learn the feature relationships between individual samples in a batch, thereby more comprehensively expanding the model's knowledge system.

[0088] Step S23: Determine the learning objective of the student model based on the knowledge distillation contrast loss function.

[0089] The adversarial robust distillation defense method based on a clean teacher divides the training of the student model into two phases. In each phase, the student model undergoes multiple rounds of learning. The learning objectives in this step refer to the learning objectives of the first phase. During this phase, the student model is not yet sufficiently reliable and therefore relies primarily on the soft labels of the clean teacher model for learning to improve its own performance. However, the clean teacher model can demonstrate a high recognition rate for adversarial examples generated by the student model in the early stages of training. Therefore, the learning objectives in this phase mainly include the first learning objective, which characterizes the student model's learning performance on clean examples, and the second learning objective, which characterizes the student model's learning performance against adversarial examples, that is, the student model's robustness.

[0090] In this application, the number of rounds of learning in the first stage is 60 rounds. The learning objectives of this stage include the first learning objective and the second learning objective. The learning objectives of the student model determined based on the knowledge distillation ratio loss function specifically include:

[0091] Step S231: Determine a first adjustable parameter of the preset sample and a second adjustable parameter of the adversarial sample.

[0092] Since the knowledge distillation contrast loss function cannot directly represent the recognition rate of the student model for clean samples and adversarial samples, this application introduces adjustable parameters to jointly determine the learning objectives with the knowledge distillation contrast loss function to characterize the model performance, and different adjustable parameters are used for different inputs.

[0093] Specifically, the clean sample x i Corresponding to the knowledge distillation loss function KDCL(S(x i ),T(x i )), the corresponding first adjustable parameter is set to λ1; adversarial sample x i ′ corresponds to the knowledge distillation loss function KDCL(S(x i ′),T(x i )), and set the corresponding second adjustable parameter to λ2.

[0094] Step S232: Input the clean sample into the clean teacher model to obtain a first output.

[0095] The clean sample x i Input the clean teacher model T and get the first output as T(x i ).

[0096] Step S233: Input the preset sample and the adversarial sample into the student model to obtain a second output based on the preset sample and a third output based on the adversarial sample.

[0097] The clean sample x iInput the student model S, and the second output based on the clean sample is S(x i ); the adversarial sample x i ' Input the student model S, and the third output based on the adversarial sample is S(x i ′).

[0098] Step S234: Obtain a first learning objective based on the first output, the second output, and the first adjustable parameter.

[0099] In this application, based on the first output T(x i ), the second output S(x i ) and the first adjustable parameter λ1, we get the first learning objective λ1*KDCL(S(x i ),T(x i )), the first learning goal is also the learning goal of the student model for clean samples in the first stage of training.

[0100] Step S235: Obtain a second learning objective based on the first output, the third output, and the second adjustable parameter.

[0101] In this application, based on the first output T(x i ), the third output S(x i ′) and the second adjustable parameter λ2, and the second learning objective λ2*KDCL(S(x i ′),T(x i )), the second learning goal is also the learning goal of the student model for adversarial samples in the first stage of training.

[0102] Step S30: Based on the preset teacher model, guide the student model to complete the learning of preset samples and adversarial samples, and obtain an optimized student model that achieves the learning goal.

[0103] Step S31: input the preset sample into the preset teacher model to obtain a first output.

[0104] Specifically, based on the selected clean teacher model T, the clean sample x i Input the clean teacher model T and get the first output as T(x i ).

[0105] Step S32: Generate a first soft label based on the first output.

[0106] Based on the first output T(x i ), generating a first soft label corresponding to the first output.

[0107] Step S33: Based on the first soft label, guide the student model to complete the learning of clean samples and adversarial samples.

[0108] The first soft label guides the student model S to guide the student model S to the clean sample x i and adversarial examples x i ′’s study.

[0109] Step S34: Obtain an optimized student model that achieves the learning goal.

[0110] Specifically, the clean teacher model T needs to guide the student model S through the first soft label until the student model achieves the first learning goal of the first stage λ1*KDCL(S(x i ),T(x i )) and the second learning objectiveλ2*KDCL(S(x i ′),T(x i )), the student model that completes the learning objectives of the first stage is used as the optimized student model.

[0111] Step S40: Optimize the learning objective based on the self-learning distillation mechanism.

[0112] In adversarial robust distillation, the data during training has adversarial features, and these adversarial examples are generated by the student model. As training progresses, the adversarial examples generated by the student model continue to change, making it increasingly difficult for the teacher model to recognize them. If the student model is guided by the clean teacher model throughout the training process, the robustness of the student model will also be weakened. At the same time, during the training process, the recognition rate of the teacher model and the student model for the adversarial examples generated by the student model is as follows: Figure 4 As shown in the figure, the implementation analysis shows that as the training progresses, the robustness of the student model gradually improves, while the recognition rate of the teacher model for adversarial samples generated by the student model continues to decline, which means that the student model itself has a certain degree of robustness.

[0113] The self-learning distillation mechanism uses the trained student model as a "robust teacher model" and, together with the clean teacher model, combines the soft labels of the two models to guide the learning of the student model. In this stage, most of the training time is spent on the "two teacher models" guiding the student model to perform dual learning. Because the trained student model is introduced in this stage, the learning objectives of this stage must also be redefined, that is, the learning objectives of the first stage in step S23 must be optimized. Based on the self-learning distillation mechanism, the learning objectives of the student model are optimized to obtain the learning objectives of the second stage, including:

[0114] Step S41: Re-determine the first adjustable parameter and the second adjustable parameter, and add a third adjustable parameter based on a self-learning distillation mechanism.

[0115] Specifically, the clean sample xi Corresponding to the knowledge distillation loss function KDCL(S(x i ),T(x i )), the corresponding first adjustable parameter is re-determined and represented by α; adversarial sample x i ′ corresponds to the knowledge distillation loss function KDCL(S(x i ′),T(x i )), the corresponding second adjustable parameter is re-determined and represented by β.

[0116] The third adjustable parameter based on the self-learning distillation mechanism is represented by γ. The knowledge distillation contrast loss function KDCL corresponding to this adjustable parameter needs to be re-determined. The specific formula is as follows:

[0117]

[0118] Where N represents the batch size of the dataset D, x i is a clean sample in dataset D, x k For the data set x i Different clean samples, S(x i ) indicates that the student model is in the clean sample x i The output on S(x k ) indicates that the student model is in the clean sample x k The output on S(x i ′) represents the student model’s response to the adversarial sample x i Output on ′.

[0119] It should be noted that the first adjustable parameter, the second adjustable parameter, and the third adjustable parameter are loss trade-off parameters and need to satisfy α+β+γ=1.

[0120] Step S42: Obtain optimized first learning objectives and second learning objectives based on the re-determined first adjustable parameters and second adjustable parameters.

[0121] In this application, based on the first output T(x i ), the second output S(x i ) and the optimized first adjustable parameter α, the optimized first learning objective α*KDCL(S(x i ),T(x i )), the first learning goal is also the learning goal of the student model for clean samples in the second stage of training.

[0122] Based on the first output T(x i ), the third output S(x i ′) and the optimized second adjustable parameter β, the optimized second learning objective β*KDCL(S(xi ′),T(x i )), the second learning objective is also the learning objective of the student model for adversarial samples in the second stage of training.

[0123] Step S43: Obtain a third learning objective based on the second output, the third output, and the third adjustable parameter.

[0124] Based on the second output S(x i ), the third output S(x i ′) and the third adjustable parameter γ, we get the third learning objective γ*KDCL(S(x i ′),S(x i )).

[0125] Step S44: The optimized learning objectives at least include: the optimized first learning objective, the optimized second learning objective, and the third learning objective.

[0126] In this application, the optimized learning objectives include at least: an optimized first learning objective, an optimized second learning objective, and a third learning objective. The optimized objective formula is as follows:

[0127]

[0128] Among them, θ s are the parameters of the student model, and the other parameters have been mentioned above.

[0129] Step S50: Based on the preset teacher model and the optimized student model, guide the student model to complete the learning of the preset samples and adversarial samples to obtain the target model of the optimized learning objective.

[0130] Step S51: input the adversarial sample into the optimized student model to obtain a third output.

[0131] Specifically, based on the optimized student model S (the optimized student model is the student model that has been trained in the first stage, and not all training has been completed at this time, so the symbol uses the original student model symbol S), the adversarial sample x i ' Input the optimized student model S and get the third output S(x i ′).

[0132] Step S52: Generate a second soft label based on the third output.

[0133] Based on the third output S(x i ′), generate the third output S(x i ′) corresponds to the second soft tag.

[0134] Step S53: Based on the first soft label and the second soft label, guide the student model to complete the learning of the preset sample and the adversarial sample.

[0135] The second soft label is used to guide the student model to enhance the robustness of the student model. Guiding the student model based on the first soft label and the second soft label is equivalent to using the student model as a "robust teacher model". Together with the clean teacher model, the soft labels of the two models are combined to guide the learning of the student model, that is, to guide the student model S to the clean sample x. i and adversarial examples x i ′’s study.

[0136] Step S54: Obtain the target model of the optimized learning target.

[0137] Specifically, most of the training time in this stage is spent on the "two teacher models" to guide the student model for dual learning, that is, the clean teacher model T and the student model S guide the student model S through the first soft label and the second soft label until the student model S achieves the learning goal of the second stage.

[0138] The present invention provides a clean teacher-based adversarial robust distillation defense method, comprising the following steps: obtaining clean samples and generating adversarial samples based on the clean samples; determining a student model and a clean teacher model, and determining a learning objective for the student model; guiding the student model to complete learning of clean samples and adversarial samples based on the clean teacher model, thereby obtaining an optimized student model that achieves the learning objective; optimizing the learning objective based on a self-learning distillation mechanism; guiding the student model to complete learning of clean samples and adversarial samples based on the clean teacher model and the optimized student model, thereby obtaining a target model that achieves the optimized learning objective. The clean teacher-based adversarial robust distillation method of the present invention improves the clean sample recognition ability and robustness of the student model by learning the soft labels of the clean teacher model in the early stages of training; introducing a self-learning distillation mechanism to guide the learning of the student model in the middle and late stages of training can further improve the robustness of the student model; and using the knowledge distillation contrast loss function (KDCL) to enable the student model to learn the feature relationships between individual samples in a batch of samples, thereby more comprehensively expanding the knowledge system of the student model.

[0139] Furthermore, to demonstrate the differences between our Clean Teacher-Based Adversarial Robust Distillation Defense (CTARD) and other advanced adversarial defense methods, we applied different adversarial defense methods to the same model and collected relevant data. Table 1 below compares the robustness and clean sample accuracy of CTARD and other advanced adversarial defense methods on the CIFAR-10 dataset, a computer vision dataset for pervasive object recognition.

[0140] Table 1 White-box defense accuracy of RN-18 and MN-V2 models on the CIFAR-10 dataset.

[0141]

[0142] In the table, RN-18 is the abbreviation of the residual network ResNet-18, MN-V2 is the abbreviation of the lightweight convolutional neural network MobileNetV2; Clean is a clean sample; FGSM is an adversarial sample obtained using the fast gradient sign attack; PGD is an adversarial sample obtained using the iterative attack method, where PGD SAT means no distillation, while PGD TRADES "SAT" represents self-distillation; "CW2" represents adversarial examples obtained through optimization-based adversarial attack methods with an L2 norm; "AA" (Activation Attack) represents adversarial examples obtained using a black-box targeted adversarial attack. "SAT" represents a non-distillation adversarial defense method; "TRADES" represents a self-distillation adversarial defense method; "ARD" represents an adversarially robust distillation defense method; "IAD" represents an adversarially robust distillation defense method based on a teacher model with the same structure; "RSLAD" represents a soft-label-based adversarial robust distillation method; "CRDND" represents a contrastive relation denoising robust distillation method; and "CTARD" represents the clean-teacher-based adversarial robust distillation defense method of this application.

[0143] As can be seen from Table 1, compared with the current advanced adversarial robust distillation methods, CTARD significantly improves the robustness of the model under most attack benchmarks. It is worth noting that the student model trained by the CTARD method has significantly improved the accuracy of clean samples. As shown in Table 1, on the CIFAR-10 dataset, the ResNet-18 model trained by the CTARD method has improved the clean accuracy by 3.77% over previous methods, and by 3.30% on the MobileNetV2 model, which fully demonstrates the important role of clean teachers in CTARD in improving the clean accuracy of student models. In addition, the model also shows significant advantages in robustness. Specifically, for the ResNet-18 model, CTARD achieved a performance improvement of 2.67% under the FGSM attack; for the MobileNetV2 model, it improved by 3.69% under the FGSM attack and by 3.96% under the CW2 attack. At the same time, under PGD SAT and PGD TRADES CTARD also achieved a leading level in attack. In addition, CTARD's excellent performance on MobileNetV2 further demonstrates the method's wide adaptability to different model architectures.

[0144] As shown in Table 2 below, Table 2 compares the white-box defense accuracy of the RN-18 model on the CIFAR-100 dataset (an extended version of the CIFAR-10 dataset) and the white-box defense accuracy of the MN-V2 model on the CIFAR-100 dataset. As can be seen from Table 2, the accuracy of the two student models trained using the CTARD method on clean samples has achieved significant improvement, among which the clean sample recognition accuracy of the MobileNetV2 model has increased by 7.99%. In addition, the CTARD method has a significant improvement in the accuracy of FGSM and PGD. SAT PGD TRADES The robustness under CW2 attack has reached the industry-leading level, fully demonstrating the superiority of the CTARD method.

[0145] Table 2 White-box defense accuracy of the RN-18 model on the CIFAR-100 dataset.

[0146]

[0147] To comprehensively evaluate the effectiveness of the CTARD method in improving model robustness, this application also evaluated it in a transferable black-box attack environment and conducted comparative experiments with six baseline methods. The experimental results on the ResNet-18 and MobileNetV2 models are shown in Tables 3 and 4, respectively.

[0148] In Tables 3 and 4, WRN is the abbreviation of Wide Residual Network (WideResNet), WRN-34-10 represents a specific wide residual network, and ResNet-50 represents a specific residual network; FGSM is an adversarial sample obtained using the fast gradient sign attack, PGD-20 is an adversarial sample obtained using the iterative attack method with 20 iterations; JSMA is an adversarial sample obtained using the JSMA (Jacobian-based Saliency Map Attack) adversarial attack algorithm.

[0149] Table 3. Black-box defense accuracy of the ResNet-18 model on the CIFAR-10 dataset.

[0150]

[0151] Table 4 Black-box defense accuracy of the MobileNetV2 model on the CIFAR-10 dataset.

[0152]

[0153] Experimental results demonstrate that the CTARD method significantly outperforms six other baseline methods in robustness under three transfer-based attack scenarios. Particularly noteworthy is that under the JSMA attack, the CTARD method achieves approximately 4% improvement in robustness compared to the baseline methods. This result demonstrates the CTARD method's superior performance in defending against transfer-based black-box attacks. In summary, the CTARD method significantly improves the model's adversarial robustness and clean sample accuracy.

[0154] This achievement is mainly due to the following three key designs: First, CTARD effectively improves the student model's learning ability on clean samples by introducing a clean teacher model; second, to address the unreliability of the teacher model on adversarial samples in the middle and late stages of training, CTARD adopts a self-learning distillation mechanism to enhance the robustness of the student model itself; finally, KDCL encourages the model to learn the feature relationships between samples in the same batch, thereby broadening the model's learning horizon and further improving its generalization ability.

[0155] Furthermore, if Figure 7 As shown, based on the above method, the present invention also provides a clean teacher-based adversarial robust distillation defense system, wherein the clean teacher-based adversarial robust distillation defense system includes:

[0156] A model determination module 71 is used to determine a student model and a clean teacher model;

[0157] A sample acquisition module 72 is configured to acquire clean samples and generate adversarial samples based on the clean samples;

[0158] A goal determination module 73, configured to determine the learning goal of the student model;

[0159] A target optimization module 74 is configured to optimize the learning target based on a self-learning distillation mechanism;

[0160] The model training module 75 is used to guide the student model to complete the learning of clean samples and adversarial samples based on the clean teacher model, and obtain an optimized student model that achieves the learning goal; based on the clean teacher model and the optimized student model, guide the student model to complete the learning of clean samples and adversarial samples, and obtain a target model that achieves the optimized learning goal.

[0161] Furthermore, if Figure 8 As shown, based on the above-mentioned clean teacher-based robust distillation defense method and clean teacher-based robust distillation defense system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0162] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 20 stores a clean teacher-based adversarial robust distillation defense program 40, which can be executed by the processor 10 to implement the method of the present invention.

[0163] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the method.

[0164] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch screen, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0165] In one embodiment, when the processor 10 executes the program 40 in the memory 20 , the steps of the above clean teacher-based adversarial robust distillation defense method are implemented.

[0166] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program, and when the program is executed by a processor, the steps of the method described above are implemented.

[0167] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0168] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0169] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A clean teacher-based adversarial robust distillation defense method, characterized by, The method comprises: Determine a student model and a preset teacher model, wherein the preset teacher model is a clean teacher model; Obtaining a preset sample, generating an adversarial sample based on the preset sample, and determining a learning objective of the student model, wherein the preset sample is a clean sample; Based on the preset teacher model, guide the student model to complete the learning of preset samples and adversarial samples, and obtain an optimized student model that achieves the learning goal; Optimizing the learning objective based on a self-learning distillation mechanism; Based on the preset teacher model and the optimized student model, the student model is guided to complete the learning of preset samples and adversarial samples, and a target model that completes the optimized learning goal is obtained.

2. The clean teacher-based adversarial robust distillation defense method according to claim 1, characterized in that Generating an adversarial sample based on the preset sample specifically comprises: attacking the preset sample to generate an adversarial sample corresponding to the preset sample; The attacking the preset sample includes: using an optimization-based adversarial attack method to add disturbance to the preset sample under a norm constraint, wherein the amplitude of the disturbance does not exceed a preset disturbance amplitude threshold.

3. The clean teacher-based adversarial robust distillation defense method according to claim 1, characterized in that The learning objectives of the student model are determined as follows: Determining a knowledge distillation contrast loss function for measuring the difference in output probabilities between the preset teacher model and the student model; A learning objective of the student model is determined based on the knowledge distillation contrast loss function.

4. The clean teacher-based adversarial robust distillation defense method according to claim 3, characterized in that The learning objectives include at least a first learning objective and a second learning objective, and the learning objectives of the student model determined based on the knowledge distillation contrast loss function include: Determining a first adjustable parameter of the preset sample and a second adjustable parameter of the adversarial sample; Inputting the preset sample into the preset teacher model to obtain a first output; Inputting the preset sample and the adversarial sample into the student model to obtain a second output based on the preset sample and a third output based on the adversarial sample; Obtaining a first learning objective based on the first output, the second output, and the first adjustable parameter; A second learning objective is obtained based on the first output, the third output, and the second adjustable parameter.

5. The clean teacher-based adversarial robust distillation defense method according to claim 4, characterized in that The self-learning distillation mechanism is based on which the learning objectives are optimized. Re-determining the first adjustable parameter and the second adjustable parameter, and adding a third adjustable parameter based on a self-learning distillation mechanism; Obtaining optimized first learning objectives and second learning objectives based on the re-determined first adjustable parameter and second adjustable parameter; obtaining a third learning objective based on the second output, the third output, and the third adjustable parameter; The optimized learning objectives at least include: an optimized first learning objective, an optimized second learning objective, and the third learning objective.

6. The clean teacher-based adversarial robust distillation defense method according to claim 1, characterized in that The step of guiding the student model to learn preset samples and adversarial samples based on the preset teacher model includes: Inputting the preset sample into the preset teacher model to obtain a first output; generating a first soft label based on the first output; Based on the first soft label, the student model is guided to complete learning of preset samples and adversarial samples.

7. The clean teacher-based adversarial robust distillation defense method according to claim 6, characterized in that The step of guiding the student model to complete learning of preset samples and adversarial samples based on the preset teacher model and the optimized student model includes: Inputting the adversarial example into the optimized student model to obtain a third output; generating a second soft label based on the third output; Based on the first soft label and the second soft label, the student model is guided to complete learning of the preset samples and the adversarial samples.

8. A clean teacher-based adversarial robust distillation defense system, characterized by, The clean teacher-based adversarial robust distillation defense system includes: A model determination module is used to determine a student model and a preset teacher model, wherein the preset teacher model is a clean teacher model; A sample acquisition module is used to acquire a preset sample and generate an adversarial sample based on the preset sample, where the preset sample is a clean sample; A goal determination module, used to determine the learning goal of the student model; A target optimization module, configured to optimize the learning target based on a self-learning distillation mechanism; A model training module is used to guide the student model to complete the learning of preset samples and adversarial samples based on the preset teacher model, and obtain an optimized student model that achieves the learning objectives; based on the preset teacher model and the optimized student model, guide the student model to complete the learning of preset samples and adversarial samples, and obtain a target model that achieves the optimized learning objectives.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a clean teacher-based adversarial robust distillation defense program stored in the memory and runnable on the processor. When the clean teacher-based adversarial robust distillation defense program is executed by the processor, the steps of the clean teacher-based adversarial robust distillation defense method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a clean teacher-based adversarial robust distillation defense program, and when the clean teacher-based adversarial robust distillation defense program is executed by a processor, the steps of the clean teacher-based adversarial robust distillation defense method according to any one of claims 1 to 7 are implemented.

Images