Plug-and-play selective module anti-robust fine tuning method

Abstract

The invention relates to the field of attack and defense resisting of computer vision, in particular to a plug-and-play selective module robust resisting fine tuning method, which comprises the following steps of: introducing a generalization sensitivity index and a robust key index of a module, and based on the robust key index and the generalization sensitivity index of each layer of network module contained in an initial model; determining a network module for implementing fine tuning by taking maximization of a robust key index and minimization of a generalization sensitivity index as targets; and finally, performing fine tuning on the parameters of the network module subjected to fine tuning by freezing the parameters of all other modules and utilizing a fine tuning sample set to obtain a robust enhanced image classification model, thereby comprehensively considering the relationship between model robustness and generalization, accurately identifying the most critical module and performing fine tuning, and improving the robustness of the image classification model. The model robustness is greatly enhanced under the condition of keeping or even slightly improving generalization, and the method has a plug-and-play characteristic, can be deployed in various adversarial trainings as a general component, and can perform secondary robustness enhancement on a robust model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision adversarial attack and defense, and in particular to a plug-and-play selective module adversarial robust fine-tuning method. Background Art

[0002] Deep neural networks have achieved remarkable success in a range of real-world tasks due to their powerful performance. However, deep neural networks are vulnerable to adversarial examples, which pose a significant security risk to critical applications. Adversarial examples can mislead the model into making incorrect classifications or detections by adding tiny, imperceptible perturbations to the input image. This attack not only affects the recognition accuracy of the model, but may also pose a serious threat to the security of systems such as remote sensing and face recognition. In addition, the concealment of adversarial attacks makes their threat even more serious in sensitive areas. For example, in military reconnaissance missions, the enemy may use adversarial attacks to interfere with the automatic target recognition system, making it unable to correctly identify ground targets, thereby affecting the decision-making process. Therefore, there is an urgent need to improve the robustness against such attacks.

[0003] At present, the defense methods against adversarial attacks mainly include sample detection, data enhancement, and adversarial training. Among them, adversarial training, also known as Adversarial Training, has been proven to be one of the most effective strategies in recent years. It introduces adversarial samples during the training process, allowing the model to actively learn and adapt to the attacker's data distribution, thereby improving the model's resistance to adversarial perturbations. For a l-layer deep neural network f with a parameter of θ, θ , and from the standard distribution Standard dataset extracted from Standard adversarial training can be formulated as a min-max optimization problem:

[0004]

[0005] in, represents the robust loss under adversarial samples, represents the objective loss function and δ represents the adversarial perturbation constrained by a bounded set Δ.

[0006] However, adversarial training often leads to a decrease in the performance of the model on benign data distributions, resulting in the so-called "robustness-generalization trade-off" problem. This means that the model performs well when dealing with adversarial samples, but the accuracy on clean samples is reduced. The so-called clean samples are original samples without added perturbations, which is unacceptable for tasks that require high precision, such as remote sensing image processing and autonomous driving technology. In addition, this problem is generally considered to be an inherent defect of deep neural networks in past studies, such as:

[0007] First, Tsipras et al. found that there may be an inherent contradiction between the adversarial robustness goal and the standard generalization goal. They showed that even in a fairly simple and natural environment, this trade-off between the standard accuracy of the model and its robustness to adversarial perturbations exists. These findings are also confirmed that similar phenomena are observed in more complex environments.

[0008] Second, Schmidt et al. conducted a large number of experiments and showed that adversarial samples, due to the inherent sample complexity, will cause a separation between the standard accuracy and robust accuracy of the model;

[0009] Javanmard et al. provide a precise and comprehensive understanding of the role of adversarial training in linear regression with Gaussian features, characterizing the fundamental trade-off between the accuracy that any algorithm can achieve regardless of computational power or training data size. They also provide a precise characterization of the standard accuracy and robust accuracy and their corresponding trade-offs achieved by the currently popular minimax adversarial training method in a high-dimensional setting.

[0010] 4. Raghunathan et al. accurately described the impact of enhancement on the standard error of linear regression when the standard error and robust error of the optimal linear predictor are zero, and found that even if the enhanced disturbance is the same as the noise-free observation value of the optimal linear predictor, the standard error will increase and the trade-off problem still exists.

[0011] In recent years, more and more research has focused on solving this trade-off problem, and methods such as TRADES, SCORE, MART, adversarial training guided by learnable boundaries, and random adversarial training through Taylor expansion have been proposed. However, these methods are committed to optimizing "non-robust" models into "robust" models. They usually treat the model as a whole during the training process and fail to fully consider the unique role and contribution of each layer of the model. This holistic strategy limits the potential of the model and leads to insufficient utilization of the model capacity.

[0012] Further research, by deleting, rearranging or reinitializing some residual blocks of the residual network after standard adversarial training, found that there are differences in the network modules of each layer of the deep neural network during the training process. Therefore, some scholars have proposed the latest RiFT (Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning, ICCV 2023) technology, which aims to improve the generalization of the "robust" model obtained through adversarial training to alleviate the "robust-generalization trade-off" problem. Specifically, RiFT introduces the Module Robust Criticality (MRC) indicator, which is an indicator that measures the importance of a given network module to the robustness of the model under the worst-case weight perturbation. It identifies the network module with the lowest MRC value as a non-robust critical module and fine-tunes its weight to obtain the fine-tuned weight.

[0013] However, as pointed out in the literature of RiFT technology, only by fine-tuning on non-robust key modules can adversarial robustness be preserved; compared with fine-tuning all layers, fine-tuning on robust key modules shows the worst trade-off between generalization and robustness. Figure 1 If we fine-tune the robust key modules, improving the model robustness will reduce generalization, and improving generalization will reduce robustness, which deepens the robustness-generalization trade-off problem. Therefore, the essence of RiFT technology is to use MRC to characterize robust redundancy, while maintaining robustness while improving generalization.

[0014] Therefore, although the RiFT technology proposes the MRC indicator, the MRC indicator represents the sharpness of the robustness loss regarding parameter fine-tuning, and is mainly used to avoid the impact of fine-tuning on robustness. It does not have guiding significance for performing robust enhancement fine-tuning while alleviating the robustness-generalization trade-off problem. Summary of the invention

[0015] The technical problem to be solved by the present invention is to propose a plug-and-play selective module adversarial robust fine-tuning method, which can comprehensively consider the relationship between model robustness and generalization, accurately identify the most critical modules for fine-tuning, and greatly enhance the robustness of the model while maintaining or even slightly improving the generalization.

[0016] The technical solution adopted by the present invention to solve the above technical problems is:

[0017] A plug-and-play selective module adversarial robust fine-tuning method comprises the following steps:

[0018] S1, input the pre-trained image classification model as the initial model;

[0019] S2. For each layer of network modules contained in the initial model, the robustness criticality index and generalization sensitivity index of each layer of network modules are calculated according to the following steps:

[0020] S21. Obtain the disturbance model:

[0021] Freeze the parameters of other network modules included in the initial model. The other network modules in this step are network modules other than the current computing network module included in the initial model. For the current computing network module, use the adversarial sample set to maximize the difference in robustness of the model before and after the perturbation, add perturbations to the parameters of the current computing network module, and obtain the first perturbation model corresponding to the current computing network module. For the current computing network module, use the clean sample set to maximize the difference in generalization of the model before and after the perturbation, and add perturbations to the parameters of the current computing network module, and obtain the second perturbation model corresponding to the current computing network module.

[0022] The clean sample set is a set of clean samples, the adversarial sample set is a set of adversarial samples, the clean samples are original samples without adding disturbances, and the adversarial samples are samples obtained after adding adversarial disturbances to the original samples;

[0023] S22. Index calculation:

[0024] Using samples of the adversarial sample set as input, the initial model and the first perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the first loss of the current computing network module, and the classification loss of the first perturbation model corresponding to the current computing network module is used as the second loss of the current computing network module; based on the changes in the first loss and the second loss of the current computing network module, the robustness criticality index of the current computing network module is calculated;

[0025] Using samples of the test sample set as input, the initial model and the second perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the third loss of the current computing network module, and the classification loss of the second perturbation model corresponding to the current computing network module is used as the fourth loss of the current computing network module; based on the changes in the third loss and the fourth loss of the current computing network module, the generalization sensitivity index of the current computing network module is calculated; the samples of the test sample set are original samples that have not been sampled in the model training stage;

[0026] S3. Based on the robust criticality index and generalization sensitivity index of each layer of network modules contained in the initial model, with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index, determine the network module for fine-tuning;

[0027] S4. Freeze the parameters of other network modules included in the initial model, and only fine-tune the parameters of the network modules that are fine-tuned using the fine-tuning sample set to obtain a robustly enhanced image classification model; the other network modules in this step are network modules other than the network modules that are fine-tuned and included in the initial model; the samples in the fine-tuning sample set are adversarial samples.

[0028] Further, in step S21, for the network module currently being calculated, using samples of the corresponding sample set, with the goal of maximizing the difference in optimization objectives of the model before and after adding the disturbance, a disturbance is added to the parameters of the current computing network module to obtain a disturbance model corresponding to the current computing network module, wherein the optimization objective is robustness when constructing the first disturbance model and generalization when constructing the second disturbance model; specifically including:

[0029] A1. Use the initial model as the current model;

[0030] A2. Input the samples in the corresponding sample set, use the current model, and obtain the predicted classification of each input sample; use the label classification and predicted classification of each input sample to calculate the classification loss of the current model;

[0031] A3. Using the classification loss of the current model, based on the gradient ascent algorithm, with the goal of maximizing the difference in the optimization objectives of the model before and after adding the disturbance, add disturbances to the parameters of the current computing network module of the current model;

[0032] A4. Determine whether the preset iteration round is reached. If so, add the perturbed model in step A3 as the perturbation model corresponding to the current computing network module; otherwise, add the perturbed model in step A3 as the current model and return to step A2.

[0033] Furthermore, perturbations are added to the parameters of the current computing network module of the current model, including:

[0034] A31. Calculate the added disturbance Δθ according to the following formula: t :

[0035]

[0036] Among them, t represents the round of the current iteration, is the classification loss of the jth input sample classified by the current model, f t represents the current model, x j is the jth sample of the input, y j is the label of the jth input sample, and J is the number of input samples; represents the gradient of loss; α is the step size of adding perturbations;

[0037] A32. Calculate the disturbance change Δd according to the following formula;

[0038] Δd=(θ t-1 +Δθ t )-θ0

[0039] Among them, θ0 is the parameter of the current computing network module in the initial model, θ t-1 are the parameters obtained by the current computing network module of the current model in the t-1th round;

[0040] A33, take ∥Δd∥2<∈∥θ0∥2 as the disturbance constraint;

[0041] If ∥Δd∥2<∈∥θ0∥2, then add disturbance to the parameters of the current computing network module of the current model according to the following formula:

[0042] θ t ←θ t-1 +Δθ t

[0043] If ∥Δd∥2≥∈∥θ0∥2, then add disturbance to the parameters of the current computing network module of the current model according to the following formula:

[0044]

[0045] Among them, ∥·∥2 is the two-norm, ∈ is the perturbation scaling factor, θ t It is the parameter obtained by the current iteration update of the current computing network module of the current model.

[0046] Furthermore, the preset iteration rounds T is 10 times, and the disturbance scaling factor ∈ is 0.1.

[0047] Furthermore, in step S22, the robust criticality index MRC is calculated according to the following formula:

[0048]

[0049] In step S22, the generalization sensitivity index MGSI is calculated according to the following formula:

[0050]

[0051] in, represents the first perturbation model, represents the second perturbation model, represents the initial model; x is the input sample, y is the label of the input sample, is the loss function, k1 and k2 are the sample numbers; K1 is the number of samples in the adversarial sample set, and K2 is the number of samples in the test sample set.

[0052] Furthermore, in step S3, based on the robust criticality index and generalization sensitivity index of each layer of network modules included in the initial model, with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index, the network module to be fine-tuned is determined, including:

[0053] S31, normalizing the robustness criticality index of each layer of network modules included in the initial model; normalizing the generalization sensitivity index of each layer of network modules included in the initial model;

[0054] S32. Determine the network module for fine-tuning according to the following formula:

[0055]

[0056] in, represents the robust criticality index of the i-th layer network module obtained by normalization, represents the generalization sensitivity index obtained by normalization of the i-th layer network module, and k is the serial number of the fine-tuning network module.

[0057] Further, in step S4, the parameters of other network modules included in the initial model are frozen, and only the parameters of the network module to be fine-tuned are fine-tuned using the fine-tuning sample set according to the following objective function to obtain the adjusted model parameters θ FT :

[0058]

[0059] Among them, x represents the part of the samples in the fine-tuning sample set that belongs to the original samples, δ represents the part of the samples in the fine-tuning sample set that belongs to the added disturbance, Δ is the value range of δ, and y is the label of the original sample corresponding to the sample in the fine-tuning sample set. is the loss function; represents the model obtained by freezing the parameters of other network modules except the k-th layer network module and fine-tuning the k-th layer network module. represents the sample space, θ FT The parameters of the model are obtained by freezing the parameters of the network modules except the k-th layer network module based on the objective function and fine-tuning the k-th layer network module, where k is the serial number of the fine-tuned network module.

[0060] Furthermore, step S4 further includes:

[0061] Using a set of mixing coefficients η, θ is adjusted according to the following formula AT and θ FT Combine to obtain a set of parameters to be evaluated:

[0062] θ η p=(1-ηp )·θ AT +η p ·θ FT

[0063] Among them, η p is the pth mixing coefficient η and η p ∈[0,1],θ AT are the parameters of the initial model, θ FT are the parameters of the model obtained after fine-tuning the k-th layer network module, where k is the serial number of the fine-tuned network module;

[0064] A robustness evaluation is performed on the model corresponding to each parameter to be evaluated, and the parameter to be evaluated with the highest adversarial accuracy and the best standard accuracy is selected as the final model parameter. The robustly enhanced image classification model is constructed with the final model parameters.

[0065] Furthermore, the adversarial samples in the adversarial sample set and the fine-tuning sample set are both generated using the PGD-10 model, and the attack strength of the adversarial samples included in the fine-tuning sample set is higher than the attack strength of the adversarial samples included in the adversarial sample set.

[0066] The beneficial effects of the present invention are:

[0067] The method of the present invention introduces the robust criticality index (Module Robust Criticality, MRC) of the RiFT technology. However, unlike the RiFT technology, which enhances generalization without losing robustness, the method of the present invention is to enhance robustness without losing generalization. That is, the present invention needs to adjust the modules with MRC as high as possible to avoid the passivation of parameter disturbances by the adjusted modules and ensure the enhancement of robustness. Therefore, the substantive problem that the method of the present invention needs to solve is: how to use MRC to determine the adjustment module while overcoming the contradiction pointed out by the RiFT technology that "only fine-tuning on non-robust key modules can retain adversarial robustness; compared with fine-tuning all layers, fine-tuning the robustness key modules shows the worst trade-off between generalization and robustness."

[0068] The inventors believe that the robustness of the model to adversarial samples mainly comes from the tiny adversarial perturbations during the fitting adversarial training process. Since such perturbations do not exist in clean samples, using clean samples to fine-tune modules with higher MRC values ​​will greatly reduce the robustness of the model. On the contrary, adversarial samples are essentially clean samples with specific noise, which retain a lot of original clean information. Therefore, after fine-tuning using adversarial samples, the generalization of the model will not be easily reduced.

[0069] Therefore, based on MRC, the inventors defined the Module Generalization Sensitivity Index (MGSI), and used MGSI to quantify the sensitivity of the module to generalization under specific weight perturbations. Based on MGSI and MRC indicators, the inventors proposed the method of the present invention, a plug-and-play Selective Module Adversarial Robust fine-Tuning (SMART) method, which can comprehensively consider the relationship between model robustness and generalization, accurately identify the most critical modules for fine-tuning, and greatly enhance the model robustness while maintaining or even slightly improving generalization.

[0070] The inventors have developed a calculation step for MGSI that is basically consistent with MRC, so that the calculation processes of the two can be easily integrated and the plug-and-play feature can be retained. As a result, the method of the present invention can be deployed as a universal component on various advanced adversarial training methods, and has high feasibility and effectiveness for different model architectures, data scales, different attack intensities and attack strategies.

[0071] Based on the plug-and-play feature, it should be pointed out that in the existing field of robust fine-tuning, most technologies are completely fine-tuned, that is, fine-tuning the network modules of all layers, and are committed to optimizing the "non-robust" model to a "robust" model; and the method of the present invention can be combined with other methods to achieve the model from "non-robust" to "robust" and then to "more robust". That is, the method of the present invention also pioneered the concept of secondary revision of the robust model, which can further consolidate the model's ability to resist attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 The present invention is a flowchart of a plug-and-play selective module robust fine-tuning method. DETAILED DESCRIPTION

[0073] This paper aims to propose a plug-and-play selective module adversarial robust fine-tuning method, which integrates MGSI and MRC indicators, comprehensively considers the relationship between model robustness and generalization from a fine-grained module-level perspective, accurately identifies the most critical modules for fine-tuning, and greatly enhances the model robustness while maintaining or even slightly improving generalization. Figure 1 As shown, the following steps are included:

[0074] S1, input the pre-trained image classification model as the initial model;

[0075] S2. For each layer of network modules contained in the initial model, the robustness criticality index and generalization sensitivity index of each layer of network modules are calculated according to the following steps:

[0076] S21. Obtain the disturbance model:

[0077] Freeze the parameters of other network modules included in the initial model. The other network modules in this step are network modules other than the current computing network module included in the initial model. For the current computing network module, use the adversarial sample set to maximize the difference in robustness of the model before and after the perturbation, add perturbations to the parameters of the current computing network module, and obtain the first perturbation model corresponding to the current computing network module. For the current computing network module, use the clean sample set to maximize the difference in generalization of the model before and after the perturbation, and add perturbations to the parameters of the current computing network module, and obtain the second perturbation model corresponding to the current computing network module.

[0078] The clean sample set is a set of clean samples, the adversarial sample set is a set of adversarial samples, the clean samples are original samples without adding disturbances, and the adversarial samples are samples obtained after adding adversarial disturbances to the original samples;

[0079] S22. Index calculation:

[0080] Using samples of the adversarial sample set as input, the initial model and the first perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the first loss of the current computing network module, and the classification loss of the first perturbation model corresponding to the current computing network module is used as the second loss of the current computing network module; based on the changes in the first loss and the second loss of the current computing network module, the robustness criticality index of the current computing network module is calculated;

[0081] Using samples of the test sample set as input, the initial model and the second perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the third loss of the current computing network module, and the classification loss of the second perturbation model corresponding to the current computing network module is used as the fourth loss of the current computing network module; based on the changes in the third loss and the fourth loss of the current computing network module, the generalization sensitivity index of the current computing network module is calculated; the samples of the test sample set are original samples that have not been sampled in the model training stage;

[0082] S3. Based on the robust criticality index and generalization sensitivity index of each layer of network modules contained in the initial model, with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index, determine the network module for fine-tuning;

[0083] S4. Freeze the parameters of other network modules included in the initial model, and only fine-tune the parameters of the network modules that are fine-tuned using the fine-tuning sample set to obtain a robustly enhanced image classification model; the other network modules in this step are network modules other than the network modules that are fine-tuned and included in the initial model; the samples in the fine-tuning sample set are adversarial samples.

[0084] The scheme of the present invention is further described below in conjunction with embodiments.

[0085] Example

[0086] A plug-and-play selective module adversarial robust fine-tuning method comprises the following steps:

[0087] S1. Input model

[0088] The purpose of this step is to obtain a model for fine-tuning, that is, to input the pre-trained image classification model as the initial model.

[0089] The implementation of the present invention is to fine-tune the parameters of a certain layer of the network module of the model based on the robust loss of adversarial samples. Therefore, it does not depend on the category of the model. The image classification model can be any existing model, such as: ResNet18, ResNet34, ResNet50, WideResNet34-10.

[0090] The pre-training here means that the model is a model that has been trained and can complete the target task. However, it should be noted that the training can be conventional training, that is, a "non-robust" model; it can also be conventional training followed by adversarial training, that is, a "robust" model. Preferably, the pre-trained image classification model is a model obtained after conventional training and then adversarial training.

[0091] S2. Index calculation

[0092] The purpose of this step is to calculate the robust criticality index MRC and generalization sensitivity index MGSI of each layer of the network module of the input model in step S1. Therefore, for each layer of network modules contained in the initial model, the robust criticality index and generalization sensitivity index of each layer of network modules are calculated according to the following steps:

[0093] S21. Obtaining the disturbance model

[0094] Freeze the parameters of other network modules included in the initial model. The other network modules in this step are network modules other than the current computing network module included in the initial model.

[0095] For the network module currently being calculated, using the adversarial sample set, with the goal of maximizing the robustness difference of the model before and after adding the disturbance, add disturbances to the parameters of the current computing network module to obtain the first perturbation model corresponding to the current computing network module;

[0096] For the current computing network module, using a clean sample set, with the goal of maximizing the generalization difference of the model before and after adding the disturbance, perturbations are added to the parameters of the current computing network module to obtain a second perturbation model corresponding to the current computing network module.

[0097] The above goal is to maximize the difference in the optimization objectives of the model before and after adding disturbances, which can be formulated as:

[0098]

[0099] The optimization objective is robustness when constructing the first perturbation model and generalization when constructing the second perturbation model; θ represents a deep neural network model with parameter θ, is a set of weight perturbations with scale invariance, Δθ represents the perturbation relative to θ i The parameter perturbation, θ i is the parameter of the i-th layer network module of the deep neural network model, that is, the current computing network module. represents the robust loss, is the corresponding sample set.

[0100] The clean sample set is a set of clean samples, and the clean samples are original samples without adding disturbances. The original samples can use general datasets such as CIFAR10, CIFAR100, TinyImageNet, SVHN, ImageNet, or aerospace remote sensing image datasets such as Aerial Landscapes.

[0101] The adversarial sample set is a collection of adversarial samples, and the adversarial samples are samples obtained by adding adversarial perturbations to the original samples. The adversarial samples can be generated using existing methods, including: PGD-1 ∞ , PGD-l2, FGSM, CW-20 and AutoAttack, etc. In the subsequent verification experiments, the adversarial sample sets for calculating the MRC / MGSI indicators and subsequent fine-tuning were all generated by PGD-10.

[0102] Based on the objective function, a perturbation model is established, and the existing method can be used, preferably, the gradient ascent method is used. Specifically, in this step, for the network module currently being calculated, the samples of the corresponding sample set are used, and the optimization target difference of the model before and after the perturbation is maximized, and the perturbation model corresponding to the current computing network module is obtained, which specifically includes:

[0103] A1. Use the initial model as the current model;

[0104] A2. Input the samples in the corresponding sample set, use the current model, and obtain the predicted classification of each input sample; use the label classification and predicted classification of each input sample to calculate the classification loss of the current model;

[0105] A3. Using the classification loss of the current model, based on the gradient ascent algorithm, with the goal of maximizing the difference in the optimization target of the model before and after adding the disturbance, add disturbances to the parameters of the current computing network module of the current model, including:

[0106] A31. Calculate the added disturbance Δθ according to the following formula: t :

[0107]

[0108] Among them, t represents the round of the current iteration, is the classification loss of the jth input sample classified by the current model, f t represents the current model, x j is the jth sample of the input, y j is the label of the jth input sample, and J is the number of input samples; represents the gradient of loss; α is the step size of adding perturbations;

[0109] A32. Calculate the disturbance change Δd according to the following formula;

[0110] Δd=(θ t-1 +Δθ t )-θ0

[0111] Among them, θ0 is the parameter of the current computing network module in the initial model, θ t-1 are the parameters obtained by the current computing network module of the current model in the t-1th round;

[0112] A33, take ∥Δd∥2<∈∥θ0∥2 as the disturbance constraint;

[0113] If ∥Δd∥2<∈∥θ0∥2, then add disturbance to the parameters of the current computing network module of the current model according to the following formula:

[0114] θt ←θ t-1 +Δθ t

[0115] If ∥Δd∥2≥∈∥θ0∥2, then add disturbance to the parameters of the current computing network module of the current model according to the following formula:

[0116]

[0117] Among them, ∥·∥2 is the two-norm, ∈ is the perturbation scaling factor, θ t The parameters of the current computing network module of the current model obtained through the current iteration update;

[0118] A4. Determine whether the preset iteration round is reached. If so, add the perturbed model in step A3 as the perturbation model corresponding to the current computing network module; otherwise, add the perturbed model in step A3 as the current model and return to step A2.

[0119] According to different original data and input models, the preset iteration rounds T is 10 times, the perturbation scaling factor ∈ is 0.1, the number of samples J input in each iteration is set to 128, and a lower learning rate such as α=0.001 is set. Through a lower learning rate and fewer rounds, excessive fine-tuning of the model can be avoided, thereby improving the efficiency and effect of fine-tuning.

[0120] The iterative process of the first perturbation model and the second perturbation model is the same, the only difference is the input samples.

[0121] S22. Index calculation

[0122] Calculation of MRC: Use samples from the adversarial sample set as input, input the initial model and the first perturbation model corresponding to the current computing network module respectively, and use the classification loss of the initial model as the first loss of the current computing network module, and use the classification loss of the first perturbation model corresponding to the current computing network module as the second loss of the current computing network module; based on the change in the first loss and the second loss of the current computing network module, calculate and obtain the robust criticality index of the current computing network module.

[0123] The above calculation can be expressed in the form of:

[0124]

[0125] Calculation of MGSI: Use samples of the test sample set as input, input the initial model and the second perturbation model corresponding to the current computing network module respectively, and use the classification loss of the initial model as the third loss of the current computing network module, and use the classification loss of the second perturbation model corresponding to the current computing network module as the fourth loss of the current computing network module; based on the changes in the third loss and the fourth loss of the current computing network module, calculate and obtain the generalization sensitivity index of the current computing network module.

[0126] The above calculation can be expressed in the form of:

[0127]

[0128] in, represents the first perturbation model, represents the second perturbation model, represents the initial model; x is the input sample, y is the label of the input sample, is the loss function, k1 and k2 are the sample numbers; K1 is the number of samples in the adversarial sample set, and K2 is the number of samples in the test sample set.

[0129] The samples of the test sample set are original samples that have not been sampled in the model training stage. The training sample set and the test sample set are usually obtained by dividing the original data set in proportion.

[0130] S3. Determine module

[0131] This step determines the network module to be adjusted based on the index calculated in step S2.

[0132] The selection of fine-tuning modules is based on two indicators: MRC and MGSI. An ideal module should have both high MRC and low MGSI values, which means that the module contributes significantly to the robustness of the model while having little impact on the generalization performance. That is, based on the robust criticality index and generalization sensitivity index of each layer of network modules contained in the initial model, the network module for fine-tuning is determined with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index.

[0133] But it should be noted that MRC and MGSI are obtained from different data distributions, so the steps include:

[0134] S31, normalizing the robustness criticality index of each layer of network modules included in the initial model; normalizing the generalization sensitivity index of each layer of network modules included in the initial model. In this embodiment, Min-Max normalization is adopted.

[0135] S32. Determine the network module for fine-tuning according to the following formula:

[0136]

[0137] in, represents the robust criticality index of the i-th layer network module obtained by normalization, represents the generalization sensitivity index obtained by normalization of the i-th layer network module, and k is the serial number of the fine-tuning network module.

[0138] In addition to the above methods, the inventors also explored It was found that as long as the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index was used, it had little effect on the final determined regulation module.

[0139] S4. Parameter adjustment

[0140] In this step, the parameters of other network modules included in the initial model are frozen, and only the parameters of the network modules that are fine-tuned are fine-tuned using the fine-tuning sample set to obtain a robustly enhanced image classification model; the other network modules in this step are network modules other than the network modules that are fine-tuned and included in the initial model.

[0141] The samples in the fine-tuning sample set are adversarial samples. In this embodiment, the adversarial samples in the adversarial sample set and the fine-tuning sample set are all generated using the PGD-10 model, and because a stronger attack is required to drive the improvement of the overall performance of the model when fine-tuning a single module at a lower learning rate, the attack strength of the adversarial samples contained in the fine-tuning sample set is higher than the attack strength of the adversarial samples contained in the adversarial sample set.

[0142] Parameter fine-tuning may be performed in an existing manner, which in this embodiment includes:

[0143] First, freeze the parameters of other network modules contained in the initial model, and only fine-tune the parameters of the network module to be fine-tuned according to the following objective function using the fine-tuning sample set to obtain the adjusted model parameters θ FT :

[0144]

[0145] Among them, x represents the part of the samples in the fine-tuning sample set that belongs to the original samples, δ represents the part of the samples in the fine-tuning sample set that belongs to the added disturbance, Δ is the value range of δ, and y is the label of the original sample corresponding to the sample in the fine-tuning sample set. is the loss function; represents the model obtained by freezing the parameters of other network modules except the k-th layer network module and fine-tuning the k-th layer network module. represents the sample space, θ FTThe parameters of the model are obtained by freezing the parameters of the network modules except the k-th layer network module based on the objective function and fine-tuning the k-th layer network module, where k is the serial number of the fine-tuned network module.

[0146] max refers to finding the best δ in the convex set Δ, so that when the clean original sample is added with this δ and input into the model, the model's prediction error is the highest; min means finding the optimal model parameters under the strongest adversarial perturbation so that the model's test error expectation is minimized.

[0147] Then, a set of mixing coefficients η is used to adjust θ according to the following formula: AT and θ FT Combine to obtain a set of parameters to be evaluated:

[0148]

[0149] Among them, η p is the pth mixing coefficient η and η p ∈[0,1],θ AT are the parameters of the initial model, θ FT is the parameter of the model obtained after fine-tuning the k-th layer network module, and k is the serial number of the network module for fine-tuning. In this embodiment, the 10 mixing coefficients η are respectively 0.1, 0.2, ..., 1.0.

[0150] After that, the model corresponding to each parameter to be evaluated is evaluated for robustness, and the parameters to be evaluated with the highest adversarial accuracy and the best standard accuracy are selected as the final model parameters, and the robust enhanced image classification model is constructed with the final model parameters. The model evaluation process is based on different attack strategies, such as PGD-10, AutoAttack, CW-20, and FGSM. At the same time, different attack intensities and attack norms are used in the PGD attack to conduct comprehensive and rigorous tests on the robustness and generalization performance of the model to ensure its stability and effectiveness under adversarial attacks.

[0151] Experimental verification

[0152] In order to verify the effectiveness of the present invention, comprehensive experiments were carried out based on the scheme of the embodiment using different image classification datasets: CIFAR10, CIFAR100, TinyImageNet, and popular CNN-based model architectures: ResNet18, ResNet34, and WideResNet34-10.

[0153] For the evaluation setting, the method evaluates the generalization and robustness of the fine-tuned model by measuring the prediction accuracy on the standard and adversarial test sets, respectively. The adversarial datasets are generated using common attacks, including PGD-l ∞, PGD-12, FGSM, CW-20 and AutoAttack. ∞ In the attack, three different attack strengths ε were used: 8 / 255, 10 / 255, and 12 / 255.

[0154] Table 1 shows the evaluation results of different data sets and model backbones under four different types of adversarial attacks; Table 2 shows the evaluation results of different data sets and model backbones under attacks of different strengths and norms. In the table, AT represents the model obtained after adversarial attack training, that is, the model before fine-tuning using the method of the present invention; SMART represents the model after the AT model is fine-tuned by the method of the present invention; Clean represents clean test samples, which are used to evaluate generalization performance; PGD, FGSM, CW-20 and AA (AutoAttack) represent different attack samples, which are used to evaluate robust performance; △ represents the percentage of performance change before and after fine-tuning by the present invention; l2, 8 / 255, 10 / 255, and 12 / 255 in the table represent attack strength and norm.

[0155] Table 1. Evaluation results of different datasets and model backbones under different adversarial attack types (unit %)

[0156]

[0157] As shown in Table 1, for different types of attacks, this method significantly improves the robustness of all models by about 0.5% to 2.5%, and the generalization of all models also shows varying degrees of enhancement.

[0158] Table 2. Evaluation results of different datasets and model backbones under different attack intensities and norms (unit %)

[0159]

[0160] As shown in Table 2, for attacks of different strengths and norms, our method significantly improves the robustness of all models by about 1% to 2.5%. Except for a few models that show a slight loss in generalization, most models show varying degrees of enhancement.

[0161] Table 3 shows the network modules of each model finally fine-tuned in the experiments of Tables 1 and 2. Table 4 shows the hyperparameter settings in the experiments of Tables 1 and 2, in the format of “PGD-10 maximum perturbation limit / PGD-10 perturbation step size / fine-tuning rounds / learning rate”.

[0162] Table 3. The final fine-tuned network modules of each model

[0163] CIFAR10 CIFAR100 Tiny-ImageNet ResNet18 layer4.1.conv2 layer4.1.conv2 layer4.0.conv2 ResNet34 layer4.0.conv2 layer4.1.conv2 layer4.0.conv1 WRN34-10 block3.layer.1.conv1 block3.layer.3.conv1 block3.layer.3.conv1

[0164] Table 4 Hyperparameter settings for different models

[0165] CIFAR10 CIFAR100 Tiny-ImageNet ResNet18 12 / 3 / 20 / 0.001 16 / 4 / 20 / 0.001 12 / 3 / 20 / 0.001 ResNet34 12 / 3 / 10 / 0.001 16 / 4 / 20 / 0.005 12 / 3 / 20 / 0.005 WRN34-10 12 / 3 / 20 / 0.001 16 / 4 / 20 / 0.001 16 / 4 / 20 / 0.001

[0166] Finally, it should be noted that the above embodiments are only preferred implementations and are not intended to limit the present invention. It should be pointed out that for those skilled in the art, several modifications, equivalent replacements, improvements, etc. can be made without departing from the scope of the present invention and the scope of protection of the claims, and all of these should be included in the protection scope of the present invention.

Claims

1. A plug-and-play selective module adversarial robust fine-tuning method, characterized in that: The steps include: S1, input the pre-trained image classification model as the initial model; S2. For each layer of network modules contained in the initial model, the robustness criticality index and generalization sensitivity index of each layer of network modules are calculated according to the following steps: S21. Obtain the disturbance model: Freeze the parameters of other network modules included in the initial model. The other network modules in this step are network modules other than the current computing network module included in the initial model. For the current computing network module, use the adversarial sample set to maximize the difference in robustness of the model before and after the perturbation, add perturbations to the parameters of the current computing network module, and obtain the first perturbation model corresponding to the current computing network module. For the current computing network module, use the clean sample set to maximize the difference in generalization of the model before and after the perturbation, and add perturbations to the parameters of the current computing network module, and obtain the second perturbation model corresponding to the current computing network module. The clean sample set is a set of clean samples, the adversarial sample set is a set of adversarial samples, the clean samples are original samples without adding disturbances, and the adversarial samples are samples obtained after adding adversarial disturbances to the original samples; S22. Index calculation: Using samples of the adversarial sample set as input, the initial model and the first perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the first loss of the current computing network module, and the classification loss of the first perturbation model corresponding to the current computing network module is used as the second loss of the current computing network module; based on the changes in the first loss and the second loss of the current computing network module, the robustness criticality index of the current computing network module is calculated; Using samples of the test sample set as input, the initial model and the second perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the third loss of the current computing network module, and the classification loss of the second perturbation model corresponding to the current computing network module is used as the fourth loss of the current computing network module; based on the changes in the third loss and the fourth loss of the current computing network module, the generalization sensitivity index of the current computing network module is calculated; the samples of the test sample set are original samples that have not been sampled in the model training stage; S3. Based on the robust criticality index and generalization sensitivity index of each layer of network modules contained in the initial model, with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index, determine the network module for fine-tuning; S4. Freeze the parameters of other network modules included in the initial model, and only fine-tune the parameters of the network modules that are fine-tuned using the fine-tuning sample set to obtain a robustly enhanced image classification model; the other network modules in this step are network modules other than the network modules that are fine-tuned and included in the initial model; the samples in the fine-tuning sample set are adversarial samples.

2. A plug-and-play selective module robust fine-tuning method as claimed in claim 1, characterized in that: In step S21, for the network module currently being calculated, using the samples of the corresponding sample set, with the goal of maximizing the difference in the optimization objectives of the model before and after adding the disturbance, a disturbance is added to the parameters of the current calculation network module to obtain a disturbance model corresponding to the current calculation network module, wherein the optimization objective is robustness when constructing the first disturbance model and generalization when constructing the second disturbance model; specifically including: A1. Use the initial model as the current model; A2. Input the samples in the corresponding sample set, use the current model, and obtain the predicted classification of each input sample; use the label classification and predicted classification of each input sample to calculate the classification loss of the current model; A3. Using the classification loss of the current model, based on the gradient ascent algorithm, with the goal of maximizing the difference in the optimization objectives of the model before and after adding the disturbance, add disturbances to the parameters of the current computing network module of the current model; A4. Determine whether the preset iteration round is reached. If so, add the perturbed model in step A3 as the perturbation model corresponding to the current computing network module; otherwise, add the perturbed model in step A3 as the current model and return to step A2.

3. A plug-and-play selective module robust fine-tuning method as claimed in claim 2, characterized in that: Add perturbations to the parameters of the current computational network module of the current model, including: A31. Calculate the added disturbance Δθ according to the following formula: t : Among them, t represents the round of the current iteration, is the classification loss of the jth input sample classified by the current model, f t represents the current model, x j is the jth sample of the input, y j is the label of the jth input sample, and J is the number of input samples; represents the gradient of loss; α is the step size of adding perturbations; A32. Calculate the disturbance change Δd according to the following formula; Δd=(θ t-1 +Δθ t )-θ0 Among them, θ0 is the parameter of the current computing network module in the initial model, θ t-1 are the parameters obtained by the current computing network module of the current model in the t-1th round; A33, use ||Δd||2<∈||θ0||2 as the perturbation constraint; If ||Δd||2<∈||θ0||2, then add disturbance to the parameters of the current computational network module of the current model according to the following formula: i t ←θ t-1 +Δθ t If ||Δd||2≥∈||θ0||2, then add disturbance to the parameters of the current computational network module of the current model according to the following formula: Among them, ||·||2 is the two-norm, ∈ is the perturbation scaling factor, θ t It is the parameter obtained by the current iteration update of the current computing network module of the current model.

4. A plug-and-play selective module robust fine-tuning method as claimed in claim 3, characterized in that: The preset iteration rounds T is 10 times, and the disturbance scaling factor ∈ is 0.

1.

5. The plug-and-play selective module robust fine-tuning method according to claim 1, characterized in that: In step S22, the robust criticality index MRC is calculated according to the following formula: In step S22, the generalization sensitivity index MGSI is calculated according to the following formula: in, represents the first perturbation model, represents the second perturbation model, represents the initial model; x is the input sample, y is the label of the input sample, is the loss function, k1 and k2 are the sample numbers; K1 is the number of samples in the input adversarial sample set, and K2 is the number of samples in the input test sample set.

6. A plug-and-play selective module robust fine-tuning method as claimed in claim 1, characterized in that: In step S3, based on the robust criticality index and generalization sensitivity index of each layer of network modules contained in the initial model, with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index, the network module to be fine-tuned is determined, including: S31, normalizing the robustness criticality index of each layer of network modules included in the initial model; normalizing the generalization sensitivity index of each layer of network modules included in the initial model; S32. Determine the network module for fine-tuning according to the following formula: in, represents the robust criticality index of the i-th layer network module obtained by normalization, represents the generalization sensitivity index obtained by normalization of the i-th layer network module, and k is the serial number of the fine-tuning network module.

7. A plug-and-play selective module robust fine-tuning method as claimed in claim 1, characterized in that: Step S4, freeze the parameters of other network modules included in the initial model, and only fine-tune the parameters of the network module to be fine-tuned according to the following objective function using the fine-tuning sample set to obtain the adjusted model parameters θ FT : Among them, x represents the part of the samples in the fine-tuning sample set that belongs to the original samples, δ represents the part of the samples in the fine-tuning sample set that belongs to the added disturbance, Δ is the value range of δ, and y is the label of the original sample corresponding to the sample in the fine-tuning sample set. is the loss function; represents the model obtained by freezing the parameters of other network modules except the k-th layer network module and fine-tuning the k-th layer network module. represents the sample space, θ FT The parameters of the model are obtained by freezing the parameters of the network modules except the k-th layer network module based on the objective function and fine-tuning the k-th layer network module, where k is the serial number of the fine-tuned network module.

8. A plug-and-play selective module robust fine-tuning method as claimed in claim 7, characterized in that: Step S4 further includes: Using a set of mixing coefficients η, θ is adjusted according to the following formula AT and θ FT Combine and obtain a set of parameters to be evaluated: Among them, η p is the pth mixing coefficient η and η p ∈[0,1],θ AT are the parameters of the initial model, θ FT are the parameters of the model obtained after fine-tuning the k-th layer network module, where k is the serial number of the fine-tuned network module; A robustness evaluation is performed on the model corresponding to each parameter to be evaluated, and the parameter to be evaluated with the highest adversarial accuracy and the best standard accuracy is selected as the final model parameter. The robustly enhanced image classification model is constructed with the final model parameters.

9. A plug-and-play selective module robust fine-tuning method according to any one of claims 1 to 8, characterized in that: The adversarial samples in the adversarial sample set and the fine-tuning sample set are both generated using the PGD-10 model, and the attack strength of the adversarial samples included in the fine-tuning sample set is higher than the attack strength of the adversarial samples included in the adversarial sample set.

Images