A plug-and-play selective module adversarial robust fine-tuning method

Abstract

The present invention relates to the field of adversarial attack and defense in computer vision, and specifically relates to a plug-and-play selective module adversarial robust fine-tuning method. By introducing a generalization sensitivity index and a robust criticality index of a module, and based on the robust criticality index and the generalization sensitivity index of each layer of network modules contained in an initial model, with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index, a network module to be fine-tuned is determined; finally, by freezing all other module parameters and using a fine-tuning sample set, the parameters of the fine-tuned network module are fine-tuned to obtain a robustly enhanced image classification model. The method can comprehensively consider the relationship between model robustness and generalization, accurately identify the most critical module for fine-tuning, and significantly enhance the model robustness while maintaining or even slightly improving the generalization. The method has plug-and-play characteristics, can be deployed as a universal component in various adversarial trainings, and can perform secondary robust enhancement on the robust model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision attack and defense, and in particular to a plug-and-play selective module adversarial robust fine-tuning method. Background Art

[0002] Deep neural networks, due to their powerful performance, have achieved remarkable success in a range of real-world tasks. However, deep neural networks are vulnerable to adversarial examples, which pose a significant security risk to critical applications. Adversarial examples add tiny, imperceptible perturbations to input images, misleading the model into making incorrect classifications or detections. Such attacks not only affect the model's recognition accuracy but can also pose a serious threat to the security of systems such as remote sensing and facial recognition. Furthermore, the concealed nature of adversarial attacks makes them even more of a threat in sensitive areas. For example, in military reconnaissance missions, the enemy may use adversarial attacks to interfere with automatic target recognition systems, preventing them from correctly identifying ground targets and thus affecting decision-making. Therefore, there is an urgent need to improve robustness against such attacks.

[0003] Currently, defense methods against adversarial attacks mainly include sample detection, data enhancement, and adversarial training. Among them, adversarial training, also known as Adversarial Training (AT), has been proven to be one of the most effective strategies in recent years. It introduces adversarial samples during the training process, allowing the model to actively learn and adapt to the attacker's data distribution, thereby improving the model's resistance to adversarial perturbations. For a l-layer deep neural network f with parameter θ, θ , and from the standard distribution Standard dataset extracted from Standard adversarial training can be formulated as a min-max optimization problem:

[0004]

[0005] in, represents the robust loss under adversarial samples, represents the objective loss function, and δ represents the adversarial perturbation constrained by the bounded set Δ.

[0006] However, adversarial training often leads to a decline in model performance on benign data distributions, resulting in the so-called "robustness-generalization trade-off." This means that while the model performs well against adversarial examples, its accuracy on clean examples—original examples without added perturbations—is degraded. This is unacceptable for tasks requiring high precision, such as remote sensing image processing and autonomous driving. Furthermore, this issue has been generally considered an inherent flaw in deep neural networks in previous research, such as:

[0007] First, Tsipras et al. found that there may be an inherent contradiction between the adversarial robustness goal and the standard generalization goal. They showed that even in a fairly simple and natural environment, this trade-off between the standard accuracy of the model and its robustness to adversarial perturbations exists. These findings are also confirmed to be observed in more complex environments.

[0008] Second, Schmidt et al. conducted a large number of experiments and showed that adversarial samples, due to the inherent sample complexity, can lead to a separation between the standard accuracy and robust accuracy of the model;

[0009] Javanmard et al. provide a precise and comprehensive understanding of the role of adversarial training in linear regression with Gaussian features, characterizing the fundamental trade-off between the accuracy that any algorithm can achieve, regardless of computational power or training data size. They also provide a precise characterization of the standard accuracy and robust accuracy trade-offs achieved by the popular minimax adversarial training method in a high-dimensional setting.

[0010] Fourth, Raghunathan et al. accurately described the impact of enhancement on the standard error of linear regression when the standard error and robust error of the optimal linear predictor are zero. They found that even if the enhanced disturbance is the same as the noise-free observation value of the optimal linear predictor, the standard error will increase and the trade-off problem still exists.

[0011] In recent years, a growing body of research has focused on addressing this trade-off, with methods such as TRADES, SCORE, MART, adversarial training guided by learnable boundaries, and stochastic adversarial training via Taylor expansion proposed. However, these methods, while focused on optimizing non-robust models to robust ones, typically treat the model as a whole during training, failing to fully consider the unique roles and contributions of each layer. This holistic approach limits the model's potential and leads to underutilization of its capacity.

[0012] Further research, by deleting, rearranging or reinitializing certain residual blocks of the residual network after standard adversarial training, found that there are differences in the network modules of each layer of the deep neural network during the training process. Therefore, some scholars proposed the latest RiFT (Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning, ICCV 2023) technology, whose purpose is to improve the generalization of the "robust" model obtained by adversarial training to alleviate the "robust-generalization trade-off" problem. Specifically, RiFT introduces the Module Robust Criticality (MRC) indicator, which is an indicator that measures the importance of a given network module to the robustness of the model under the worst-case weight perturbation. It identifies the network module with the lowest MRC value as a non-robust critical module and fine-tunes its weight to obtain the fine-tuned weight.

[0013] However, as pointed out in the literature of RiFT technology, only by fine-tuning on non-robust key modules can adversarial robustness be preserved; compared with fine-tuning on all layers, fine-tuning on robust key modules shows the worst trade-off between generalization and robustness. Figure 1 As you can see, if you fine-tune the robust key modules, improving the model robustness will reduce generalization, and improving generalization will reduce robustness, which deepens the robustness-generalization trade-off problem. Therefore, the essence of RiFT technology is to use MRC to characterize robust redundancy, maintaining robustness while improving generalization.

[0014] Therefore, although the RiFT technology proposes the MRC indicator, the MRC indicator characterizes the sharpness of the robustness loss regarding parameter fine-tuning, and is mainly used to avoid the impact of fine-tuning on robustness. It does not provide guidance for performing robust enhancement fine-tuning while alleviating the robustness-generalization trade-off problem. Summary of the Invention

[0015] The technical problem to be solved by the present invention is to propose a plug-and-play selective module adversarial robust fine-tuning method, which can comprehensively consider the relationship between model robustness and generalization, accurately identify the most critical modules for fine-tuning, and greatly enhance the robustness of the model while maintaining or even slightly improving the generalization.

[0016] The technical solution adopted by the present invention to solve the above technical problems is:

[0017] A plug-and-play selective module adversarial robust fine-tuning method comprises the following steps:

[0018] S1. Input the pre-trained image classification model as the initial model;

[0019] S2. For each layer of network modules in the initial model, calculate the robustness criticality index and generalization sensitivity index of each layer of network modules according to the following steps:

[0020] S21. Obtain the disturbance model:

[0021] Freeze the parameters of other network modules included in the initial model. The other network modules in this step are network modules other than the current computing network module included in the initial model. For the current computing network module, use the adversarial sample set to maximize the difference in robustness of the model before and after the perturbation, add perturbations to the parameters of the current computing network module, and obtain a first perturbation model corresponding to the current computing network module. For the current computing network module, use the clean sample set to maximize the difference in generalization of the model before and after the perturbation, and add perturbations to the parameters of the current computing network module, and obtain a second perturbation model corresponding to the current computing network module.

[0022] The clean sample set is a set of clean samples, the adversarial sample set is a set of adversarial samples, the clean samples are original samples without adding disturbances, and the adversarial samples are samples obtained after adding adversarial disturbances to the original samples;

[0023] S22. Index calculation:

[0024] Using samples from the adversarial sample set as input, the initial model and the first perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the first loss of the current computing network module, and the classification loss of the first perturbation model corresponding to the current computing network module is used as the second loss of the current computing network module; based on the change in the first loss and the second loss of the current computing network module, the robust criticality index of the current computing network module is calculated;

[0025] Using samples from the test sample set as input, the initial model and the second perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the third loss of the current computing network module, and the classification loss of the second perturbation model corresponding to the current computing network module is used as the fourth loss of the current computing network module; based on the changes in the third loss and the fourth loss of the current computing network module, the generalization sensitivity index of the current computing network module is calculated; the samples of the test sample set are original samples that have not been sampled in the model training stage;

[0026] S3. Based on the robust criticality index and generalization sensitivity index of each layer of network modules contained in the initial model, determine the network modules to be fine-tuned with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index;

[0027] S4. Freeze the parameters of other network modules included in the initial model, and only fine-tune the parameters of the network modules that are fine-tuned using the fine-tuning sample set to obtain a robustly enhanced image classification model; the other network modules in this step are network modules other than the fine-tuned network modules included in the initial model; the samples in the fine-tuning sample set are adversarial samples.

[0028] Furthermore, in step S21, for the network module currently being calculated, using samples from the corresponding sample set, with the goal of maximizing the difference in optimization objectives of the model before and after adding the perturbation, perturbations are added to the parameters of the current computing network module to obtain a perturbation model corresponding to the current computing network module, wherein the optimization objective is robustness when constructing the first perturbation model and generalization when constructing the second perturbation model; specifically, the steps include:

[0029] A1. Use the initial model as the current model;

[0030] A2. Input samples from the corresponding sample set and use the current model to obtain the predicted classification of each input sample. Calculate the classification loss of the current model using the label classification and predicted classification of each input sample.

[0031] A3. Using the classification loss of the current model and the gradient ascent algorithm, perturb the parameters of the current computational network module of the current model with the goal of maximizing the difference in the optimization objectives of the model before and after adding the perturbation.

[0032] A4. Determine whether the preset iteration rounds have been reached. If so, add the perturbed model in step A3 as the perturbation model corresponding to the current computing network module; otherwise, add the perturbed model in step A3 as the current model and return to step A2.

[0033] Furthermore, perturbations are added to the parameters of the current computational network module of the current model, including:

[0034] A31. Calculate the added disturbance Δθ according to the following formula: t :

[0035]

[0036] Among them, t represents the round of the current iteration, is the classification loss of the jth input sample classified by the current model, f t Represents the current model, x j is the jth sample of the input, y j is the label of the jth input sample, J is the number of input samples; represents the gradient of loss; α is the step size of adding perturbations;

[0037] A32. Calculate the disturbance change Δd according to the following formula;

[0038] Δd=(θ t-1 +Δθ t )-θ0

[0039] Among them, θ0 is the parameter of the current computing network module in the initial model, θ t-1 is the parameter obtained by the current computing network module of the current model in the t-1th round;

[0040] A33, take ∥Δd∥2<∈∥θ0∥2 as the perturbation constraint;

[0041] If ∥Δd∥2<∈∥θ0∥2, then add perturbations to the parameters of the current computational network module of the current model according to the following formula:

[0042] θ t ←θ t-1 +Δθ t

[0043] If ∥Δd∥2≥∈∥θ0∥2, then add perturbations to the parameters of the current computational network module of the current model according to the following formula:

[0044]

[0045] Among them, ∥·∥2 is the two-norm, ∈ is the perturbation scaling factor, θ t The parameters of the current computing network module of the current model obtained through the current iterative update.

[0046] Furthermore, the preset iteration rounds T is 10 times, and the disturbance scaling factor ∈ is 0.1.

[0047] Furthermore, in step S22, the robust criticality index MRC is calculated according to the following formula:

[0048]

[0049] In step S22, the generalization sensitivity index MGSI is calculated according to the following formula:

[0050]

[0051] in, represents the first perturbation model, represents the second perturbation model, Represents the initial model; x is the input sample, y is the label of the input sample, is the loss function, k1 and k2 are sample numbers; K1 is the number of samples in the input adversarial sample set, and K2 is the number of samples in the input test sample set.

[0052] Furthermore, in step S3, based on the robust criticality index and generalization sensitivity index of each layer of network modules included in the initial model, with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index, the network modules to be fine-tuned are determined, including:

[0053] S31, normalizing the robust criticality index of each layer of network modules included in the initial model; normalizing the generalization sensitivity index of each layer of network modules included in the initial model;

[0054] S32. Determine the network module for fine-tuning according to the following formula:

[0055]

[0056] in, represents the robust criticality index obtained by normalization of the i-th layer network module, represents the generalization sensitivity index obtained by normalization of the i-th layer network module, and k is the sequence number of the network module for fine-tuning.

[0057] Furthermore, in step S4, the parameters of other network modules included in the initial model are frozen, and only the parameters of the network module to be fine-tuned are fine-tuned using the fine-tuning sample set according to the following objective function to obtain the adjusted model parameters θ FT :

[0058]

[0059] Among them, x represents the part of the sample in the fine-tuning sample set that belongs to the original sample, δ represents the part of the sample in the fine-tuning sample set that belongs to the added perturbation, Δ is the value range of δ, and y is the label of the original sample corresponding to the sample in the fine-tuning sample set. is the loss function; represents the model obtained by freezing the parameters of the network modules except the k-th layer network module and fine-tuning the k-th layer network module. represents the sample space, θ FT The parameters of the model are obtained by freezing the parameters of the network modules except the k-th layer network module based on the objective function and fine-tuning the k-th layer network module, where k is the serial number of the fine-tuned network module.

[0060] Furthermore, step S4 further includes:

[0061] Using a set of mixing coefficients η, θ is calculated according to the following formula AT and θ FT Combine to obtain a set of parameters to be evaluated:

[0062] θ η p=(1-ηp )·θ AT +η p ·θ FT

[0063] Among them, η p is the pth mixing coefficient η and η p ∈[0,1],θ AT are the parameters of the initial model, θ FT are the parameters of the model obtained after fine-tuning the k-th layer network module, where k is the serial number of the network module for fine-tuning;

[0064] A robustness evaluation is performed on the model corresponding to each parameter to be evaluated, and the parameter to be evaluated with the highest adversarial accuracy and the best standard accuracy is selected as the final model parameter. The robustly enhanced image classification model is constructed with the final model parameters.

[0065] Furthermore, the adversarial samples in the adversarial sample set and the fine-tuning sample set are both generated using the PGD-10 model, and the attack strength of the adversarial samples contained in the fine-tuning sample set is higher than the attack strength of the adversarial samples contained in the adversarial sample set.

[0066] The beneficial effects of the present invention are:

[0067] The method of the present invention introduces the robust criticality index (Module Robust Criticality, MRC) of the RiFT technology. However, unlike the RiFT technology which enhances generalization without losing robustness, the method of the present invention is to enhance robustness without losing generalization. That is, the present invention needs to adjust the modules with MRC as high as possible to avoid the passivation of the adjusted modules to parameter perturbations and ensure the enhancement of robustness. Therefore, the substantive problem that the method of the present invention needs to solve is: how to use MRC to determine the adjustment module while overcoming the contradiction pointed out by the RiFT technology that "only fine-tuning on non-robust key modules can retain adversarial robustness; compared with fine-tuning all layers, fine-tuning the robustness key modules shows the worst trade-off between generalization and robustness."

[0068] The inventors believe that the model's robustness to adversarial samples mainly comes from the tiny adversarial perturbations during the fitting adversarial training process. Since such perturbations do not exist in clean samples, using clean samples to fine-tune modules with high MRC values ​​will greatly reduce the robustness of the model. On the contrary, adversarial samples are essentially clean samples with specific noise, which retains a lot of original clean information. Therefore, after fine-tuning using adversarial samples, the model generalization will not be easily reduced.

[0069] Therefore, based on MRC, the inventors defined the Module Generalization Sensitivity Index (MGSI), which quantifies the module's sensitivity to generalization under specific weight perturbations. Based on the MGSI and MRC indicators, the inventors proposed the present method, a plug-and-play Selective Module Adversarial Robust fine-tuning (SMART) method, which comprehensively considers the relationship between model robustness and generalization, accurately identifies the most critical modules for fine-tuning, and significantly enhances model robustness while maintaining or even slightly improving generalization.

[0070] The inventors have also developed calculation steps for MGSI that are basically consistent with MRC, making it easy to integrate the calculation processes of the two while retaining the plug-and-play feature. This allows the method of the present invention to be deployed as a universal component on various advanced adversarial training methods, and has high feasibility and effectiveness for different model architectures, data scales, different attack intensities, and attack strategies.

[0071] Given its plug-and-play nature, it's important to note that in the existing robust fine-tuning field, most techniques rely on full fine-tuning, specifically fine-tuning all network modules at all layers, aiming to optimize a non-robust model into a robust one. However, the method of this invention can be combined with other methods to achieve a model upgrade from non-robust to robust and then even more robust. This method also introduces the concept of secondary revision of robust models, further strengthening the model's ability to resist attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 The figure is a flow chart of a plug-and-play selective module robust fine-tuning method of the present invention. DETAILED DESCRIPTION

[0073] This paper aims to propose a plug-and-play selective module adversarial robust fine-tuning method, which integrates MGSI and MRC indicators, comprehensively considers the relationship between model robustness and generalization from a fine-grained module-level perspective, accurately identifies the most critical modules for fine-tuning, and significantly enhances model robustness while maintaining or even slightly improving generalization. Figure 1 As shown, the following steps are included:

[0074] S1. Input the pre-trained image classification model as the initial model;

[0075] S2. For each layer of network modules in the initial model, calculate the robustness criticality index and generalization sensitivity index of each layer of network modules according to the following steps:

[0076] S21. Obtain the disturbance model:

[0077] Freeze the parameters of other network modules included in the initial model. The other network modules in this step are network modules other than the current computing network module included in the initial model. For the current computing network module, use the adversarial sample set to maximize the difference in robustness of the model before and after the perturbation, add perturbations to the parameters of the current computing network module, and obtain a first perturbation model corresponding to the current computing network module. For the current computing network module, use the clean sample set to maximize the difference in generalization of the model before and after the perturbation, and add perturbations to the parameters of the current computing network module, and obtain a second perturbation model corresponding to the current computing network module.

[0078] The clean sample set is a set of clean samples, the adversarial sample set is a set of adversarial samples, the clean samples are original samples without adding disturbances, and the adversarial samples are samples obtained after adding adversarial disturbances to the original samples;

[0079] S22. Index calculation:

[0080] Using samples from the adversarial sample set as input, the initial model and the first perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the first loss of the current computing network module, and the classification loss of the first perturbation model corresponding to the current computing network module is used as the second loss of the current computing network module; based on the change in the first loss and the second loss of the current computing network module, the robust criticality index of the current computing network module is calculated;

[0081] Using samples from the test sample set as input, the initial model and the second perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the third loss of the current computing network module, and the classification loss of the second perturbation model corresponding to the current computing network module is used as the fourth loss of the current computing network module; based on the changes in the third loss and the fourth loss of the current computing network module, the generalization sensitivity index of the current computing network module is calculated; the samples of the test sample set are original samples that have not been sampled in the model training stage;

[0082] S3. Based on the robust criticality index and generalization sensitivity index of each layer of network modules contained in the initial model, determine the network modules to be fine-tuned with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index;

[0083] S4. Freeze the parameters of other network modules included in the initial model, and only fine-tune the parameters of the network modules that are fine-tuned using the fine-tuning sample set to obtain a robustly enhanced image classification model; the other network modules in this step are network modules other than the fine-tuned network modules included in the initial model; the samples in the fine-tuning sample set are adversarial samples.

[0084] The solution of the present invention is further described below with reference to the embodiments.

[0085] Example

[0086] A plug-and-play selective module adversarial robust fine-tuning method comprises the following steps:

[0087] S1. Input model

[0088] The purpose of this step is to obtain a model for fine-tuning, that is, to input the pre-trained image classification model as the initial model.

[0089] The implementation of the present invention is to fine-tune the parameters of a certain layer of the network module of the model based on the robust loss of adversarial samples. Therefore, it does not depend on the category of the model. The image classification model can be any existing model, such as: ResNet18, ResNet34, ResNet50, WideResNet34-10.

[0090] The term "pre-trained" here means that the model has already been trained to perform the target task. However, it should be noted that this training can be conventional training, i.e., a "non-robust" model, or conventional training followed by adversarial training, i.e., a "robust" model. Preferably, the pre-trained image classification model is a model obtained after conventional training followed by adversarial training.

[0091] S2. Index calculation

[0092] The purpose of this step is to calculate the robust criticality index (MRC) and generalization sensitivity index (MGSI) of each layer of the network module in the input model in step S1. Therefore, for each layer of the network module in the initial model, the robust criticality index and generalization sensitivity index of each layer of the network module are calculated according to the following steps:

[0093] S21. Obtaining the perturbation model

[0094] Freeze the parameters of other network modules included in the initial model. The other network modules in this step are network modules other than the current computing network module included in the initial model.

[0095] For the current computing network module, using the adversarial sample set, with the goal of maximizing the difference in robustness of the model before and after adding the perturbation, add perturbations to the parameters of the current computing network module to obtain the first perturbation model corresponding to the current computing network module;

[0096] For the current computing network module, using the clean sample set, with the goal of maximizing the generalization difference of the model before and after adding the perturbation, perturbations are added to the parameters of the current computing network module to obtain the second perturbation model corresponding to the current computing network module.

[0097] The above goal is to maximize the difference in the optimization objectives of the model before and after adding the disturbance, which can be formulated as:

[0098]

[0099] The optimization goal is robustness when constructing the first perturbation model and generalization when constructing the second perturbation model; θ represents a deep neural network model with parameters θ, is a set of weight perturbations with scale invariance, Δθ represents the perturbation relative to θ i Parameter perturbation, θ i is the parameter of the i-th layer network module of the deep neural network model, that is, the current computing network module, represents the robust loss, is the corresponding sample set.

[0100] The clean sample set is a collection of clean samples, which are original samples without any perturbations. Original samples can be general datasets such as CIFAR10, CIFAR100, TinyImageNet, SVHN, and ImageNet, or aerospace remote sensing image datasets such as Aerial Landscapes.

[0101] The adversarial sample set is a collection of adversarial samples, and the adversarial samples are samples obtained by adding adversarial perturbations to the original samples. The adversarial samples can be generated using existing methods, including: PGD-1 ∞ , PGD-l2, FGSM, CW-20, and AutoAttack, etc. In subsequent verification experiments, the adversarial example sets used to calculate the MRC / MGSI indicators and subsequent fine-tuning were all generated by PGD-10.

[0102] Based on the objective function, a perturbation model is established. Existing methods can be used, preferably, a gradient ascent method. Specifically, in this step, for the network module currently being calculated, samples from the corresponding sample set are used to maximize the difference in the optimization target of the model before and after adding the perturbation. Perturbations are added to the parameters of the current computing network module to obtain the perturbation model corresponding to the current computing network module. Specifically, the perturbation model includes:

[0103] A1. Use the initial model as the current model;

[0104] A2. Input samples from the corresponding sample set and use the current model to obtain the predicted classification of each input sample. Calculate the classification loss of the current model using the label classification and predicted classification of each input sample.

[0105] A3. Using the classification loss of the current model and based on the gradient ascent algorithm, perturbations are added to the parameters of the current computational network module of the current model with the goal of maximizing the difference in optimization objectives before and after the perturbation. This includes:

[0106] A31. Calculate the added disturbance Δθ according to the following formula: t :

[0107]

[0108] Among them, t represents the round of the current iteration, is the classification loss of the jth input sample classified by the current model, f t Represents the current model, x j is the jth sample of the input, y j is the label of the jth input sample, J is the number of input samples; represents the gradient of loss; α is the step size of adding perturbations;

[0109] A32. Calculate the disturbance change Δd according to the following formula;

[0110] Δd=(θ t-1 +Δθ t )-θ0

[0111] Among them, θ0 is the parameter of the current computing network module in the initial model, θ t-1 is the parameter obtained by the current computing network module of the current model in the t-1th round;

[0112] A33, take ∥Δd∥2<∈∥θ0∥2 as the perturbation constraint;

[0113] If ∥Δd∥2<∈∥θ0∥2, then add perturbations to the parameters of the current computational network module of the current model according to the following formula:

[0114] θt ←θ t-1 +Δθ t

[0115] If ∥Δd∥2≥∈∥θ0∥2, then add perturbations to the parameters of the current computational network module of the current model according to the following formula:

[0116]

[0117] Among them, ∥·∥2 is the two-norm, ∈ is the perturbation scaling factor, θ t The parameters of the current computing network module of the current model obtained through the current iteration update;

[0118] A4. Determine whether the preset iteration rounds have been reached. If so, add the perturbed model in step A3 as the perturbation model corresponding to the current computing network module; otherwise, add the perturbed model in step A3 as the current model and return to step A2.

[0119] According to different original data and input models, the preset iteration rounds T is 10 times, the perturbation scaling factor ∈ is 0.1, the number of samples J input in each iteration is set to 128, and a lower learning rate is set such as α = 0.001. By using a lower learning rate and fewer rounds, excessive fine-tuning of the model can be avoided, thereby improving the efficiency and effect of fine-tuning.

[0120] The iterative process of the first perturbation model and the second perturbation model is the same, the only difference is the input samples.

[0121] S22. Index calculation

[0122] Calculation of MRC: Use samples from the adversarial sample set as input, input the initial model and the first perturbation model corresponding to the current computing network module respectively, and use the classification loss of the initial model as the first loss of the current computing network module, and the classification loss of the first perturbation model corresponding to the current computing network module as the second loss of the current computing network module; based on the change in the first loss and the second loss of the current computing network module, calculate and obtain the robust criticality index of the current computing network module.

[0123] The above calculation can be expressed in the form of:

[0124]

[0125] Calculation of MGSI: Use samples from the test sample set as input, input the initial model and the second perturbation model corresponding to the current computing network module respectively, and use the classification loss of the initial model as the third loss of the current computing network module, and the classification loss of the second perturbation model corresponding to the current computing network module as the fourth loss of the current computing network module; based on the changes in the third and fourth losses of the current computing network module, calculate the generalization sensitivity index of the current computing network module.

[0126] The above calculation can be expressed in the form of:

[0127]

[0128] in, represents the first perturbation model, represents the second perturbation model, Represents the initial model; x is the input sample, y is the label of the input sample, is the loss function, k1 and k2 are sample numbers; K1 is the number of samples in the input adversarial sample set, and K2 is the number of samples in the input test sample set.

[0129] The samples of the test sample set are original samples that have not been sampled in the model training stage. The training sample set and the test sample set are usually obtained by dividing the original data set in proportion.

[0130] S3. Determine module

[0131] This step determines the network module to be adjusted based on the indicators calculated in step S2.

[0132] Fine-tuning modules are selected based on two metrics: MRC and MGSI. An ideal module should have both high MRC and low MGSI values, indicating that it contributes significantly to model robustness while having minimal impact on generalization performance. Specifically, the network modules for fine-tuning are determined based on the robustness criticality index and generalization sensitivity index of each network module in the initial model, with the goal of maximizing the robustness criticality index and minimizing the generalization sensitivity index.

[0133] However, it should be noted that MRC and MGSI are obtained from different data distributions. Therefore, the steps include:

[0134] S31. Normalize the robustness criticality index of each layer of network modules included in the initial model; and normalize the generalization sensitivity index of each layer of network modules included in the initial model. In this embodiment, Min-Max normalization is adopted.

[0135] S32. Determine the network module for fine-tuning according to the following formula:

[0136]

[0137] in, represents the robust criticality index obtained by normalization of the i-th layer network module, represents the generalization sensitivity index obtained by normalization of the i-th layer network module, and k is the sequence number of the network module for fine-tuning.

[0138] In addition to the above methods, the inventors also explored It is found that as long as the goal is to maximize the robust criticality index and minimize the generalization sensitivity index, it has little effect on the final determined regulation module.

[0139] S4. Parameter adjustment

[0140] In this step, the parameters of other network modules included in the initial model are frozen, and only the parameters of the network modules that are fine-tuned are fine-tuned using the fine-tuning sample set to obtain a robustly enhanced image classification model; the other network modules in this step are network modules other than the network modules that are fine-tuned and included in the initial model.

[0141] The samples in the fine-tuning sample set are adversarial samples. In this embodiment, the adversarial samples in both the adversarial sample set and the fine-tuning sample set are generated using the PGD-10 model. Because fine-tuning a single module at a lower learning rate requires a stronger attack to drive improvements in the overall model performance, the attack strength of the adversarial samples in the fine-tuning sample set is higher than that of the adversarial samples in the adversarial sample set.

[0142] Parameter fine-tuning can be performed using existing methods. In this embodiment, the method includes:

[0143] First, freeze the parameters of other network modules included in the initial model, and only fine-tune the parameters of the network module to be fine-tuned according to the following objective function using the fine-tuning sample set to obtain the adjusted model parameters θ FT :

[0144]

[0145] Among them, x represents the part of the sample in the fine-tuning sample set that belongs to the original sample, δ represents the part of the sample in the fine-tuning sample set that belongs to the added perturbation, Δ is the value range of δ, and y is the label of the original sample corresponding to the sample in the fine-tuning sample set. is the loss function; represents the model obtained by freezing the parameters of the network modules except the k-th layer network module and fine-tuning the k-th layer network module. represents the sample space, θ FTThe parameters of the model are obtained by freezing the parameters of the network modules except the k-th layer network module based on the objective function and fine-tuning the k-th layer network module, where k is the serial number of the fine-tuned network module.

[0146] Max refers to finding the best δ in the convex set Δ, so that when the clean original sample is added with δ and input into the model, the model prediction error is the highest; min indicates that under the strongest adversarial perturbation, the optimal model parameters are found so that the model test error expectation is minimized.

[0147] Then, a set of mixing coefficients η is used to adjust θ according to the following formula: AT and θ FT Combine to obtain a set of parameters to be evaluated:

[0148]

[0149] Among them, η p is the pth mixing coefficient η and η p ∈[0,1],θ AT are the parameters of the initial model, θ FT is the parameter of the model obtained after fine-tuning the k-th layer network module, and k is the serial number of the network module to which the fine-tuning is performed. In this embodiment, the 10 mixing coefficients η are respectively 0.1, 0.2, ..., 1.0.

[0150] The robustness of the model corresponding to each parameter to be evaluated is then evaluated. The parameters with the highest adversarial accuracy and the best standard accuracy are selected as the final model parameters. A robust image classification model is constructed using these final model parameters. The model evaluation process is based on different attack strategies, such as PGD-10, AutoAttack, CW-20, and FGSM. Different attack strengths and attack norms are used in the PGD attack. The robustness and generalization performance of the model are comprehensively and rigorously tested to ensure its stability and effectiveness under adversarial attacks.

[0151] Experimental verification

[0152] In order to verify the effectiveness of the present invention, comprehensive experiments were conducted based on the scheme of the embodiment using different image classification datasets: CIFAR10, CIFAR100, TinyImageNet, and popular CNN-based model architectures: ResNet18, ResNet34, and WideResNet34-10.

[0153] For the evaluation setting, the method evaluates the generalization and robustness of the fine-tuned model by measuring the prediction accuracy on the standard and adversarial test sets, respectively. The adversarial datasets are generated using common attacks, including PGD- ∞, PGD-12, FGSM, CW-20 and AutoAttack. ∞ In the attack, three different attack strengths ε were used: 8 / 255, 10 / 255, and 12 / 255.

[0154] Table 1 shows the evaluation results of different datasets and model backbones under four different adversarial attack types; Table 2 shows the evaluation results of different datasets and model backbones under attacks of different strengths and norms. In the table, AT represents the model obtained after adversarial attack training, that is, the model before fine-tuning using the method of the present invention; SMART represents the AT model after fine-tuning using the method of the present invention; Clean represents clean test samples, used to evaluate generalization performance; PGD, FGSM, CW-20, and AA (AutoAttack) represent different attack samples, used to evaluate robustness; △ represents the percentage change in performance before and after fine-tuning using the method of the present invention; l2, 8 / 255, 10 / 255, and 12 / 255 in the table represent the attack strength and norm.

[0155] Table 1. Evaluation results of different datasets and model backbones under different adversarial attack types (unit %)

[0156]

[0157] As shown in Table 1, for different types of attacks, this method significantly improves the robustness of all models by approximately 0.5% to 2.5%, and the generalization of all models also shows varying degrees of enhancement.

[0158] Table 2. Evaluation results of different datasets and model backbones under different attack intensities and norms (unit %)

[0159]

[0160] As shown in Table 2, for attacks of different strengths and norms, our method significantly improves the robustness of all models by about 1% to 2.5%. Except for a few models that show a slight loss in generalization, most models show varying degrees of enhancement.

[0161] Table 3 shows the final fine-tuned network modules of each model in the experiments in Tables 1 and 2. Table 4 shows the hyperparameter settings in the experiments in Tables 1 and 2, in the format of “PGD-10 maximum perturbation limit / PGD-10 perturbation step size / fine-tuning rounds / learning rate”.

[0162] Table 3. Final fine-tuned network modules of each model

[0163] CIFAR10 CIFAR100 Tiny-ImageNet ResNet18 layer4.1.conv2 layer4.1.conv2 layer4.0.conv2 ResNet34 layer4.0.conv2 layer4.1.conv2 layer4.0.conv1 WRN34-10 block3.layer.1.conv1 block3.layer.3.conv1 block3.layer.3.conv1

[0164] Table 4 Hyperparameter settings for different models

[0165] CIFAR10 CIFAR100 Tiny-ImageNet ResNet18 12 / 3 / 20 / 0.001 16 / 4 / 20 / 0.001 12 / 3 / 20 / 0.001 ResNet34 12 / 3 / 10 / 0.001 16 / 4 / 20 / 0.005 12 / 3 / 20 / 0.005 WRN34-10 12 / 3 / 20 / 0.001 16 / 4 / 20 / 0.001 16 / 4 / 20 / 0.001

[0166] Finally, it should be noted that the above embodiments are merely preferred implementations and are not intended to limit the present invention. It should be noted that those skilled in the art will be able to make modifications, equivalent substitutions, and improvements without departing from the spirit and scope of the present invention and the claims, all of which should be included within the scope of protection of the present invention.

Claims

1. A plug-and-play selective module adversarial robust fine-tuning method, characterized in that: The steps include: S1. Input the pre-trained image classification model as the initial model; S2. For each layer of network modules in the initial model, calculate the robustness criticality index and generalization sensitivity index of each layer of network modules according to the following steps: S21. Obtain the disturbance model: Freeze the parameters of other network modules included in the initial model. The other network modules in this step are network modules other than the current computing network module included in the initial model. For the current computing network module, use the adversarial sample set to maximize the difference in robustness of the model before and after the perturbation, add perturbations to the parameters of the current computing network module, and obtain a first perturbation model corresponding to the current computing network module. For the current computing network module, use the clean sample set to maximize the difference in generalization of the model before and after the perturbation, and add perturbations to the parameters of the current computing network module, and obtain a second perturbation model corresponding to the current computing network module. The clean sample set is a set of clean samples, the adversarial sample set is a set of adversarial samples, the clean samples are original samples without adding disturbances, and the adversarial samples are samples obtained after adding adversarial disturbances to the original samples; S22. Index calculation: Using samples from the adversarial sample set as input, the initial model and the first perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the first loss of the current computing network module, and the classification loss of the first perturbation model corresponding to the current computing network module is used as the second loss of the current computing network module; based on the change in the first loss and the second loss of the current computing network module, the robust criticality index of the current computing network module is calculated; Using samples from the test sample set as input, the initial model and the second perturbation model corresponding to the current computing network module are input respectively, and the classification loss of the initial model is used as the third loss of the current computing network module, and the classification loss of the second perturbation model corresponding to the current computing network module is used as the fourth loss of the current computing network module; based on the changes in the third loss and the fourth loss of the current computing network module, the generalization sensitivity index of the current computing network module is calculated; the samples of the test sample set are original samples that have not been sampled in the model training stage; S3. Based on the robust criticality index and generalization sensitivity index of each layer of network modules contained in the initial model, determine the network modules to be fine-tuned with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index; S4. Freeze the parameters of other network modules included in the initial model, and only fine-tune the parameters of the network modules that are fine-tuned using the fine-tuning sample set to obtain a robustly enhanced image classification model; the other network modules in this step are network modules other than the fine-tuned network modules included in the initial model; the samples in the fine-tuning sample set are adversarial samples.

2. The plug-and-play selective module robust fine-tuning method according to claim 1, wherein: In step S21, for the network module currently being calculated, using samples from the corresponding sample set, with the goal of maximizing the difference in optimization objectives of the model before and after adding the perturbation, perturbations are added to the parameters of the current computing network module to obtain a perturbation model corresponding to the current computing network module, wherein the optimization objective is robustness when constructing the first perturbation model and generalization when constructing the second perturbation model; specifically, the steps include: A1. Use the initial model as the current model; A2. Input samples from the corresponding sample set and use the current model to obtain the predicted classification of each input sample. Calculate the classification loss of the current model using the label classification and predicted classification of each input sample. A3. Using the classification loss of the current model and the gradient ascent algorithm, perturb the parameters of the current computational network module of the current model with the goal of maximizing the difference in the optimization objectives of the model before and after adding the perturbation. A4. Determine whether the preset iteration rounds have been reached. If so, add the perturbed model in step A3 as the perturbation model corresponding to the current computing network module; otherwise, add the perturbed model in step A3 as the current model and return to step A2.

3. The plug-and-play selective module robust fine-tuning method according to claim 2, wherein: Add perturbations to the parameters of the current computational network module of the current model, including: A31. Calculate the added disturbance Δθ according to the following formula: t : Among them, t represents the round of the current iteration, is the classification loss of the jth input sample classified by the current model, f t Represents the current model, x j is the jth sample of the input, y j is the label of the jth input sample, J is the number of input samples; represents the gradient of loss; α is the step size of adding perturbations; A32. Calculate the disturbance change Δd according to the following formula; Δd=(θ t-1 +Δθ t )-θ0 Among them, θ0 is the parameter of the current computing network module in the initial model, θ t-1 is the parameter obtained by the current computing network module of the current model in the t-1th round; A33, use ||Δd||2<∈||θ0||2 as the perturbation constraint; If ||Δd||2<∈||θ0||2, then add perturbations to the parameters of the current computational network module of the current model according to the following formula: i t ←θ t-1 +Δθ t If ||Δd||2≥∈||θ0||2, then add perturbations to the parameters of the current computational network module of the current model according to the following formula: Among them, ||·||2 is the two-norm, ∈ is the perturbation scaling factor, θ t The parameters of the current computing network module of the current model obtained through the current iterative update.

4. The plug-and-play selective module robust fine-tuning method according to claim 3, wherein: The preset iteration rounds T is 10 times, and the disturbance scaling factor ∈ is 0.

1.

5. The plug-and-play selective module robust fine-tuning method according to claim 1, wherein: In step S22, the robust criticality index MRC is calculated according to the following formula: In step S22, the generalization sensitivity index MGSI is calculated according to the following formula: in, represents the first perturbation model, represents the second perturbation model, Represents the initial model; x is the input sample, y is the label of the input sample, is the loss function, k1 and k2 are sample numbers; K1 is the number of samples in the input adversarial sample set, and K2 is the number of samples in the input test sample set.

6. The plug-and-play selective module robust fine-tuning method according to claim 1, wherein: In step S3, based on the robust criticality index and generalization sensitivity index of each layer of network modules included in the initial model, with the goal of maximizing the robust criticality index and minimizing the generalization sensitivity index, the network modules to be fine-tuned are determined, including: S31, normalizing the robust criticality index of each layer of network modules included in the initial model; normalizing the generalization sensitivity index of each layer of network modules included in the initial model; S32. Determine the network module for fine-tuning according to the following formula: in, represents the robust criticality index obtained by normalization of the i-th layer network module, represents the generalization sensitivity index obtained by normalization of the i-th layer network module, and k is the sequence number of the network module for fine-tuning.

7. The plug-and-play selective module robust fine-tuning method according to claim 1, wherein: Step S4: Freeze the parameters of other network modules included in the initial model, and only fine-tune the parameters of the network module to be fine-tuned according to the following objective function using the fine-tuning sample set to obtain the adjusted model parameters θ FT : Among them, x represents the part of the sample in the fine-tuning sample set that belongs to the original sample, δ represents the part of the sample in the fine-tuning sample set that belongs to the added perturbation, Δ is the value range of δ, and y is the label of the original sample corresponding to the sample in the fine-tuning sample set. is the loss function; represents the model obtained by freezing the parameters of the network modules except the k-th layer network module and fine-tuning the k-th layer network module. represents the sample space, θ FT The parameters of the model are obtained by freezing the parameters of the network modules except the k-th layer network module based on the objective function and fine-tuning the k-th layer network module, where k is the serial number of the fine-tuned network module.

8. The plug-and-play selective module robust fine-tuning method according to claim 7, wherein: Step S4 further includes: Using a set of mixing coefficients η, θ is calculated according to the following formula AT and θ FT Combine to obtain a set of parameters to be evaluated: Among them, η p is the pth mixing coefficient η and η p ∈[0,1],θ AT are the parameters of the initial model, θ FT are the parameters of the model obtained after fine-tuning the k-th layer network module, where k is the serial number of the network module for fine-tuning; A robustness evaluation is performed on the model corresponding to each parameter to be evaluated, and the parameter to be evaluated with the highest adversarial accuracy and the best standard accuracy is selected as the final model parameter. The robustly enhanced image classification model is constructed with the final model parameters.

9. A plug-and-play selective module robust fine-tuning method according to any one of claims 1 to 8, characterized in that: The adversarial samples in the adversarial sample set and the fine-tuning sample set are both generated using the PGD-10 model, and the attack strength of the adversarial samples contained in the fine-tuning sample set is higher than the attack strength of the adversarial samples contained in the adversarial sample set.

Images