Neural network robustness enhancement method based on edge-conditioned authentication training

Abstract

This invention discloses a robust enhancement method for neural networks based on edge-conditional authentication training, comprising the following enhancement steps: Step 1, using a neural network model as the basic model architecture and initializing parameters; Step 2, setting batch normalization, setting batch normalization statistics based on clean input; Step 3, searching for L1 regularization strength until the neural network model training performance and neural network validation performance are approximately matched, where L1 regularization is the absolute sum of the neural network model weights; Step 4, entering the training loop to complete the robust enhancement of the neural network model. This invention improves generalization by performing a range search through L1 regularization strength, which is more conducive to obtaining a sparse and concise robust model, facilitating analysis and deployment; using a combination of L1 and L1 regularization for training facilitates a smooth transition from purely natural training to purely robust training, resulting in more stable neural network model training and faster convergence.

Description

Technical Field

[0001] This invention relates to the field of robust enhancement of neural networks, and more specifically, to a robust enhancement method for neural networks based on edge-conditional authentication training. Background Technology

[0002] As a key component of trustworthy artificial intelligence, adversarial robustness—the ability to resist small input perturbations—has become an important research area. Initially, the community focused on using heuristics to generate adversarial examples and defenses against them, but these defenses have proven to be often fragile and easily circumvented by adaptive attackers. Therefore, neural network certification has emerged as a method to provide provably robust guarantees for a given network.

[0003] Two classes of neural network verification methods have been proposed: the complete method, which can compute accurate boundaries but has extremely high computational costs; and the convex relaxation-based method, which is more scalable but provides approximate boundaries. The state-of-the-art (SOTA) verifier combines these two methods by using convex relaxation to accelerate the process of solving the complete method via branch and bound.

[0004] Batch normalization is crucial in certification training, and we consider it an inherent component of CTBench. Specifically, the best practice to date is to set up batch normalization statistics based on clean input and use them to compute the IBP boundary. However, there are some implementation issues with batch normalization: (1) When gradient accumulation is involved, the batch normalization statistics are updated incorrectly due to training with sub-batch statistics; (2) The batch normalization statistics change more than once before the gradient update step, since exponentially accumulated statistics are often used for PGD attacks to evaluate... ,and Batch statistics are then used for evaluation. The first problem is that gradient accumulation becomes ineffective because the quality of batch statistics is highly dependent on batch size; the second problem is that parameter variations hinder their use. Training is performed; models trained with hyperparameters often exhibit strong overfitting patterns. Summary of the Invention

[0005] To address the problems existing in the prior art, the purpose of this invention is to provide a robust enhancement method for neural networks based on edge-conditional authentication training.

[0006] To solve the above problems, the present invention adopts the following technical solution; A robust enhancement method for neural networks based on edge-conditional authentication training includes the following enhancement steps: Step 1: Use a neural network model as the basic model architecture and initialize the parameters; Step 2: Batch normalization settings, setting batch normalization statistics based on clean input; Step 3: Search for the L1 regularization strength until the neural network model training performance and neural network validation performance are roughly matched, and then stop the search. Here, L1 regularization is the absolute sum of the weights of the neural network model. Step 4: Enter the training loop to complete the robust enhancement of the neural network model; Step 5: Model Evaluation; Step 6: Output the robust enhanced neural network model.

[0007] As a further description of the above technical solution: The training loop steps in step four are as follows: S1. Training warm-up: Perturbation radius of normalized statistics for clean input. The learning rate increases linearly from 0 to the target value while remaining constant. S2. Loss function construction, where the total training loss is defined as: ; in Loss due to natural causes For robust loss, For regularization loss, For robust loss weights, the weight range is 0-1; S3. The gradient accumulation parameter correction process of the neural network model uses the normalized statistics of the entire batch, and then backpropagates to update the parameters of the neural network model. S4, if the disturbance radius Once the target value is reached, the learning rate is decayed twice by a constant multiplier, and steps S1 to S3 are repeated until training converges.

[0008] As a further description of the above technical solution: The and Calculate all losses using the same batch normalized statistic, which is set only once per batch.

[0009] As a further description of the above technical solution: If no index can be selected for L1 regularization strength in step three, then random weighted average SWA is enabled.

[0010] As a further description of the above technical solution: The The calculation method is as follows: ; in For neural network models, This is an edge condition map. The loss is the cross-entropy loss, where x is the original input. These are the parameters of the neural network model.

[0011] As a further description of the above technical solution: The evaluation directions in step five include lost terrain, error patterns, model utilization, test distribution out-generalization, and fair comparison.

[0012] As a further description of the above technical solution: In step one, the initialization methods are divided into two types: parameters are initialized using IBP during certification training and parameters are initialized uniformly using Kaiming during adversarial training.

[0013] Compared with the prior art, the advantages of this invention are: (1) In this scheme, the range search is performed by the L1 regularization strength until the training performance and the validation performance are roughly matched. This improves generalization and is more conducive to obtaining a sparse, concise, and robust model, which is easier to analyze and deploy.

[0014] (2) In this scheme, the full batch statistics are used during gradient accumulation, which incurs very little overhead but allows for arbitrary gradient accumulation; we also use batch statistics for PGD attacks and evaluate all content using the current batch statistics, allowing for the use of and Training is conducted using combinations of these methods. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating the principle of the neural network robustness enhancement method of the present invention; Figure 2 This is a schematic diagram of the neural network robustness enhancement method of the present invention; Figure 3 This is a schematic diagram of the neural network certification training process of the present invention. Detailed Implementation

[0016] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention; Please see Figure 1-3 The present invention provides Embodiment 1: A robust enhancement method for neural networks based on edge-conditional authentication training includes the following enhancement steps: Step 1: Use a neural network model as the basic model architecture and initialize the parameters; Step 2: Batch normalization settings, setting batch normalization statistics based on clean input; Step 3: Search for the L1 regularization strength until the neural network model training performance and neural network validation performance are roughly matched, and then stop the search. Here, L1 regularization is the absolute sum of the weights of the neural network model. Step 4: Enter the training loop to complete the robust enhancement of the neural network model; Step 5: Model Evaluation; Step 6: Output the robust enhanced neural network model.

[0017] In step one, the initialization methods are divided into two types: parameters are initialized using IBP during certification training and parameters are initialized uniformly using Kaiming during adversarial training.

[0018] To address the issue of hyperparameter-trained models frequently exhibiting strong overfitting patterns, a range search was performed using L1 regularization strength until training and validation performance were roughly matched. This improved generalization and resulted in sparser, more robust models that are easier to analyze and deploy.

[0019] If no index can be selected for L1 regularization strength in step three, then random weighted average SWA will be used. If the neural network model cannot select indicators through L1 regularization, the random weighted average (SWA) method is used to set the weight parameters of the neural network, thereby increasing the generalization ability of the authentication method.

[0020] The evaluation directions in step five include loss terrain, error patterns, model utilization, test distribution out-generalization, and fair comparison. Regularization strength and utility enable certified training schemes to reduce fragmentation of the loss surface, adaptively maintain model utilization, make common errors on some samples, and generalize well to certain types of corrupted data.

[0021] Please see Figure 1-3 Based on Example 1, the present invention also provides Example 2: The training loop steps in step four are as follows: S1. Training warm-up: Perturbation radius of normalized statistics for clean input. The learning rate increases linearly from 0 to the target value while remaining constant. S2. Loss function construction, where the total training loss is defined as: ; in Loss due to natural causes For robust loss, For regularization loss, For robust loss weights, the weight range is 0-1; S3. The gradient accumulation parameter correction process of the neural network model uses the normalized statistics of the entire batch, and then backpropagates to update the parameters of the neural network model. S4, if the disturbance radius Once the target value is reached, the learning rate is decayed twice by a constant multiplier, and steps S1 to S3 are repeated until training converges.

[0022] The and Calculate all losses using the same batch normalized statistic, which is set only once per batch. Using the full batch statistics during gradient accumulation incurs very little overhead but allows for arbitrary gradient accumulation because a single forward propagation in certified training is typically much less costly than a single full batch update. PGD ​​attacks are performed using batch statistics, and all content is evaluated using the current batch statistics. This way, the batch normalization statistics are set only once per batch, just like in standard training, thus allowing for... and The combination of these methods facilitates a smooth transition from purely natural training to purely robust training, resulting in more stable neural network model training and faster convergence.

[0023] The The calculation method is as follows: ; in For neural network models, This is an edge condition map. The loss is the cross-entropy loss, where x is the original input. These are the parameters of the neural network model.

[0024] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concepts, should be covered within the scope of protection of the present invention.

Claims

1. A robust enhancement method for neural networks based on edge-conditional authentication training, characterized in that, The following enhancement steps are included: Step 1: Use a neural network model as the basic model architecture and initialize the parameters; Step 2: Batch normalization settings, setting batch normalization statistics based on clean input; Step 3: Search for the L1 regularization strength until the neural network model training performance and neural network validation performance are roughly matched, and then stop the search. Here, L1 regularization is the absolute sum of the weights of the neural network model. Step 4: Enter the training loop to complete the robust enhancement of the neural network model; Step 5: Model Evaluation; Step 6: Output the robust enhanced neural network model.

2. The robust enhancement method for neural networks based on edge-conditional authentication training according to claim 1, characterized in that: The training loop steps in step four are as follows: S1. Training warm-up: Perturbation radius of normalized statistics for clean input. The learning rate increases linearly from 0 to the target value while remaining constant. S2. Loss function construction, where the total training loss is defined as: ; in Loss due to natural causes For robust loss, For regularization loss, For robust loss weights, the weight range is 0-1; S3. The gradient accumulation parameter correction process of the neural network model uses the normalized statistics of the entire batch, and then backpropagates to update the parameters of the neural network model. S4, if the disturbance radius Once the target value is reached, the learning rate is decayed twice by a constant multiplier, and steps S1 to S3 are repeated until training converges.

3. The robust enhancement method for neural networks based on edge-conditional authentication training according to claim 1, characterized in that: The and Calculate all losses using the same batch normalized statistic, which is set only once per batch.

4. The robust enhancement method for neural networks based on edge-conditional authentication training according to claim 1, characterized in that: If no index can be selected for L1 regularization strength in step three, then random weighted average SWA is enabled.

5. The robust enhancement method for neural networks based on edge-conditional authentication training according to claim 1, characterized in that: The The calculation method is as follows: D [ CE (f θ (x,e(x)), y)]; in For neural network models, This is an edge condition map. The loss is the cross-entropy loss, where x is the original input. These are the parameters for the neural network model.

6. The robust enhancement method for neural networks based on edge-conditional authentication training according to claim 1, characterized in that: The evaluation directions in step five include lost terrain, error patterns, model utilization, test distribution out-generalization, and fair comparison.

7. The robust enhancement method for neural networks based on edge-conditional authentication training according to claim 1, characterized in that: In step one, the initialization methods are divided into two types: parameters are initialized using IBP during certification training and parameters are initialized uniformly using Kaiming during adversarial training.

Images