Adversarial training neural network image classification method and device and storage medium

Abstract

The application discloses a kind of based on the method, device and storable medium of adversarial training neural network image classification, it is related to image processing technical field, wherein method includes the following steps: obtaining image to be classified;Repair classification neural network and probe network are constructed, and the repair of adversarial training is carried out to repair classification neural network using probe network;Using the repair of adversarial training of repair classification neural network to the image to be classified is classified, and classification result is obtained;The information of model middle layer is obtained by the way of inserting probe network in model middle layer in the application, the difference knowledge between two fields is learned, and input data is enhanced by the method of adversarial attack, the method of combining adversarial training is used to iterate repair to model, and the effect of model fault repair is further improved, to improve the precision of image classification.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image classification method, apparatus, and storage medium based on adversarial training neural networks. Background Technology

[0002] Currently, deep neural networks have achieved outstanding performance in image classification and recognition. Image classification and recognition, as a core technology in autonomous driving, is the foundation for achieving other tasks.

[0003] However, with the widespread application of deep neural networks in image classification, their shortcomings have gradually become apparent. Deep neural networks suffer from domain adaptation problems, and image classification models often perform poorly on samples outside the training distribution. Deep learning algorithms are data-driven algorithms that learn latent knowledge representations through collected training sets. However, in real-world scenarios, there are often data biases and domain shifts between training data and actual data. Actual data may contain noise disturbances, blurring, and other interferences. Directly using the model may result in faulty predictions, leading to very low accuracy in classifying actual data and even serious consequences. However, re-collecting data and retraining the model is often extremely costly or even impractical. Therefore, it is essential to correct model faults using a small amount of faulty data before actual deployment.

[0004] Current technologies for repairing deep neural network models mainly involve fine-tuning the model using erroneous samples collected from real-world scenarios. However, this approach often leads to overfitting and catastrophic forgetting, resulting in a significant reduction in model performance and failing to achieve the desired repair effect, thus impacting classification results.

[0005] Therefore, how to provide an adversarial training neural network image classification method that can solve the above problems is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides an image classification method, apparatus and storage medium based on adversarial training neural network. By inserting probe networks into the intermediate layers of the model to obtain information from the intermediate layers, the differences between the two domains are learned. The input data is then enhanced adversarially through adversarial attacks. Combined with adversarial training, the model is iteratively repaired to further improve the model's fault repair effect, thereby improving the accuracy of image classification.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] An image classification method based on adversarial neural network training includes the following steps:

[0009] Obtain the image to be classified;

[0010] Construct a classification neural network to be repaired and a probe network, and use the probe network to perform adversarial training to repair the classification neural network to be repaired;

[0011] The image to be classified is classified using the classification neural network to be repaired through adversarial training, and the classification result is obtained.

[0012] Preferably, the specific process of constructing a classification neural network to be repaired and a probe network, and using the probe network to perform adversarial training to repair the classification neural network to be repaired includes:

[0013] Construct a classification neural network to be repaired and a probe network, and uniformly insert the probe network into the intermediate layer of the classification neural network to be repaired to obtain the corresponding intermediate layer features;

[0014] The probe network is trained using the features of the intermediate layer;

[0015] The trained probe network is used to perform adversarial training to repair the classification neural network to be repaired;

[0016] Determine whether the classification neural network to be repaired after adversarial training meets the requirements of adversarial training repair; if so, the process ends.

[0017] Preferably, the specific process of training the probe network using the intermediate layer features includes:

[0018] Obtain the repair dataset and divide it into a clean dataset and an error dataset;

[0019] Select different intermediate layer positions of the classification neural network to be repaired, insert multiple probe networks, and label the data domain labels according to the type of the repair dataset while inputting the intermediate layer features;

[0020] The weight parameters of the classification neural network to be repaired are frozen at this time. The probe network is trained by gradient descent and the corresponding loss is calculated to complete the training.

[0021] Preferably, the specific process of using the trained probe network to perform adversarial training and repair on the classification neural network to be repaired includes:

[0022] The same batch of data from both the clean dataset and the error dataset is selected for augmentation processing.

[0023] Freeze the classification neural network to be repaired and the probe network, input the enhanced batch of data into the classification neural network to be repaired and the probe network, generate adversarial examples, and calculate the classification loss;

[0024] While freezing the probe network, update the parameters of the classification neural network to be repaired;

[0025] Finally, while freezing the classification neural network to be repaired, the parameters of the probe network are updated to complete the adversarial training.

[0026] Preferably, the specific process for determining whether the classification neural network to be repaired after adversarial training meets the requirements for adversarial training repair includes:

[0027] Obtain the clean test dataset and the test error dataset;

[0028] The test clean dataset and the test error dataset are classified using the classification neural network to be repaired, and the corresponding classification accuracy is obtained.

[0029] The model performance metrics are determined based on the classification accuracy to determine whether adversarial training repair has been completed.

[0030] Preferably, the specific process for determining whether adversarial training repair is complete includes:

[0031] The specific conditions for determining whether to terminate adversarial training repair are as follows: repair will stop if any of the following conditions are met.

[0032] (1) When the model performance index meets the preset performance requirement value;

[0033] (2) When the number of repair cycles is greater than the maximum number of repair cycles.

[0034] Preferably, the specific process for obtaining the corresponding intermediate layer features includes:

[0035] After the classification neural network to be repaired receives multiple probe networks, it obtains an output feature map and performs global average pooling on the output feature map to obtain a corresponding one-dimensional feature vector, which is an intermediate layer feature.

[0036] The present invention also provides a classification apparatus utilizing the image classification method based on adversarial training neural networks as described in any one of the preceding claims, comprising:

[0037] The acquisition module is used to acquire the image to be classified.

[0038] A construction module is used to construct a classification neural network to be repaired and a probe network, and to use the probe network to perform adversarial training to repair the classification neural network to be repaired.

[0039] The classification module is used to classify the image to be classified using the classification neural network to be repaired through adversarial training, and to obtain the classification result.

[0040] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image classification method as described in any of the preceding claims.

[0041] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses an image classification method, apparatus, and storage medium based on adversarial training neural networks, which has the following beneficial effects:

[0042] 1. This invention can generate samples for model repair at the level of internal features.

[0043] 2. This invention is universal and can be used for all deep neural network models.

[0044] 3. This invention proposes an effective method for repairing deep neural network models. By inserting a probe network into the model and utilizing the feature information of the intermediate layers of the model, adversarial examples are generated for model repair.

[0045] 4. This invention obtains information from the intermediate layers of the model by inserting probe networks, learns the differences between the two domains, and enhances the input data through adversarial attacks. Combined with adversarial training, the model is iteratively repaired to further improve the model's fault repair effect, thereby improving the accuracy of image classification. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 This invention provides an overall flowchart of an image classification method based on adversarial training neural networks;

[0048] Figure 2 A flowchart of the training probe network provided in an embodiment of the present invention;

[0049] Figure 3 A flowchart for adversarial training repair provided in an embodiment of the present invention;

[0050] Figure 4This is a flowchart for determining whether adversarial training repair has been completed, provided in an embodiment of the present invention.

[0051] Figure 5 The present invention provides a structural principle block diagram of an image classification device based on an adversarial training neural network. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] See appendix Figure 1 As shown, this embodiment of the invention discloses an image classification method based on adversarial training neural networks, including the following steps:

[0054] Obtain the image to be classified;

[0055] Construct a classification neural network F to be repaired and a probe network P. Use the probe network P to perform adversarial training to repair the classification neural network F. The classification neural network F to be repaired can be a deep neural network, and the probe network P can be a single-layer or multi-layer fully connected classification network that is inserted into the intermediate layer of the model and uses intermediate features as input.

[0056] The image to be classified is classified using the classification neural network F that has been trained and repaired through adversarial training, and the classification result is obtained.

[0057] In a specific embodiment, the process of constructing a classification neural network F to be repaired and a probe network P, and using the probe network P to perform adversarial training to repair the classification neural network F includes:

[0058] Construct a classification neural network F to be repaired and a probe network P. Insert the probe network P evenly into the intermediate layer of the classification neural network F to be repaired and obtain the corresponding intermediate layer features.

[0059] The probe network P is trained using features from the intermediate layers;

[0060] The trained probe network P is used to perform adversarial training and repair on the classification neural network F to be repaired;

[0061] Determine whether the classification neural network F to be repaired after adversarial training meets the requirements for adversarial training repair; if so, the process ends.

[0062] See appendix Figure 2As shown, in a specific embodiment, the specific process of training the probe network P using intermediate layer features includes:

[0063] Obtain the repair dataset and divide it into clean datasets.

[0064] D1 = {(x1,y1),(x2,y2),...,(x c ,y c The error dataset D2 = {x1, x2, ..., x} f};

[0065] Multiple probe networks P are inserted at different intermediate layer positions of the classification neural network F to be repaired. Simultaneously, the intermediate layer features are input, and the data domain labels y are labeled according to the type of the dataset to be repaired. domain Label D1 is label 0, and label D2 is label 1;

[0066] Freeze the weight parameters of the classification neural network F to be repaired at this time. The weight parameters may include trainable parameters such as convolutional layers, fully connected layers, and normalized layers.

[0067] Image samples from the repair dataset are input into the classification neural network F to be repaired, and the output of the probe network P is obtained. The output and the data domain labels y are then compared. domain Substituting the values ​​into the loss function, the gradient is calculated and the weight parameters of the probe network P are updated using the gradient backpropagation algorithm. The specific expression of the loss function is as follows:

[0068] Loss = CE_Loss(p l (f l (x)),y domain (1)

[0069] In the formula, CE_Loss represents the cross-entropy loss function, f l p represents the l-th layer features of the classification neural network F to be repaired. l Representative and f l The probe network at the corresponding location, where x represents the model input image, containing both clean and faulty samples.

[0070] See appendix Figure 3 As shown, in a specific embodiment, the specific process of using a trained probe network P to perform adversarial training and repair on the classification neural network F to be repaired includes:

[0071] Select the same batch of data (x) from both the clean dataset D1 and the erroneous dataset D2. c ,y c ) and x f , for x c and x fRandomly perform data augmentation processes such as flipping and cropping;

[0072] Freeze the classification neural network F and probe network P to be repaired, and then enhance the x... c and x f Inputs are fed into the classification neural network F to be repaired and the probe network P to obtain the probe output of the model. Adversarial examples are generated iteratively using PGD attack. and Simultaneously, the classification loss is calculated using the cross-entropy loss function, where the specific expression for generating adversarial examples is:

[0073]

[0074] In the formula, x t Let α represent the adversarial example generated in the t-th iteration, α represent the perturbation magnitude, and sign represent the sign function. Represents x t The gradient value, J represents the cross-entropy loss function used, p ln f represents the nth probe. ln This represents acquiring the features corresponding to the probe;

[0075] While freezing the probe network P, the parameters of the classification neural network F to be repaired are updated. The loss function for updating the classification neural network F consists of the original classification loss and the repair loss, and its specific expression is as follows:

[0076] L total =λ*L clean +(1-λ)L repair (3)

[0077] In the formula, λ represents the loss function balancing coefficient, which balances the magnitudes of the two losses, L clean L represents the original classification loss. repair Indicates repairing the loss;

[0078] Finally, while freezing the classification neural network F to be repaired, the parameters of the probe network P are updated. The generated adversarial examples and original samples are input into model F and probe network model P to obtain the probe network output, which is then used to update probe network P. By adding adversarial examples to the training, the robustness of probe network P is improved, completing the adversarial training. The specific loss function expression is as follows:

[0079]

[0080] In the formula, The domain label represents the adversarial example. Since adversarial attacks do not change the domain label, the domain label is the same as the original domain label y. domainConsistent, the parameters of the probe network P are updated using gradient descent.

[0081] Repeat the above steps multiple times until the entire repair dataset has been traversed.

[0082] Specifically, the purpose of the original classification loss is to maintain the model's classification accuracy on clean samples. The classification loss is calculated using clean samples. The formula for calculating the loss is:

[0083] L clean =CE_Loss(F(x) c ),y c (5)

[0084] In the formula, F represents the model, and x c and y c These represent clean samples and their labels, respectively.

[0085] The purpose of loss repair is to ensure the model's accuracy on erroneous samples. The KL divergence loss is calculated using the original samples and adversarial samples, and its formula is as follows:

[0086] L repair =KL_loss(F(x adv ),F(x)) (6)

[0087] In the formula, KL_loss represents the KL divergence loss function, x represents the original sample, and x adv This represents the adversarial sample generated from the original sample.

[0088] In a specific embodiment, the process of determining whether the classification neural network F to be repaired after adversarial training meets the requirements of adversarial training repair includes:

[0089] Obtain a clean test dataset T1 = {(x1,y1),(x2,y2),...,(x tc ,y tc The test error dataset T2 = {(x1,y1),(x2,y2),...,(x1,y1),(x2,y2),...,(x2,y2),...,(x2,y1 ... tf ,y tf )};

[0090] The classification neural network F to be repaired is used to classify the clean test dataset T1 and the incorrect test dataset T2, and the corresponding classification accuracy is obtained. and

[0091] The model performance metric is determined based on classification accuracy to determine whether adversarial training repair has been completed. The specific expression is as follows:

[0092]

[0093] In the formula, λ is the balance coefficient, which is used to weigh the importance between the original performance and the repair performance of the model. When λ = 1, the model repair performance will only consider the accuracy of the model on T2, that is, only the repair performance will be considered.

[0094] See appendix Figure 4 As shown, in a specific embodiment, the process of determining whether adversarial training repair is complete includes:

[0095] The specific conditions for determining whether to terminate adversarial training repair are as follows: Repair will stop if any of the following conditions are met:

[0096] ① When the model performance index φ meets the preset performance requirement value, that is, φ>Φ;

[0097] ②When the number of repair cycles is greater than the maximum number of repair cycles.

[0098] In addition, when the performance metrics do not meet the performance requirements, the model with the highest performance metrics during the repair process will be output.

[0099] The final classification neural network F to be repaired is output.

[0100] In a specific embodiment, the process of obtaining the corresponding intermediate layer features includes:

[0101] After the classification neural network F to be repaired receives multiple probe networks P, the output feature map is obtained, and global average pooling is performed on the output feature map to obtain the corresponding one-dimensional feature vector. The one-dimensional feature vector is the intermediate layer feature.

[0102] Referring to Figure 5, this embodiment of the invention also provides a classification apparatus for an image classification method based on an adversarial training neural network utilizing any of the above-mentioned methods, comprising:

[0103] The acquisition module is used to acquire the image to be classified.

[0104] The building module is used to construct the classification neural network to be repaired and the probe network. The probe network is used to perform adversarial training to repair the classification neural network.

[0105] The classification module is used to classify the image to be classified using the repaired classification neural network that has been trained adversarially, and obtain the classification result.

[0106] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image classification method as described in any of the above embodiments.

[0107] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0108] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image classification method based on adversarial training neural networks, characterized in that, Includes the following steps: Obtain the image to be classified; Constructing a classification neural network to be repaired and a probe network, and using the probe network to perform adversarial training to repair the classification neural network to be repaired, wherein the probe network is a single-layer or multi-layer fully connected classification network, the specific process includes: Construct a classification neural network to be repaired and a probe network, and uniformly insert the probe network into the intermediate layer of the classification neural network to be repaired to obtain the corresponding intermediate layer features; The probe network is trained using the features of the intermediate layer; The trained probe network is used to perform adversarial training to repair the classification neural network to be repaired; Determine whether the classification neural network to be repaired after adversarial training meets the requirements of adversarial training repair; if so, end the process. The image to be classified is classified using the classification neural network to be repaired through adversarial training, and the classification result is obtained.

2. The image classification method based on adversarial training neural networks according to claim 1, characterized in that, The specific process of training the probe network using the intermediate layer features includes: Obtain the repair dataset and divide it into a clean dataset and an error dataset; Select different intermediate layer positions of the classification neural network to be repaired, insert multiple probe networks, and simultaneously label the data domain according to the type of the repair dataset while inputting the intermediate layer features; The weight parameters of the classification neural network to be repaired are frozen at this time. The probe network is trained by gradient descent and the corresponding loss is calculated to complete the training.

3. The image classification method based on adversarial training neural networks according to claim 2, characterized in that, The specific process of using the trained probe network to perform adversarial training and repair on the classification neural network to be repaired includes: The same batch of data from both the clean dataset and the error dataset is selected for augmentation processing. Freeze the classification neural network to be repaired and the probe network, input the enhanced batch of data into the classification neural network to be repaired and the probe network, generate adversarial examples, and calculate the classification loss; While freezing the probe network, update the parameters of the classification neural network to be repaired; Finally, while freezing the classification neural network to be repaired, the parameters of the probe network are updated to complete the adversarial training.

4. The image classification method based on adversarial training neural networks according to claim 1, characterized in that, The specific process for determining whether the classification neural network to be repaired after adversarial training meets the requirements for adversarial training repair includes: Obtain the clean test dataset and the test error dataset; The test clean dataset and the test error dataset are classified using the classification neural network to be repaired, and the corresponding classification accuracy is obtained. The model performance metrics are determined based on the classification accuracy to determine whether adversarial training repair has been completed.

5. The image classification method based on adversarial training neural networks according to claim 4, characterized in that, The specific process for determining whether adversarial training repair has been completed includes: The specific conditions for determining whether to terminate adversarial training repair are as follows: repair will stop if any of the following conditions are met. (1) When the model performance index meets the preset performance requirement value; (2) When the number of repair rounds is greater than the maximum number of repair rounds.

6. The image classification method based on adversarial training neural network according to claim 1, characterized in that, The specific process of obtaining the corresponding intermediate layer features includes: After the classification neural network to be repaired receives multiple probe networks, it obtains an output feature map and performs global average pooling on the output feature map to obtain a corresponding one-dimensional feature vector, which is an intermediate layer feature.

7. A classification apparatus utilizing the image classification method based on adversarial training neural networks as described in any one of claims 1-6, characterized in that, include: The acquisition module is used to acquire the image to be classified. A construction module is used to construct a classification neural network to be repaired and a probe network, and to use the probe network to perform adversarial training to repair the classification neural network to be repaired. The classification module is used to classify the image to be classified using the classification neural network to be repaired through adversarial training, and to obtain the classification result.

8. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the image classification method as described in any one of claims 1 to 6.

Images