Model defense method and device based on image enhancement and adversarial training

Abstract

The invention provides a model defense method and device based on image enhancement and adversarial training, and the method comprises the steps: obtaining an image in a target detection data set, and carrying out the data enhancement; generating an adversarial sample from the current image subjected to data enhancement through an adversarial attack; sending the adversarial sample and the data enhanced image into a preset generative adversarial network to remove disturbance in the adversarial sample to obtain a clean sample; and sending the image subjected to data enhancement, the adversarial sample and the clean sample into a target detector for training so as to increase the robustness of the target detector, and outputting a category and a position corresponding to a target detection data set. According to the method, the robustness of the target detector is improved in various attack scenes through an adversarial training method.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a model defense method and device based on image enhancement and adversarial training. Background Art

[0002] The task of target detection is to output the location and category of the target object in a given video or image. With the continuous development and breakthrough achievements of deep learning technology, target detection methods based on deep learning have gradually replaced traditional detection algorithms based on manual features. As one of the basic core tasks of computer vision, target detection is widely used in face recognition, target tracking, autonomous driving, medical diagnosis, intelligent security and other fields. Its safety is extremely important and has a huge and profound impact on the way of life and production of human society.

[0003] Recent studies have shown that adversarial examples can successfully interfere with detectors in both digital and physical fields, posing a great threat to detector-based applications. Therefore, it is necessary to further explore the mechanism of adversarial examples on detectors to help us improve the robustness of detector-based AI applications. Summary of the invention

[0004] The technical problem to be solved by the present invention is how to improve the robustness of target detectors under various attack scenarios; in view of this, the present invention provides a model defense method and device based on image enhancement and adversarial training.

[0005] The technical solution adopted by the present invention is a model defense method based on image enhancement and adversarial training, comprising: Step 1: Obtain images in the target detection dataset and perform data enhancement; Step 2: Generate adversarial samples through adversarial attacks on the current image that has undergone data augmentation; Step 3: Send the adversarial sample and the data-enhanced image to a preset generative adversarial network to remove the disturbance in the adversarial sample and obtain a clean sample; Step 4: Send the data-enhanced image, the adversarial sample and the clean sample to the target detector training to increase the robustness of the target detector, and output the category and position corresponding to the target detection data set.

[0006] In one embodiment, in step 1, the data enhancement method includes: Mirroring and flipping, rotation and scaling, noise processing, data interpolation, generative adversarial networks, sample synthesis, data resampling.

[0007] In one embodiment, in step 2, the adversarial attack includes: a black box attack and / or a white box attack.

[0008] In one embodiment, the step three further comprises: The adversarial sample is input into the generator to generate the clean sample.

[0009] Using the data-augmented image and the generated clean sample, determining the loss of the discriminator and updating the parameters of the discriminator; Using the clean samples generated by the generator, the loss of the generator is determined, and the parameters of the generator are updated.

[0010] Another aspect of the present invention further provides a model defense device based on image enhancement and adversarial training, comprising: An enhancement unit, configured to obtain images in a target detection dataset and perform data enhancement; The adversarial unit is configured to generate adversarial samples through adversarial attacks on the current image that has been augmented with data; A denoising unit, configured to send the adversarial sample and the data-enhanced image to a preset generative adversarial network to remove the disturbance in the adversarial sample and obtain a clean sample; The training unit is configured to send the data-augmented image, the adversarial sample and the clean sample to the target detector training to increase the robustness of the target detector, and output the category and position corresponding to the target detection data set.

[0011] In one embodiment, in the enhancement unit, the data enhancement method includes: Mirroring and flipping, rotation and scaling, noise processing, data interpolation, generative adversarial networks, sample synthesis, data resampling.

[0012] In one embodiment, in the adversarial unit, the adversarial attack includes: a black box attack and / or a white box attack.

[0013] In one embodiment, the denoising unit is further configured as follows: The adversarial sample is input into the generator to generate the clean sample.

[0014] Using the data-augmented image and the generated clean sample, determining the loss of the discriminator and updating the parameters of the discriminator; Using the clean samples generated by the generator, the loss of the generator is determined, and the parameters of the generator are updated.

[0015] Another aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the model defense method based on image enhancement and adversarial training as described in any one of the above items.

[0016] Another aspect of the present invention further provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model defense method based on image enhancement and adversarial training as described in any of the above items are implemented.

[0017] Compared with the existing defense method based on adversarial training, the present invention increases the diversity of training samples and can further improve the robustness of the target detector. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a flow chart of a model defense method based on image enhancement and adversarial training according to an embodiment of the present invention; Figure 2 A logical schematic diagram of a model defense method based on image enhancement and adversarial training according to an embodiment of the present invention; Figure 3 A schematic diagram of implementing data enhancement according to an embodiment of the present invention; Figure 4 A schematic diagram of a framework of an attack method based on optimization iteration according to an embodiment of the present invention; Figure 5 A schematic diagram of a framework of an attack method based on generator generation according to an embodiment of the present invention; Figure 6 A diagram of a generative adversarial network architecture according to an embodiment of the present invention; Figure 7 A schematic diagram of generating adversarial network training according to an embodiment of the present invention; Figure 8 A schematic diagram of the composition of a model defense device based on image enhancement and adversarial training according to an embodiment of the present invention; Fig. 9 FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the present invention is described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0020] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined in this article.

[0021] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] The embodiment of the present invention provides a model defense method based on image enhancement and adversarial training, such as Figure 1 As shown, including: Step 1: Obtain images in the target detection dataset and perform data enhancement; Step 2: Generate adversarial samples through adversarial attacks on the current image that has undergone data augmentation; Step 3: Send the adversarial sample and the data-enhanced image to a preset generative adversarial network to remove the disturbance in the adversarial sample and obtain a clean sample; Step 4: Send the data-enhanced image, the adversarial sample and the clean sample to the target detector training to increase the robustness of the target detector, and output the category and position corresponding to the target detection data set.

[0023] refer to Figure 2 , the method provided in this embodiment will be described in detail step by step below.

[0024] Step 1: Select the corresponding target detection dataset, avoid overfitting through data enhancement, and improve the model Robustness, reducing the model's sensitivity to images.

[0025] Overfitting refers to the situation where the model selected during learning contains too many parameters, so that the model predicts the known data well but predicts the unknown data poorly. In this case, the model may just remember the training set data instead of learning the data features.

[0026] Better ways to deal with overfitting include: Get more data from the source; Estimate the data distribution parameters based on the current data set and use this distribution to generate more data; Data Augmentation In this embodiment, data enhancement is used to avoid overfitting.

[0027] It should be noted that data enhancement is also called data amplification, which means making limited data produce the equivalent value of more data without substantially increasing the data.

[0028] In deep learning, data augmentation is to change the input data in a certain way to generate more training samples, thereby improving the generalization ability and effect of the model. Data augmentation can reduce the model's excessive dependence on certain features, thereby avoiding overfitting.

[0029] In deep learning, sufficient samples are required. The more samples there are, the better the trained model will be and the stronger the generalization ability of the model will be. However, in practice, if the number of samples is insufficient or the sample quality is not good enough, data enhancement is needed to improve the sample quality.

[0030] Figure 3 These are some common operations for image enhancement. The categories of data enhancement include: 1) Mirroring and flipping: Flip the image data horizontally or vertically to create a mirror image and increase data diversity.

[0031] 2) Rotation and scaling: Rotate, scale, or crop the image to increase the model’s robustness to changes in different scales and angles.

[0032] 3) Noise processing: Add random noise to the data, such as Gaussian noise, random cropping, color transformation, etc., to make the model more adaptable to reality.

[0033] 4) Data interpolation: Fill in data through linear or nonlinear interpolation methods, which is particularly suitable for time series data and missing value processing.

[0034] 5) Generative Adversarial Networks (GANs): Generate data that is similar to the original data but slightly different from it to expand the dataset.

[0035] 6) Sample synthesis: Create new synthetic data samples based on the properties of existing data samples.

[0036] 7) Data resampling: For unbalanced datasets, increase the samples of the minority class or reduce the samples of the majority class to improve data balance.

[0037] The following is a partial implementation of the code for image enhancement: import torch import torchvision.transforms as transforms from PIL import Image import numpy as np import random # Define the class for image enhancement class ImageAugmentation: def __init__(self): self.transform = transforms.Compose([ transforms.RandomHorizontalFlip(p=0.5), # Random horizontal flip transforms.RandomVerticalFlip(p=0.5), # Random vertical flip transforms.RandomRotation(degrees=30), #Random rotation transforms.RandomResizedCrop(size=(224, 224), scale=(0.8, 1.2)), #Random scaling and cropping transforms.ToTensor() #Convert to Tensor ]) #Add noise processing def add_noise(self, img_tensor, mean=0, std=0.1): noise = torch.randn(img_tensor.size()) * std + mean noisy_img = img_tensor + noise return torch.clip(noisy_img, 0., 1.) # Ensure that the pixel value is in the range [0, 1] def __call__(self, img): img = self.transform(img) img = self.add_noise(img) return img The following are the object detection datasets and their data augmentation representations: X represents the target detection dataset, and the set form is expressed as X={X1,X2,......,Xn} (Xi represents an image in the dataset).

[0038] X' represents the image after data enhancement, and the set form is expressed as X'={X1',X2',......,Xn'}.

[0039] Step 2: Generate adversarial samples from the data-enhanced image X' through corresponding black-box attack or white-box attack Ben X''.

[0040] According to the attacker's ability or knowledge, adversarial attacks in target detection can be divided into white-box attacks and black-box attacks.

[0041] In a white-box attack, the attacker's idea is generally to input adversarial samples so that their feature maps, model logits or model output results are close to the specified target label results of the targeted attack or far away from the real label results of the image. The key step is that the attacker constructs a loss function, uses gradient derivation optimization, directly generates samples or trains a generative model. Furthermore, according to the generation method of adversarial noise, white-box attacks can be divided into attack methods based on optimization iteration and attack methods based on generator generation. The advantages of the former are simple generation and stable training, while the advantages of the latter are fast speed and high efficiency.

[0042] The attack method based on optimization iteration has the core step of designing the loss function and using gradient propagation to reversely optimize the adversarial samples. In this process, no additional training of other networks is required. The method of generating adversarial samples is simple, but the speed is slow and the efficiency is low. Figure 4 It is an attack method framework based on optimization iteration.

[0043] The attack method based on optimization iteration needs to continuously reversely iterate and optimize the noise for specific images or specific categories, which consumes a lot of resources, takes a long time to calculate, and is inefficient, but this method is intuitive and simple. The core idea of ​​the attack method based on generator generation is to learn the distribution of adversarial noise and directly use the generator to generate adversarial noise. Figure 5 It is an attack method framework based on generator generation.

[0044] In real scenarios, attackers usually cannot obtain the details of the attacked model, so black-box attacks naturally receive more attention from academia and industry.

[0045] There are two main methods of black-box attack: query-based black-box attack and transfer-based black-box attack. Among them, query-based black-box attack allows to obtain the output information of the target model, however, the model of the real scene often only provides a limited number of queries and the query cost is high; while the transfer-based black-box attack method does not need to obtain any information of the target model, which is more conducive to the attack of the real scene, but the success rate is also often lower.

[0046] Many black-box attacks based on migration rely on multi-model integration to improve the migration of attacks. That is, in the stage of training adversarial samples, multiple different white-box models are integrated in the hope that the trained adversarial samples can show stronger attack capabilities on the new black-box models. However, this method often consumes a lot of training time and resources. At the same time, obtaining multiple models for the same task may also be difficult to implement in real applications.

[0047] In this invention, our adversarial training defense method shows good performance in both white-box attack and black-box attack scenarios. The following are the steps to generate adversarial samples: 1) Model represents the target detection model, and its expression is model=Fw(x), where x represents the image input to the model.

[0048] 2) First, the prediction box and category confidence expression are obtained through the target detection model as Bbox,conf=Fw(X), where X'' represents the image after black box attack or white box attack.

[0049] 3) The expression in set form is X''={A(X' )}, where A represents the specific attack method selected. Note: The expression in set form will be different depending on the selected attack method.

[0050] Step 3: Send the adversarial sample X'' and the data-enhanced image X' into the generative adversarial network to remove the adversarial sample Perturbations in X'', to obtain clean samples X'''.

[0051] The Generative Adversarial Network is actually a combination of two networks: the Generator is responsible for generating simulated data, and the Discriminator is responsible for judging whether the input data is real or generated. The Generator must continuously optimize the data it generates so that the Discriminator cannot judge, and the Discriminator must also optimize itself to make its judgment more accurate. The relationship between the two forms an adversarial relationship, so it is called an adversarial network.

[0052] For GAN, a simple understanding is that it can be regarded as a game process. We can regard the generative model and the discriminative model as the two sides of the game. For example, in the game between party A and party B, the generative model G is equivalent to the corresponding model of party A, and its purpose is to strive for a favorable situation as much as possible based on its own conditions and party B's current strategy. The discriminative model D is equivalent to the corresponding model of party B, and its purpose is to identify party A's strategy as much as possible. In this way, through the competition between parties A and B and the improvement towards the goal, the generative model can finally reach the Nash equilibrium effect (the success rate of the competition between the two parties is 0.5). Figure 6 This is the basic architecture diagram of GAN in this embodiment.

[0053] In this embodiment, the adversarial sample X'' generated after the white-box attack or the black-box attack can be used as the input of the generator G, and the data-enhanced image X' can be used as the true data. The false data is the clean sample X''' from which the adversarial perturbation is removed by the generator G. Figure 7 This is a framework diagram of a generative adversarial network in this embodiment.

[0054] The following are the training steps of the generative adversarial network in the present invention: S1, input the adversarial perturbation image X'' into the generator to generate the de-perturbed image X'''.

[0055] S2, update the discriminator (Discriminator) using the real clean image X' and the generated de-perturbation image X''', calculate the loss of the discriminator, and update the parameters of the discriminator.

[0056] S3, Update the Generator (Generator) Use the de-perturbed image generated by the generator, calculate the loss of the generator, and update the parameters of the generator.

[0057] For example, the following is part of the code for generating an adversarial network import torch import torch.nn as nn import torch.optim as optim import torchvision import torchvision.transforms as transforms # Define the generator class Generator(nn.Module): def __init__(self, input_size, hidden_size, output_size): super(Generator, self).__init__() self.main = nn.Sequential( nn.Linear(input_size, hidden_size), nn.ReLU(True), nn.Linear(hidden_size, hidden_size), nn.ReLU(True), nn.Linear(hidden_size, output_size), nn.Tanh() ) def forward(self, x): return self.main(x) # Define the discriminator class Discriminator(nn.Module): def __init__(self, input_size, hidden_size): super(Discriminator, self).__init__() self.main = nn.Sequential( nn.Linear(input_size, hidden_size), nn.LeakyReLU(0.2, inplace=True), nn.Linear(hidden_size, hidden_size), nn.LeakyReLU(0.2, inplace=True), nn.Linear(hidden_size, 1), nn.Sigmoid() ) def forward(self, x): return self.main(x) Step 4: Compare the data-augmented image X' with the adversarial sample generated by black-box attack or white-box attack X'' and the clean sample X''' generated by the adversarial generative network are sent to the target detector for training to increase the robustness of the target detector sex.

[0058] According to the detection steps, it can be divided into single-stage target detection algorithm and two-stage target detection algorithm. The single-stage target detection algorithm abstracts the target detection problem into a regression problem, directly integrates feature extraction, position regression and target classification into one stage, and calculates the original image. Representative algorithms for single-stage target detection include YOLO series, SSD (singleshot multibox detector), RetinaNet, etc.

[0059] The two-stage target detection algorithm first calculates the original image through a selective search algorithm or introduces a region proposal network (RPN) to generate candidate regions; Then, the candidate areas generated in the first stage are corrected for the second time to obtain the classification and location results.

[0060] Representative algorithms for two-stage object detection include the region-based convolutional neural network (R-CNN) series, R-FCN (region-based fully convolutional network), etc. In comparison, the two-stage algorithm needs to generate a large number of candidate boxes for positioning and classification, which is generally slow and has a complex network structure but high accuracy; the single-stage algorithm is fast but relatively low in accuracy. However, some advanced single-stage algorithms have made significant progress in accuracy. Therefore, in this invention, our adversarial defense is mainly aimed at the single-stage object detection algorithm.

[0061] The following is a rough training process for a single-stage object detector: Collect data: Collect image datasets containing target objects and annotate them. Commonly used annotation formats include Pascal VOC and COCO formats.

[0062] Initialization parameters: Use pre-trained weights (such as Darknet-53 weights pre-trained on ImageNet) to initialize the model to speed up convergence.

[0063] Forward propagation: Input the image and perform forward propagation through the convolutional neural network to generate a feature map.

[0064] Calculate loss: Calculate the loss based on the predicted results and the true annotations.

[0065] Backpropagation and parameter update: Gradients are calculated through the backpropagation algorithm and the model parameters are updated using an optimizer such as SGD or Adam.

[0066] Iterative training: Iterate multiple times (multiple epochs) on the entire training dataset and continuously adjust the model parameters to minimize the loss function.

[0067] Compared with the prior art, this embodiment can be used to improve the robustness of various single-stage target detectors in various attack scenarios. Specifically, it uses an adversarial defense method based on model training to improve the robustness of target detectors.

[0068] The second embodiment of the present invention corresponds to the first embodiment. This embodiment introduces a model defense device based on image enhancement and adversarial training, such as Figure 8 As shown, it includes the following components: An enhancement unit, configured to obtain images in a target detection dataset and perform data enhancement; The adversarial unit is configured to generate adversarial samples through adversarial attacks on the current image that has been augmented with data; A denoising unit, configured to send the adversarial sample and the data-enhanced image to a preset generative adversarial network to remove the disturbance in the adversarial sample and obtain a clean sample; The training unit is configured to send the data-augmented image, the adversarial sample and the clean sample to the target detector training to increase the robustness of the target detector, and output the category and position corresponding to the target detection data set.

[0069] In this embodiment, in the enhancement unit, the data enhancement method includes: Mirroring and flipping, rotation and scaling, noise processing, data interpolation, generative adversarial networks, sample synthesis, data resampling.

[0070] In this embodiment, in the adversarial unit, the adversarial attack includes: a black box attack and / or a white box attack.

[0071] In this embodiment, the denoising unit is further configured as: The adversarial sample is input into the generator to generate a clean sample.

[0072] Using the data-augmented image and the generated clean sample, determining the loss of the discriminator and updating the parameters of the discriminator; Using the clean samples generated by the generator, the loss of the generator is determined, and the parameters of the generator are updated.

[0073] A third embodiment of the present invention is an electronic device, such as Fig. 9 As shown, it can be understood as a physical device, including a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the following operations are performed: Step 1: Obtain images in the target detection dataset and perform data enhancement; Step 2: Generate adversarial samples through adversarial attacks on the current image that has undergone data augmentation; Step 3: Send the adversarial sample and the data-enhanced image to a preset generative adversarial network to remove the disturbance in the adversarial sample and obtain a clean sample; Step 4: Send the data-enhanced image, the adversarial sample and the clean sample to the target detector training to increase the robustness of the target detector, and output the category and position corresponding to the target detection data set.

[0074] The fourth embodiment of the present invention, the process of the model defense method based on image enhancement and adversarial training in this embodiment is the same as that of the first, second or third embodiment, the difference is that in engineering implementation, this embodiment can be implemented by means of software plus a necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the method of the present invention can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for a device to execute the method described in the embodiment of the present invention.

[0075] Through the description of the specific implementation methods, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the predetermined purpose should be obtained. However, the accompanying drawings are only for reference and illustration purposes and are not intended to limit the present invention.

Claims

1. A model defense method based on image enhancement and adversarial training, characterized in that: include: Step 1: Obtain images in the target detection dataset and perform data enhancement; Step 2: Generate adversarial samples through adversarial attacks on the current image that has undergone data augmentation; Step 3: Send the adversarial sample and the data-enhanced image to a preset generative adversarial network to remove the disturbance in the adversarial sample and obtain a clean sample; Step 4: Send the data-enhanced image, the adversarial sample and the clean sample to the target detector training to increase the robustness of the target detector, and output the category and position corresponding to the target detection data set.

2. The model defense method based on image enhancement and adversarial training according to claim 1 is characterized in that: In the step 1, the data enhancement method includes: Mirroring and flipping, rotation and scaling, noise processing, data interpolation, generative adversarial networks, sample synthesis, data resampling.

3. The model defense method based on image enhancement and adversarial training according to claim 2 is characterized in that: In the step 2, the adversarial attack includes: a black box attack and / or a white box attack.

4. The model defense method based on image enhancement and adversarial training according to claim 3 is characterized in that: The step three also includes: The adversarial sample is input into the generator to generate the clean sample. Using the data-augmented image and the generated clean sample, determining the loss of the discriminator and updating the parameters of the discriminator; Using the clean samples generated by the generator, the loss of the generator is determined, and the parameters of the generator are updated.

5. A model defense device based on image enhancement and adversarial training, characterized in that: include: An enhancement unit, configured to obtain images in a target detection dataset and perform data enhancement; The adversarial unit is configured to generate adversarial samples through adversarial attacks on the current image that has been augmented with data; A denoising unit, configured to send the adversarial sample and the data-enhanced image to a preset generative adversarial network to remove the disturbance in the adversarial sample and obtain a clean sample; The training unit is configured to send the data-augmented image, the adversarial sample and the clean sample to the target detector training to increase the robustness of the target detector, and output the category and position corresponding to the target detection data set.

6. The model defense device based on image enhancement and adversarial training according to claim 5 is characterized in that: In the enhancement unit, the data enhancement method includes: Mirroring and flipping, rotation and scaling, noise processing, data interpolation, generative adversarial networks, sample synthesis, data resampling.

7. The model defense device based on image enhancement and adversarial training according to claim 6 is characterized in that: In the adversarial unit, the adversarial attack includes: a black box attack and / or a white box attack.

8. The model defense device based on image enhancement and adversarial training according to claim 7 is characterized in that: The denoising unit is further configured as: The adversarial sample is input into the generator to generate the clean sample. Using the data-augmented image and the generated clean sample, determining the loss of the discriminator and updating the parameters of the discriminator; Using the clean samples generated by the generator, the loss of the generator is determined, and the parameters of the generator are updated.

9. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the model defense method based on image enhancement and adversarial training as described in any one of claims 1 to 4 are implemented.

10. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the model defense method based on image enhancement and adversarial training as described in any one of claims 1 to 4.

Images