A multi-scale deep residual denoising network for eliminating adversarial noise of neural networks

Abstract

The application discloses a kind of multi-scale deep residual denoising networks for eliminating neural network adversarial noise, it is related to the field of neural network-based image recognition processing, to solve the problem of neural network model prediction sample error rate increase caused by existing technology in adversarial sample, the technical scheme is, including residual network, model structure, training set and loss function, the scaling factor of residual block scaling layer of residual network is 0.1;Model structure adopts the new model obtained by the connection of two different structure models;Training set selects FGSM and PGD;And a new loss function is proposed.By adding the residual network structure that can improve the denoising performance, it has a multi-scale network structure that better matches the denoising task, and under the action of high-performance training set and loss function, higher denoising effect is realized.

Description

Technical Field

[0001] This invention relates to the field of image recognition processing based on neural networks, specifically to a multi-scale deep residual denoising network for eliminating adversarial noise in neural networks. Background Technology

[0002] Deep neural networks possess excellent nonlinear fitting capabilities, high-order feature combination capabilities, and generalization abilities. Leveraging these capabilities, deep learning has rapidly developed in recent years in fields such as computer vision, natural language processing, speech recognition, image generation, and autonomous driving, with its research results providing convenience to many aspects of human life. However, related research (see Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good-fellow, and Rob Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013) shows that deep neural networks are easily affected by adversarial examples. Adversarial examples consist of a clean original sample and a small, imperceptible perturbation. When the generated adversarial example is input into the neural network, the network will produce an incorrect result. This phenomenon demonstrates the vulnerability of deep neural networks, leading researchers to explain and study the potential weaknesses of deep neural networks and attempt to improve their robustness.

[0003] The emergence of adversarial examples has challenged the credibility of deep neural networks. Researching defensive methods against adversarial examples is crucial and urgent. Since adversarial examples are constructed by adding noise to the original examples, denoising them before sending them to the target model is a natural idea. Unlike defensive approaches that modify the original deep neural network model, such as adversarial training (see Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014), removing adversarial noise does not require modifying the original model. In other words, this defensive method can be used for all deployed or planned models, regardless of whether the model has been trained.

[0004] In research on eliminating adversarial noise to restore adversarial examples to clean samples, existing techniques have proposed a method (see Huan Xu, Chengfei Pei, Gaoming Yang, et al., “Adversarial example defense based on image reconstruction,” PeerJ Computer Science, vol.7, pp.e811, 2021). This method first destroys the attack elements of adversarial examples by compressing the pixel depth of the image, and then restores the compressed image with the help of a depth image reconstruction network. Further research (see Huan Xu, Chengfei Pei, Gaoming Yang, et al., “Adversarial example defense based on image reconstruction,” PeerJ Computer Science, vol.7, pp.e811, 2021) has observed that the foreground region has more interference than the background region in adversarial examples. Therefore, a weighted hotspot region extraction method based on weakly supervised target localization is proposed to label the foreground and background regions. An operator-based boundary detection method is also proposed to separate these two regions. Finally, a bicubic interpolation filtering algorithm is used to eliminate adversarial interference in the foreground region. However, the image compression and filtering algorithms mentioned may destroy the original features of the image, posing an additional challenge to the recognition ability of the final image recognizer. Existing technologies address this problem using an alternative approach: training an adversarial sample denoiser using a denoising autoencoder (DAE). A DAE is a neural network with a downsampling encoder and an upsampling decoder. The DAE is trained unsupervised, taking a noisy image as input by adding random Gaussian noise of varying magnitudes, including zero, and outputting a clean image. The encoder maps the noisy image to a latent representation of the clean image, while the decoder maps it back to the clean image. Ultimately, the DAE achieves noise reduction through the encoding and decoding processes. However, because the noisy image used is Gaussian noise, this training method only considers the feature distribution of the Gaussian noise image and not the feature distribution of the adversarial sample image. Furthermore, the encoder and decoder may not be the optimal model structure for noise removal.

[0005] Therefore, in summary, existing technologies have three main limitations: 1) Autoencoders have reached a performance bottleneck; 2) Autoencoders were originally proposed to solve image segmentation tasks, but their network structure is not well-suited for image reconstruction tasks; 3) Using only Gaussian noise images to learn denoising tasks prevents denoising neural networks from recognizing and understanding the feature distribution of adversarial noise images, leading to a decrease in their defense against adversarial samples. Summary of the Invention

[0006] In view of the problems existing in the prior art, the present invention discloses a multi-scale deep residual denoising network for eliminating adversarial noise in neural networks. It incorporates a high-performance residual network, a multi-scale image size processing structure, a more reasonable training set, and a new loss function, thereby achieving higher performance.

[0007] The technical solution adopted includes residual networks, model structure, training set, and loss function;

[0008] Part 1: Residual Networks

[0009] Residual networks are an excellent image recognition network architecture. By adding residual blocks to the network, the number of parameters is increased, thereby improving the overall learning ability and image recognition accuracy. In the field of image super-resolution, when the number of feature maps in a super-resolution network model increases to a certain extent, the numerical values ​​during training become unstable (see Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in Thirty-first AAAI conference on artificial intelligence, 2017). This problem can be solved by using a residual scaling factor of 0.1. The residual blocks added to the residual network include convolutional layers, ReLU activation layers, and scaling layers, arranged in the following order:

[0010] Convolutional layer, ReLU activation layer, convolutional layer, scaling layer, where the scaling factor of the scaling layer is 0.1;

[0011] Part Two: Model Structure

[0012] The model has two structures. The first structure starts from the input and consists of a convolutional layer, a residual layer, another convolutional layer, another convolutional layer, and another convolutional layer. Below the first convolutional layer, there are two channels: one channel connects to the residual layer to process the data, and the other channel bypasses the residual layer without processing the data. After passing through the first convolutional layer, the data is split into two parts and passes through the two channels. Then, below the residual layer, the results of the two channels are added together. This structure provides an intermediate value, which is then connected to the second structure, a traditional DAE structure, consisting of a convolutional layer, a residual layer, a convolutional layer, a deconvolutional layer, and another deconvolutional layer. This is then connected to the output. Below the first convolutional layer, there are two channels: one channel connects to the residual layer to process the data, and the other channel bypasses the residual layer without processing the data. After passing through the first convolutional layer, the data is split into two parts and passes through the two channels. Then, below the residual layer, the results of the two channels are added together.

[0013] The residual layers are all the network layers in the residual block of the first part; each convolutional layer and deconvolutional layer is followed by an activation layer;

[0014] The second structure is called the "1q1" model structure. Here, the first "1" refers to the initial size of the input image; the middle "q" refers to the hidden layer data after the image data is downsampled within the model, which simplifies to reducing the image size by half, making its area one-quarter of the original; and the final "1" refers to the size of the output image after upsampling, which remains consistent with the input image. Similarly, the first structure is named "111," meaning that the image is neither downsampled nor upsampled, and that the size of each layer remains consistent.

[0015] Part Three: Training Set

[0016] The training set uses representative adversarial samples: FGSM and PGD;

[0017] Both types of adversarial examples are gradient-based attacks, directly using data generated during the model's lifecycle without processing. They are more representative of adversarial example attacks, meaning they better represent the feature distribution of adversarial example noise.

[0018] Part Four: Loss Function

[0019] After the training set is changed, if the model is trained first using type A adversarial examples and then using type B adversarial examples, the denoising performance of the model on type A adversarial examples will decrease after each use of type B training. This phenomenon is similar to the human forgetting mechanism. To avoid this forgetting phenomenon, a new loss function is proposed:

[0020] l old =l mse (D(x adv ), x)+lmse (D(x), x)

[0021]

[0022] Where D represents our denoising network, l mse This represents the quadratic crossover loss, where x represents the original sample. adv1 x adv2 These are two types of adversarial samples.

[0023] As a preferred embodiment of the present invention, both the residual block and the convolutional layer in the model structure are Conv2d layers with a stride of 1.

[0024] As a preferred embodiment of the present invention, the deconvolution layer in the model structure is a ConvTranspose2d layer with a stride of 2.

[0025] As a preferred embodiment of the present invention, in the model structure, the activation function of the last convolutional layer in the first structure and the last deconvolutional layer in the second structure is Sigmoid, and the activation function of the remaining convolutional layers and deconvolutional layers is Leaky Rule.

[0026] The beneficial effects of this invention are as follows: This invention reconstructs the image at its original size by adding multiple residual blocks (a residual network unit) to the original autoencoder (DAE) and using a convolutional neural network layer with a stride of 1 to simultaneously replace the original downsampling and upsampling operations. Finally, to enable the denoising network to learn the feature distribution of noise in adversarial samples from both adversarial and clean samples, thereby eliminating adversarial noise, this technical solution designs new training details for the denoising network. Specifically, it uses two types of highly aggressive adversarial attacks based on gradients as training inputs and provides a new loss function, thus possessing a multi-scale network structure more suited to the denoising task. With the help of a high-performance training set and loss function, it achieves higher denoising performance. Attached Figure Description

[0027] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0028] Figure 1 This is a schematic diagram of the residual block of the present invention;

[0029] Figure 2 This is a schematic diagram of the model structure of the present invention;

[0030] Figure 3 This is an example image illustrating the repair of adversarial sample noise in the CIFAR-10 dataset using the present invention. Detailed Implementation

[0031] Example 1

[0032] This invention discloses a multi-scale deep residual denoising network for eliminating adversarial noise in neural networks. The technical solution adopted includes four parts: residual network, model structure, training set, and loss function.

[0033] Residual Networks

[0034] In recent years, residual networks have demonstrated superior performance in computer vision problems from low to high levels (see Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 136–144). Existing technology (see Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778) has proposed ResNet, an excellent image recognition network architecture. By adding residual blocks to the network to increase the number of parameters, the entire model gains better learning ability and higher image recognition accuracy. In the field of image super-resolution, when the number of feature maps in a super-resolution network model increases to a certain extent, the numerical values ​​during training become unstable (see Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi).

[0035] In the presentation "Inception-v4, Inception-resnet and the impact of residual connection-on-learning," presented at the Thirty-first AAAI conference on artificial intelligence (2017), consistent with the solutions in this field, this technical solution addresses this issue by using a residual scaling factor of 0.1. The network structure of the residual block is as follows: Figure 1 As shown.

[0036] In the residual block network structure, the "convolutional layer" is a Conv2d layer with a stride of 1. The "scaling layer" has a scaling factor of 0.1.

[0037] Model Structure

[0038] The idea behind this model is based on a denoising automatic encoder (DAE) (see Shixiang Gu and Luca Rigazio, “Towards deep neutral network architectures robust to adversarial examples,” arXiv preprint arXiv:1412.5068, 2014). A DAE is a variant of an automatic encoder where the input is a noisy image and the output is a clean image. The DAE encoder learns to map a high-dimensional noisy image to a low-dimensional latent representation corresponding to a clean image, while the DAE decoder maps it back to the clean image. In the field of pattern recognition, network models with multi-scale image sizes have shown superior performance (see Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 136–144). Therefore, this technical solution designs a new network structure module to distinguish it from the DAE encoder-decoder structure that downsamples first and then upsamples.

[0039] After synthesizing all the ideas, a new network structure was proposed, such as... Figure 2 As shown.

[0040] In the model structure, "Convolutional Layer" refers to a Conv2d layer with a stride of 1. "Deconvolutional Layer" refers to a ConvTranspose2d layer with a stride of 2. "(C)" indicates the number of channels in the dataset. "Residual Layer" refers to all network layers in the residual block mentioned in the previous section. Additionally, each convolutional and deconvolutional layer is followed by an activation layer. The activation functions of the activation layers after the fourth convolutional layer and the second deconvolutional layer are Sigmoid, while the others use the Leaky Rule.

[0041] exist Figure 2 It can be observed that the width of some convolutional layers becomes narrower, which means that the size of the image data passing through this layer is also reduced. Figure 2 On the right is the traditional DAE model structure, called the "1q1" model structure. Here, the first "1" refers to the initial size of the input image; the middle "q" refers to the hidden layer data after the image data is downsampled within the model, which simplifies to reducing the image size by half, making its area a quarter of the original; the final "1" refers to the size of the upsampled output image, which remains consistent with the input image. Similarly, ... Figure 2 The model structure on the left is named "111," meaning that the images are neither downsampled nor upsampled, and that the image size remains constant across each layer. Final experiments demonstrate that these two different structures perform differently against various adversarial attacks, meaning they have different advantages. This model structure effectively integrates their strengths.

[0042] training set

[0043] In the training dataset used for the denoising network, existing techniques use images with different levels of Gaussian noise (see Dvij Kalaria, Aritra Hazra, and Partha Pratim Chakrabarti, “Towards adversarial purification using denoising autoencoders,” arXiv preprint arXiv:2208.13838, 2022). Existing techniques also use a single type of adversarial sample (see Guoqing Jin, Shiwei Shen, Dongming Zhang, Feng Dai, and Yongdong Zhang, “Ape-gan: Adversarial perturbation elimination with gan,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 3842–3846). To enable the model to learn the feature distribution of adversarial sample noise from multiple adversarial samples, as well as the denoising knowledge from adversarial samples to the original samples, this technical solution uses two highly aggressive and representative adversarial samples: FGSM and PGD. They were chosen because both are gradient-based attacks, directly using data generated during the model's lifecycle without processing, and are more representative of adversarial sample attacks, thus better representing the feature distribution of adversarial sample noise.

[0044] loss function

[0045] After the training set is changed, if the model is trained first using type A adversarial examples and then using type B adversarial examples, the denoising effect of the model on type A adversarial examples will decrease after each use of type B training. This phenomenon is similar to the human forgetting mechanism. To avoid this forgetting phenomenon, this technical solution proposes a new loss function instead of the old loss function that only considers a single adversarial example:

[0046] l old =l mse (D(x adv ), x)+l mse (D(x), x)

[0047]

[0048] Where D represents our denoising network, lmse This represents the quadratic crossover loss, where x represents the original sample. adv1 x adv2 These represent type A adversarial samples and type B adversarial samples, respectively.

[0049] experiment

[0050] The performance of the above denoising network was tested.

[0051] The datasets used in this experiment include MNIST (see Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998), a database of handwritten digits with 70,000 28x28 grayscale images in 10 categories (digits 0-9); CIFAR-10 (see Alex Krizhevsky, Geoffrey Hinton, et al., “Learning multiple layers of features from tiny images,” 2009), a dataset of 60,000 32x32 color images in 10 categories; and CIFAR-100 (see Alex Krizhevsky, Geoffrey Hinton, et al., “Learning multiple layers of features from tiny images,”

[0052] (2009), an image recognition dataset with the same image size and number as CIFAR-10, comprising 100 categories. The experimental environment consisted of an i7-9700k, 32GB RAM, and a GTX2080Ti-11G graphics card. The coding language was PyTorch. The optimization algorithm was Adam.

[0053] Experiment 1: Adding a Residual Network

[0054] Data type: Cifar10, Adversarial sample type: FGSM (attack parameter 0.1), prediction accuracy for adversarial samples: 17.00%, prediction accuracy for clean samples: 94.98%, number of training epochs: 100, "h": initial denoising network model, "h'": a residual block. Test results are as follows:

[0055] With the same number of training epochs, the denoising network achieves better restoration results as the number of residual blocks increases. Furthermore, increasing the number of channels in the residual blocks also improves the restoration effect, although it requires more training time. Therefore, using residual blocks, and using more residual blocks, will improve the performance of the denoising network in eliminating adversarial sample noise to a certain extent.

[0056] Experiment 2 uses a new training set and loss function

[0057] Data type: Cifar10, Number of training epochs: 100, Model structure: 111. "Experience Group": 95 PGD runs + 2 FGSM runs. "Cross Group": Starting with PGD adversarial examples, each PGD run is followed by one FGSM run, for a total of 100 runs. "Hybrid Group": Training using PGD and FGSM in the same epoch. "No. X": The epoch where the denoising network performs optimally. The experimental results of the network's repair effect before and after using the new training dataset and loss function are shown in the table below:

[0058]

[0059] Compared to the experience group, the cross-group denoising network performed better under PGD attacks, but its performance significantly declined under CW attacks. However, after modifying the training set and loss function, the proposed method, the hybrid group, effectively preserved the denoising performance for FGSM and PGD adversarial samples while slightly increasing the denoising network's performance for the other three types of adversarial samples. This indicates that it is a superior and more reasonable training method.

[0060] Experiment 3 constructed a multi-scale structure. Data type: CIFAR10, number of training epochs: 100, training set: PGD+FGSM, training method: hybrid training, "No.X": the epoch X in which the denoising network performs best. "Parallel": Both parts 111 and 1q1 of the component have the same input, and the final result is the average of the outputs of the two parts. "Consecutive": First, 111 receives the input, then 1q1 receives the output of 111, and the final result is the output of 1q1. The experimental results under different denoising network model structures are shown in the table below:

[0061]

[0062] The standalone 1q1 denoising model structure performs well against gradient-based FGSM and PGD attacks, but its performance against optimization-based CW and DeepFool attacks is inferior. To better leverage the strengths of these two different model structures, a suitable combination is highly beneficial. One approach is to connect them in parallel, averaging the outputs of both components. However, experimental results show that the new denoising model performs even worse against CW and DeepFool attacks than the standalone 1q1 model, making this an unsuitable combination method. Instead, concatenating the two components results in a new denoising network structure that combines the characteristics of both components, achieving the highest adversarial sample noise denoising performance.

[0063] Experiment 4 Final Denoising Performance

[0064] After fully combining the denoising networks, a deep residual denoising neural network is obtained to eliminate adversarial sample noise. Taking PGD attacks under CIFAR-10 as an example, it can... Figure 3 The study found that the denoising network can restore images with adversarial noise to a level very close to the original clean image. Furthermore, compared to a current adaptive denoising network also designed to eliminate adversarial sample noise (see Dvij Kalaria, Aritra Hazra, and Partha Pratim Chakrabarti, “Towards adversarial purification using denoising autoencoders,” arXiv preprint arXiv:2208.13838,2022), the denoising network exhibits better restoration performance on the same CIFAR-10 dataset.

[0065] Performance experiments of the denoising network on different datasets. Gaussian noise, FGSM, and PGD attack strength: attack parameters ∈ 0.3 on the MNIST dataset, and ∈ 0.1 on the CIFAR-10 and CIFAR-100 datasets. CW and DeepFool attack modes: l2. Number of residual blocks in the denoising network: 4 on MNIST, and 16 on CIFAR-10 and CIFAR-100. Experimental results are as follows:

[0066]

[0067] If we divide all the data in the table above by the data in the clean samples of the same dataset, that is, we take the prediction accuracy of the classifier on the clean samples as the benchmark, all the results of the performance experiment of the denoising network in repairing adversarial samples on different datasets are calculated based on the data of the performance experiment of the denoising network on different datasets. The calculation method is to divide each result in each dataset in the table by the data in the corresponding row where the "Clean" column is "NoDefence". The results are as follows:

[0068]

[0069] The above results better demonstrate the superior performance of this denoising network.

[0070] Components not described in detail in this article are existing technologies.

[0071] While the specific embodiments of the present invention have been described in detail above, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention, and modifications or variations without creative effort are still within the protection scope of the present invention.

Claims

1. A multi-scale deep residual denoising method for eliminating adversarial noise in neural networks, characterized in that: Establish a multi-scale deep residual denoising network, including the residual network, model structure, training set and loss function; Part 1: Residual Networks The residual blocks added to the residual network include convolutional layers, ReLU activation layers, and scaling layers, arranged in the following order: convolutional layer, ReLU activation layer, convolutional layer, scaling layer, where the scaling factor of the scaling layer is 0.

1. Part Two: Model Structure The model has two structures. The first structure starts from the input and consists of a convolutional layer, a residual layer, another convolutional layer, another convolutional layer, and another convolutional layer. Below the first convolutional layer, there are two channels. One channel connects to the residual layer to process the data, while the other channel bypasses the residual layer and does not process the data. The results of the two channels are then added below the residual layer. After obtaining the intermediate value through this structure, the model is connected to the second structure, which is a traditional DAE structure. The second structure consists of a convolutional layer, a residual layer, a convolutional layer, a deconvolutional layer, and another deconvolutional layer, and then connects to the output. The residual layers are all the network layers in the residual block of the first part; each convolutional layer and deconvolutional layer is followed by an activation layer; Part Three: Training Set The training set uses representative adversarial samples: FGSM and PGD; Part Four: Loss Function The loss function is: ； ； ； Where D represents the denoising network, l mse This represents the quadratic crossover loss, where x represents the original sample. adv1 x adv2 These are two types of adversarial samples; By adding multiple residual blocks to the original autoencoder DAE, each residual block is a residual network unit. A convolutional neural network layer with a stride of 1 is used to simultaneously replace the original downsampling and upsampling operations, thereby reconstructing the image at the original image size. Finally, in order for the denoising network to learn the feature distribution of noise in adversarial samples from both adversarial and clean samples, and thus eliminate adversarial noise, new training details were designed for the denoising network. Specifically, two types of strong adversarial attacks based on gradients were used as training inputs, and a new loss function was provided, resulting in a multi-scale network structure that is more suitable for the denoising task. With the help of a high-performance training set and loss function, higher denoising performance was achieved.

2. The multi-scale deep residual denoising method for eliminating adversarial noise in neural networks according to claim 1, characterized in that: Both the residual block and the convolutional layer in the model structure are Conv2d layers with a stride of 1.

3. The multi-scale deep residual denoising method for eliminating adversarial noise in neural networks according to claim 1, characterized in that: In the model structure, the deconvolution layer is a ConvTranspose2d layer with a stride of 2.

4. The multi-scale deep residual denoising method for eliminating adversarial noise in neural networks according to claim 1, characterized in that: In the model structure, the activation function of the last convolutional layer in the first structure and the last deconvolutional layer in the second structure is Sigmoid, while the activation function of the remaining convolutional and deconvolutional layers is LeakyRule.

Images