An adversarial sample visual explanation method based on class activation mapping

Abstract

The application discloses an adversarial sample visual explanation method based on a class activation map, and comprises an adversarial sample generation stage, an adversarial sample corresponding to a normal sample is generated by adding disturbance to the normal sample through an adversarial attack algorithm; a feature extraction stage, an adversarial sample feature and a normal sample feature are obtained by inputting an adversarial sample picture and a normal sample picture into a trained deep learning model for feature extraction, and sample classification is performed on the adversarial sample and the normal sample; a feature saliency map generation stage, a gradient of a classification result of the adversarial sample to the adversarial sample feature and a gradient of a classification result of the normal sample to the normal sample feature are calculated; and the gradient difference obtained by calculating the normal sample and the adversarial sample is taken as a weight, and linearly weighted fusion is performed on the adversarial sample feature to obtain a final adversarial sample feature saliency map.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the field of deep learning, and particularly relates to a class activation map-based visual explanation method for adversarial samples. BACKGROUND

[0002] In recent years, deep learning has made remarkable progress in the fields of computer vision, natural language processing, speech signal recognition, etc. However, with the deepening of the research on deep learning algorithms, deep neural networks have shown a relatively universal vulnerability. In 2014, Szegedy et al. [1] found that by adding some perturbations that cannot be recognized by human eyes to normal samples, deep learning models can be effectively deceived. Such samples added with perturbations are called adversarial samples, and the process of generating adversarial samples is called adversarial attack.

[0003] There have been a large number of related works on adversarial attack algorithms for deep learning, among which the gradient-based attack method is the most popular attack method. Specifically, this attack method mainly propagates the loss function to the input image in reverse, and modifies the image according to the corresponding gradient to make it an adversarial sample. The most typical method is the FGSM (Fast Gradient Sign Method) algorithm proposed by Goodfellow et al. [2] , which disturbs the image by maximizing the loss of the classifier on the image. The efficiency of the FGSM algorithm lies in that it only needs one-step calculation to generate adversarial samples. Kurakin et al. [3] proposed an extension method BIM (Basic Iterative Method) of FGSM. Unlike FGSM, which increases the loss function of the classifier by a large step operation, BIM iteratively perturbs the image by multiple small steps and adjusts the direction after each step to make the perturbation more accurate. After the BIM algorithm, Madry et al. [4] introduced a variant of BIM called PGD (Projected Gradient Descent) attack. The PGD algorithm uses a random starting projected gradient descent method to perturb the input image multiple times according to the sign direction of the gradient with a certain step size to make up for the shortcomings of single-step gradient descent.

[0004] The emergence of this series of adversarial attack algorithms poses an incalculable threat to the security of deep learning models, making their application in high-security scenarios face great challenges, so it is necessary to carry out in-depth research on the explainability of adversarial attack algorithms for deep neural networks. By explaining and analyzing the reasons why adversarial samples are misclassified by the model, the weaknesses of the deep learning model can be found, and a more robust deep learning model can be built to resist adversarial attacks. Therefore, it is particularly important to study and analyze the explainability of adversarial samples.

[0005] Studying the interpretability of adversarial examples first requires investigating the interpretability of deep neural networks. In recent years, researchers have attempted to improve the interpretability of deep neural networks from multiple perspectives, primarily categorized into methods for visualizing the internal workings of the model and methods for visualizing the model's results. Methods for visualizing the internal workings of the model mainly involve visualizing internal information during its operation. A representative approach is that of Liu et al. [5] The proposed model visualization analysis method visually represents the relationships between different neurons in a deep learning model and introduces two algorithms to visualize the derivation relationships of neurons in the network model. The model result visualization interpretation method mainly focuses on locating the regions of interest in the model. The mainstream approach in this direction is the Class Activation Mapping (CAM) visualization interpretation method. [6] (Class Activation Mapping) is based on the idea of ​​using the output of convolutional layers, i.e., feature maps, as the raw information for obtaining the interpretation result. Weights are calculated in some way, and the feature maps are then weighted and summed to obtain the final interpretation result. However, this method lacks flexibility and cannot be applied to all models. Based on the idea of ​​CAM, Selvaraju, Chattopadhay, and others proposed Grad-CAM. [7] and Grad-CAM++ [8] It uses the gradient of the predicted score of a specific category with respect to the feature layer to calculate the weights, which is more generalizable than CAM and can be used for different model structures and tasks.

[0006] Currently, model visualization and interpretation methods mainly perform interpretive analysis on normal samples. However, these methods are sensitive to the robustness of the model and cannot locate sample feature regions that can be interpreted by humans. They cannot discover the weaknesses of deep learning models, and the deep learning models they construct are not stable enough.

[0007] References:

[0008] [1] C. Szegedy, W. Zaremba, I. Sutskever, et al. Intriguing properties of neural networks [C]. Proceedings of the 2nd International Conference on Learning Representations (ICLR), 2014.

[0009] [2] I. Goodfellow, J. Shlens, C. Szegedy. Explaining and harnessing adversarial examples [C]. International Conference on Learning Representations (ICLR), 2015.

[0010] [3] A. Kurakin, I. Goodfellow, S. Bengio. Adversarial examples in the physical world [C]. International Conference on Learning Representations (ICLR), 2016.

[0011] [4] A. Madry, A. Makelov, L. Schmidt, et al. Towards deep learning models resistant to adversarial attacks [C]. International Conference on Learning Representations (ICLR), 2018.

[0012] [5] M. Liu, J. Shi, Z. Li, et al. Towards better analysis of deep convolutional neural networks [J]. IEEE Transactions on Visualization & Computer Graphics, 2016.

[0013] [6] B. Zhou, A. Khosla, A. Lapedriza, et al. Learning deep features for discriminative localization [C]. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2016.

[0014] [7] R. Selvaraju, M. Cogswell, A. Das, et al. Grad-cam: Visual explanations from deep networks via gradient-based localization [C]. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2017.

[0015] [8] A. Chattopadhay, A. Sarkar, P. Howlader, et al. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks [C]. IEEE Winter Conference on Applications of Computer Vision (CVPR), 2018. SUMMARY

[0016] The purpose of the present application is to solve the problem that it is difficult to reasonably explain the misclassification of adversarial samples by a deep learning model, and provide an adversarial sample visual explanation method based on class activation mapping, which generates a feature saliency map for the adversarial sample, locates the picture area where the adversarial sample focuses on the features, and realizes the explanation of the behavior of the adversarial sample misclassified by the deep learning model.

[0017] The purpose of the present application is realized by the following technical solutions:

[0018] An adversarial sample visual explanation method based on class activation mapping, comprising an adversarial sample generation stage, a feature extraction stage and a feature saliency map generation stage, specifically as follows:

[0019] The adversarial sample generation stage; the corresponding adversarial sample is generated by adding disturbance to the normal sample through the adversarial attack algorithm;

[0020] The feature extraction stage; the adversarial sample and the normal sample are input into the trained deep learning model for feature extraction to obtain the adversarial sample feature and the normal sample feature;

[0021] The feature saliency map generation stage; the classification results of the adversarial sample and the normal sample are obtained by classifying the adversarial sample feature and the normal sample feature, and the gradient of the classification result of the adversarial sample to the adversarial sample feature and the gradient of the classification result of the normal sample to the normal sample feature are calculated; the gradient difference calculated for the normal sample and the adversarial sample is taken as a weight, and the adversarial sample feature is linearly weighted and fused to obtain the final adversarial sample feature saliency map.

[0022] Further, in the adversarial sample generation stage, an optimization-based attack method is used to generate an adversarial sample. In the optimization process, the adversarial sample is represented by an optimized parameter. The optimization target is to make the distance between the adversarial sample and the normal sample smaller, and to make the error class confidence predicted by the deep learning model higher.

[0023] Further, in the feature extraction stage, a feature extraction network is used. The specific sample feature extraction steps are as follows:

[0024] (1) The dimension of the adversarial sample is [1, 3, 224, 224], and the adversarial sample is input into the feature extraction network;

[0025] (2) The adversarial sample is input into a convolutional layer with an output channel of 64, a convolution kernel of 7x7, a step of 2, and a padding of 3 for feature extraction, and the obtained feature has a dimension of [1, 64, 112, 112]; then the feature with a dimension of [1, 64, 112, 112] is input into a maximum pooling layer with a convolution kernel of 3x3, a step of 2, and a padding of 1 for further feature extraction, and the obtained feature has a dimension of [1, 64, 56, 56];

[0026] (3) The feature obtained in step (2) is input into the first residual convolution block of the feature extraction network; the first residual convolution block includes four convolutional layers with an output channel of 64, a convolution kernel of 3x3, a step of 1, and a padding of 1; the feature processed by the second convolutional layer is added and fused with the feature input into the first residual convolution block, and then input into the third convolutional layer; the feature processed by the last convolutional layer is added and fused with the feature output by the second convolutional layer, and the feature obtained by the first residual convolution block has a dimension of [1, 64, 56, 56];

[0027] (4) input the feature finally obtained in step (3) into a second residual convolutional block of the feature extraction network; the second residual convolutional block comprises four convolutional layers with an output channel of 128, a convolution kernel of 3x3 and padding of 1; wherein the step length of the first convolutional layer is 2 and the step length of the remaining convolutional layers is 1; the feature input into the second residual convolutional block is first processed by a convolutional layer with an output channel of 128, a convolution kernel of 1x1, a step length of 2 and padding of 0, and then added and fused with the feature processed by the second convolutional layer to be input into the next convolutional layer; the feature processed by the last convolutional layer is added and fused with the feature output by the second convolutional layer, and the dimension of the finally obtained feature is [1, 128, 28, 28];

[0028] (5) input the feature finally obtained in step (4) into a third residual convolutional block of the feature extraction network; the third residual convolutional block comprises four convolutional layers with an output channel of 256, a convolution kernel of 3x3 and padding of 1; wherein the step length of the first convolutional layer is 2 and the step length of the remaining convolutional layers is 1; the feature input into the third residual convolutional block is first processed by a convolutional layer with an output channel of 256, a convolution kernel of 1x1, a step length of 2 and padding of 0, and then added and fused with the feature processed by the second convolutional layer to be input into the next convolutional layer; the feature processed by the last convolutional layer is added and fused with the feature output by the second convolutional layer, and the dimension of the finally obtained feature is [1, 256, 14, 14];

[0029] (6) input the feature finally obtained in step (5) into a last residual convolutional block of the feature extraction network; the last residual convolutional block comprises four convolutional layers with an output channel of 512, a convolution kernel of 3x3 and padding of 1; wherein the step length of the first convolutional layer is 2 and the step length of the remaining convolutional layers is 1; the feature input into the last residual convolutional block is first processed by a convolutional layer with an output channel of 512, a convolution kernel of 1x1, a step length of 2 and padding of 0, and then added and fused with the feature processed by the second convolutional layer to be input into the next convolutional layer; the feature processed by the last convolutional layer is added and fused with the feature output by the second convolutional layer, and the dimension of the finally obtained feature is [1, 512, 7, 7];

[0030] (7) take the feature finally obtained in step (6) as the extracted adversarial sample feature; correspondingly, input the normal sample into the feature extraction network, repeat steps (2) to (6) to obtain the normal sample feature.

[0031] Further, in the feature saliency map generation stage, the obtained sample features are sent to a Softmax classification layer for classification to obtain a classification result of the adversarial sample and a classification result of the normal sample; the classification result of the adversarial sample is defined as C, and the classification result of the normal sample is defined as C'; first, the gradient of the classification result C of the adversarial sample with respect to the features A of each channel dimension in the last convolutional layer of the deep learning model is calculated by using a back propagation mechanism of the deep learning model, and the gradient is average-pooled to obtain the gradient of each channel dimension of the adversarial sample k

[0032]

[0033] wherein H is the matrix height of the sample features in the channel dimension, W is the matrix width of the sample features in the channel dimension, y C is the score of the corresponding category C, represents the value at the (i,j) position in the kth channel of the sample features, and k is the number of channels of the sample features;

[0034] secondly, the same method is used to replace y C with y C′ to calculate the gradient of each normal sample wherein the gradient of the normal sample represents the weight of the normal sample features in the channel dimension, and the gradient of the adversarial sample represents the weight of the adversarial sample features in the channel dimension;

[0035] finally, the difference between the gradients calculated for the normal sample and the adversarial sample is taken as the weight to perform weighted summation on the adversarial sample features, the negative values are removed, and then upsampling is performed to obtain the feature saliency map for the adversarial sample

[0036]

[0037] ReLU represents an activation function, and Up represents an upsampling function.

[0038] The application further provides an adversarial sample visual explanation device based on class activation mapping, comprising:

[0039] an adversarial sample generation unit configured to generate an adversarial sample;

[0040] a feature extraction unit configured to extract sample features;

[0041] a feature saliency map generation unit configured to generate an adversarial sample feature saliency map.

[0042] ​The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the class activation mapping-based visual explanation method of an adversarial sample when executing the program.

[0043] The application further provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the class activation mapping-based visual explanation method of an adversarial sample when executed by a processor.

[0044] Compared with the prior art, the technical scheme of the application has the beneficial effects that:

[0045] The method can visually explain and analyze the reason why an adversarial sample is misclassified by a deep learning model. Existing visual explanation methods mainly perform visual explanation on normal samples, and generally highlight the picture area in an input picture that can respond to the picture true class feature in the form of a heat map. The visual explanation method proposed in the application performs visual explanation on an adversarial sample, and the method not only utilizes gradient information of a normal sample but also introduces gradient information of an adversarial sample, and the difference between the two is used as a weight to perform weighted fusion on the adversarial sample feature, so that a feature saliency map for the adversarial sample can be obtained, and the area related to the adversarial sample class feature can be located, thereby explaining the reason why the adversarial sample is classified into the class. By analyzing the reason why an adversarial sample is misclassified by a deep learning model, the weakness of the deep learning model can be found out, thereby providing support for constructing a robust deep learning model. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The overall flowchart for generating the feature saliency map for the adversarial sample of the application.

[0047] Figure 2 The feature extraction network structure.

[0048] Figure 3 The visual result comparison schematic diagram. DETAILED DESCRIPTION

[0049] The application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0050] The embodiment provides a class activation mapping-based visual explanation method of an adversarial sample, which includes an adversarial sample generation stage, a feature extraction stage, and a feature saliency map generation stage, and the specific steps are as follows:

[0051] I. Adversarial sample generation stage

[0052] The corresponding adversarial sample picture is generated by adding disturbance to the normal sample picture through the adversarial attack algorithm. In order to carry out targeted adversarial attack, while realizing the two goals of high attack accuracy and low adversarial disturbance, an optimization-based attack method is used to generate adversarial samples in this stage. In the optimization process, the adversarial sample is represented by the optimized parameter, and the optimization target is to make the distance d n between the adversarial sample and the normal sample smaller, and the confidence of the wrong class predicted by the deep learning model higher. The calculation formula of the distance d n between the adversarial sample and the normal sample is as follows:

[0053]

[0054] Where ω n is the optimization parameter of the adversarial disturbance, X n is the normal sample. Here the optimization parameter ω n is transformed by tanh space mapping, which can ensure that the adversarial sample will not exceed the range of pixel points. The calculation formula of the optimization target function is as follows:

[0055]

[0056] Where the function f(·) is defined as follows:

[0057] f(x′)=max(max{Z(x′) i :i≠t}-Z(x′) t ,-p)

[0058] Where Z(x′) t is the tth class value of the output result of the deep learning model after the adversarial sample is input into the model, p is the model confidence, and μ is the hyperparameter to balance the relationship between the two losses. By continuously optimizing, max{Z(x′) i :i≠t}-Z(x′) t becomes smaller, the model confidence p becomes higher, and finally an optimized adversarial disturbance ω n is obtained. Finally, the optimized adversarial disturbance is added to the normal sample to obtain the adversarial sample.

[0059] II. Feature extraction stage

[0060] The adversarial sample generated in the adversarial sample generation stage and the normal sample are input into the trained deep learning model for feature extraction to obtain the adversarial sample feature and the normal sample feature. The specific sample feature extraction steps are as follows:

[0061] (1) The dimension of the adversarial sample is [1, 3, 224, 224], and the adversarial sample is input into the feature extraction network.

[0062] (2) The anti-sample is first input to a convolutional layer with 64 output channels, a convolution kernel of 7x7, a step size of 2, and padding of 3 for feature extraction, resulting in a feature dimension of [1, 64, 112, 112]. Then the feature is input to a max pooling layer with a convolution kernel of 3x3, a step size of 2, and padding of 1 for further feature extraction, resulting in a feature dimension of [1, 64, 56, 56].

[0063] (3) The final feature obtained in step (2) is input to the first residual convolutional block of the feature extraction network. The residual convolutional block includes four convolutional layers with 64 output channels, a convolution kernel of 3x3, a step size of 1, and padding of 1. The feature processed by the second convolutional layer is added to the input feature of the residual convolutional block, and then input to the third convolutional layer. The feature processed by the last convolutional layer is added to the output feature of the second convolutional layer, and the final feature dimension is [1, 64, 56, 56].

[0064] (4) The final feature obtained in step (3) is input to the second residual convolutional block of the feature extraction network. The residual convolutional block includes four convolutional layers with 128 output channels, a convolution kernel of 3x3, and padding of 1. In order to perform down-sampling processing on the feature, the step size of the first convolutional layer is 2, and the step size of the remaining convolutional layers is 1. The input feature of the residual convolutional block is processed by a convolutional layer with 128 output channels, a convolution kernel of 1x1, a step size of 2, and padding of 0, and then added to the feature processed by the second convolutional layer, and input to the next convolutional layer. The feature processed by the last convolutional layer is added to the output feature of the second convolutional layer, and the final feature dimension is [1, 128, 28, 28].

[0065] (5) The final feature obtained in step (4) is input to the third residual convolutional block of the feature extraction network. The residual convolutional block includes four convolutional layers with 256 output channels, a convolution kernel of 3x3, and padding of 1. In order to perform down-sampling processing on the feature, the step size of the first convolutional layer is 2, and the step size of the remaining convolutional layers is 1. The input feature of the residual convolutional block is processed by a convolutional layer with 256 output channels, a convolution kernel of 1x1, a step size of 2, and padding of 0, and then added to the feature processed by the second convolutional layer, and input to the next convolutional layer. The feature processed by the last convolutional layer is added to the output feature of the second convolutional layer, and the final feature dimension is [1, 256, 14, 14].

[0066] (6) The final feature obtained in step (5) is input into the last residual convolution block of the feature extraction network. The residual convolution block includes four convolution layers with 512 output channels, a 3x3 convolution kernel, and a padding of 1. In order to down-sample the features, the step size of the first convolution layer is 2, and the step size of the remaining convolution layers is 1. The features input into the residual convolution block are processed by a convolution layer with 512 output channels, a 1x1 convolution kernel, a step size of 2, and a padding of 0, then added and fused with the features processed by the second convolution layer, and input into the next convolution layer. The features processed by the last convolution layer are added and fused with the features output by the second convolution layer, and the final feature has a dimension of [1, 512, 7, 7].

[0067] (7) The final feature obtained in step (6) is taken as the extracted adversarial sample feature. Correspondingly, the normal sample is input into the feature extraction network, and steps (2) to (6) are repeated to obtain the normal sample feature.

[0068] III. Feature saliency map generation stage

[0069] The obtained sample feature is sent to the Softmax classification layer for classification. The classification result of the adversarial sample is defined as C, and the classification result of the normal sample is defined as C'. First, the gradient of the classification result C of the adversarial sample with respect to the feature A k in each channel dimension of the last convolution layer is calculated using the back propagation mechanism of the deep learning model, and the gradient is averaged-pooled to obtain the gradient

[0070]

[0071] where H is the matrix height of the sample feature in the channel dimension, W is the matrix width of the sample feature in the channel dimension, y C is the score corresponding to the category C, represents the value at position (i, j) in the kth channel of the sample feature, and k is the number of channels of the sample feature.

[0072] Secondly, using the same method, y C is replaced by y C′ to calculate the gradient of each normal sample, where the gradient of the normal sample represents the weight of the normal sample feature in the channel dimension, and the gradient of the adversarial sample represents the weight of the adversarial sample feature in the channel dimension.

[0073] Finally, the difference between the gradients calculated for the normal sample and the adversarial sample is taken as the weight to perform weighted summation on the adversarial sample feature, then the negative values are removed, and then up-sampling is performed to obtain the feature saliency map

[0074]

[0075] Figure 3 For the visualization results of the method of the present application, the first column is the normal sample, the second column is the class activation map of the normal sample, the third column is the target class of the adversarial attack, the fourth column is the class activation map of the target class of the adversarial attack, and the fifth column is the feature saliency map obtained by using the adversarial sample visualization explanation method based on the class activation map proposed in the present application. From Figure 3 It can be seen that the feature saliency map obtained by the method proposed in the present application successfully locates the region related to the feature of the target class, and can explain the misclassification of the adversarial sample by the model.

[0076] The embodiment of the present application also provides a specific implementation of an electronic device capable of implementing all steps in the adversarial sample visualization explanation method based on the class activation map in the above-mentioned embodiments, and the electronic device specifically includes the following contents:

[0077] a processor, a memory, a communications interface, and a bus;

[0078] The processor, the memory, and the communications interface complete mutual communication through the bus; the communications interface is used for realizing information transmission between related devices such as a server-side device, a metering device, and a user-side device.

[0079] The processor is used for calling a computer program in the memory, and the processor realizes all steps in the adversarial sample visualization explanation method based on the class activation map in the above-mentioned embodiments when executing the computer program,

[0080] The embodiment of the present application also provides a computer readable storage medium capable of implementing all steps in the adversarial sample visualization explanation method based on the class activation map in the above-mentioned embodiments, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize all steps in the adversarial sample visualization explanation method based on the class activation map in the above-mentioned embodiments.

[0081] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of being implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0082] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart and / or block diagram in the variations disclosed herein illustrate the architecture, functionality, and operation of possible implementations of apparatuses and computer program products according to various embodiments. In this regard, each flowchart block and / or block in the flowcharts and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that, in some alternative implementations, the flowchart block and / or block diagrams can represent a Figure 1 The apparatus can be a special purpose or general purpose computer, microprocessor, microprocessor core, or other programmable data processing apparatus.

[0083] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart and / or block diagram in the variations disclosed herein illustrate the architecture, functionality, and operation of possible implementations of apparatuses and computer program products according to various embodiments. In this regard, each flowchart block and / or block in the flowcharts and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that, in some alternative implementations, the flowchart block and / or block diagrams can represent a Figure 1 The apparatus can be a special purpose or general purpose computer, microprocessor, microprocessor core, or other programmable data processing apparatus.

[0084] The application is not limited to the embodiments described above. The above description of specific embodiments is intended to describe and enable the technical solutions of the present application, and the specific embodiments described above are merely illustrative and are not restrictive. Those skilled in the art can make many specific modifications under the guidance of the present application without departing from the purpose of the present application and the scope of the claims, and these are all within the scope of protection of the present application.

Claims

1. A method for visualizing explanation of adversarial examples based on class activation mapping, characterized in that, The method comprises an adversarial sample generation stage, a feature extraction stage and a feature saliency map generation stage, and specifically as follows: The adversarial sample generation stage; through an adversarial attack algorithm, an adversarial sample corresponding to a normal sample is generated by adding disturbance to the normal sample; The feature extraction stage; the adversarial sample and the normal sample are input into a trained deep learning model for feature extraction to obtain an adversarial sample feature and a normal sample feature; The feature saliency map generation stage; the adversarial sample feature and the normal sample feature are classified to obtain a classification result of the adversarial sample and a classification result of the normal sample, and the gradient of the classification result of the adversarial sample to the adversarial sample feature and the gradient of the classification result of the normal sample to the normal sample feature are calculated; the gradient difference calculated for the normal sample and the adversarial sample is taken as a weight, and the adversarial sample feature is linearly weighted and fused to obtain a final adversarial sample feature saliency map. 2.The method of claim 1, wherein, In the adversarial sample generation stage, an optimization-based attack method is used to generate an adversarial sample. In the optimization process, the adversarial sample is represented by an optimized parameter. The optimization goal is to make the distance between the adversarial sample and the normal sample smaller, and to make the confidence of the wrong class predicted by the deep learning model higher. 3.The method of claim 1, wherein, In the feature extraction stage, a feature extraction network is used. The specific sample feature extraction steps are as follows: (1) the dimension of the adversarial sample is [1, 3, 224, 224], and the adversarial sample is input into the feature extraction network; (2) the adversarial sample is input into a convolutional layer with an output channel of 64, a convolution kernel of 7x7, a step of 2 and a padding of 3 for feature extraction, and the obtained feature has a dimension of [1, 64, 112, 112]; then the feature with a dimension of [1, 64, 112, 112] is input into a maximum pooling layer with a convolution kernel of 3x3, a step of 2 and a padding of 1 for further feature extraction, and the obtained feature has a dimension of [1, 64, 56, 56]; (3) the feature obtained in step (2) is input into the first residual convolution block of the feature extraction network; the first residual convolution block includes four convolutional layers with an output channel of 64, a convolution kernel of 3x3, a step of 1 and a padding of 1; the feature processed by the second convolutional layer is added and fused with the feature input into the first residual convolution block, and then input into the third convolutional layer; the feature processed by the last convolutional layer is added and fused with the feature output by the second convolutional layer, and the feature obtained by the first residual convolution block has a dimension of [1, 64, 56, 56]; (4) input the feature finally obtained in step (3) into a second residual convolutional block of the feature extraction network; the second residual convolutional block comprises four convolutional layers with an output channel of 128, a convolution kernel of 3*3 and padding of 1; wherein the step length of the first convolutional layer is 2, and the step length of the remaining convolutional layers is 1; the feature input into the second residual convolutional block is first processed by a convolutional layer with an output channel of 128, a convolution kernel of 1*1, a step length of 2 and padding of 0, and then added and fused with the feature processed by the second convolutional layer to be input into the next convolutional layer; the feature processed by the last convolutional layer is added and fused with the feature output by the second convolutional layer, and the finally obtained feature has a dimension of [1, 128, 28, 28]; (5) input the feature finally obtained in step (4) into a third residual convolutional block of the feature extraction network; the third residual convolutional block comprises four convolutional layers with an output channel of 256, a convolution kernel of 3*3 and padding of 1; wherein the step length of the first convolutional layer is 2, and the step length of the remaining convolutional layers is 1; the feature input into the third residual convolutional block is first processed by a convolutional layer with an output channel of 256, a convolution kernel of 1*1, a step length of 2 and padding of 0, and then added and fused with the feature processed by the second convolutional layer to be input into the next convolutional layer; the feature processed by the last convolutional layer is added and fused with the feature output by the second convolutional layer, and the finally obtained feature has a dimension of [1, 256, 14, 14]; (6) input the feature finally obtained in step (5) into a last residual convolutional block of the feature extraction network; the last residual convolutional block comprises four convolutional layers with an output channel of 512, a convolution kernel of 3*3 and padding of 1; wherein the step length of the first convolutional layer is 2, and the step length of the remaining convolutional layers is 1; the feature input into the last residual convolutional block is first processed by a convolutional layer with an output channel of 512, a convolution kernel of 1*1, a step length of 2 and padding of 0, and then added and fused with the feature processed by the second convolutional layer to be input into the next convolutional layer; the feature processed by the last convolutional layer is added and fused with the feature output by the second convolutional layer, and the finally obtained feature has a dimension of [1, 512, 7, 7]; (7) take the feature finally obtained in step (6) as the extracted adversarial sample feature; correspondingly, input the normal sample into the feature extraction network, repeat steps (2) to (6) to obtain the normal sample feature. 4.The method of claim 1, wherein, In the feature saliency map generation stage, the obtained sample feature is sent into a Softmax classification layer to obtain the classification result of the adversarial sample and the classification result of the normal sample; The classification result of the adversarial sample is defined as , and the classification result of the normal sample is ; first, the classification result of the adversarial sample is calculated by using the back propagation mechanism of the deep learning model The gradient of each channel dimension in the last convolution layer is calculated , and average pooling is performed on the gradient to obtain the gradient of each channel dimension of the adversarial sample : ； wherein, is a matrix height of the sample feature in the channel dimension, is a matrix width of the sample feature in the channel dimension, is a score of the corresponding class denotes a value at position in the channel of the sample feature, is a number of channels of the sample feature;​ Second, using the same method, we calculate the gradient of the adversarial sample The gradient of each normal sample is calculated ; wherein the gradient of the normal sample represents the weight of the normal sample feature in the channel dimension, and the gradient of the adversarial sample represents the weight of the adversarial sample feature in the channel dimension; Finally, the gradient difference calculated for normal samples and adversarial samples is used as a weight to sum the features of the adversarial samples, and the negative values are removed, and then up-sampling is performed to obtain the feature saliency map for the adversarial samples : ； denotes an activation function, denotes an up-sampling function.

5. An adversarial sample visual explanation device based on class activation mapping, characterized in that, comprise: an adversarial sample generation unit configured to generate a corresponding adversarial sample by adding perturbation to the normal sample through an adversarial attack algorithm; a feature extraction unit configured to input the adversarial sample and the normal sample into the trained deep learning model to extract features to obtain an adversarial sample feature and a normal sample feature; The feature saliency map generation unit is configured to classify the adversarial sample feature and the normal sample feature to obtain a classification result of the adversarial sample and a classification result of the normal sample, calculate a gradient of the classification result of the adversarial sample with respect to the adversarial sample feature and a gradient of the classification result of the normal sample with respect to the normal sample feature, and linearly weight and fuse the adversarial sample feature by taking the gradient difference calculated for the normal sample and the adversarial sample as a weight to obtain a final adversarial sample feature saliency map.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the class activation mapping-based adversarial sample visual explanation method of any one of claims 1 to 4 when executing the program.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the class activation mapping-based adversarial sample visual explanation method of any one of claims 1 to 4 when executed by the processor.

Images