Adversarial robustness evaluation method and terminal device based on class activation map
Through the evaluation method based on class activation map, the class activation map is generated and the indicators are calculated using SmoothGradCAM++ technology, which solves the problem of deep learning model's anti-robusiness evaluation, and achieves effective evaluation and explanatory enhancement of anti-robusiness.
Patent Information
- Application Number
- CN202211669056.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-23
- Publication Date
- 2025-08-22
- Estimated Expiration
- 2042-12-23
AI Technical Summary
In the prior art, the adversarial robustness evaluation method of deep learning models is not mature enough, difficult to explain and difficult to apply to all models, making it difficult to eliminate the security risks of adversarial attacks.
The evaluation method based on class activation map is adopted, and the adversarial samples are generated through the adversarial attack algorithm, the class activation map of the original sample and the adversarial samples are obtained, and the class activation map of the model is generated using SmoothGradCAM++ technology, and the adversarial robustness of the model is evaluated through the average high-heat area coincidence ratio and the average coverage coincidence ratio indicator.
It enhances the interpretability and accuracy of the adversarial robustness evaluation, and can evaluate the adversarial robustness of the model from the perspective of internal mechanism of the model, verify the effectiveness of the indicators, and improve the adversarial defense capabilities of the model.
Smart Images
Figure CN115880546B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an adversarial robustness evaluation method based on class activation maps for image classification and a terminal device. Background Art
[0002] In recent years, deep learning (DL) has developed rapidly and has been widely applied in numerous fields, including image classification, medical diagnosis, autonomous driving, virtual gaming, and smart agriculture. Although DL has achieved superior performance in various complex tasks, its security has received far less attention from the academic community. Szegedy et al. discovered that by adding subtle perturbations, imperceptible to humans, to samples, a trained deep neural network model can be tricked into making incorrect decisions. Other research has shown that this adversarial attack technique can also be applied in the real world. For example, a hacker can maliciously disrupt their facial image, causing a facial recognition system to match them to anyone other than the victim or hacker, thereby threatening the personal and financial safety of others. Adversarial attack techniques in deep learning are currently receiving significant attention, with a growing number of adversarial attack algorithms, primarily for image classification.
[0003] To mitigate the potential security risks posed by adversarial attacks to deep learning models, a growing number of researchers have begun researching how to effectively enhance models' ability to resist adversarial attacks (i.e., improve their adversarial robustness). A series of adversarial defense techniques, such as adversarial training, data processing, feature compression, and gradient masking, have been proposed. While these techniques can improve models' adversarial robustness to a certain extent, the current framework for evaluating adversarial robustness is not robust. Furthermore, many attack and defense algorithms are difficult to apply to all models due to experimental limitations. These difficulties pose significant challenges to adversarial robustness evaluation.
[0004] Effectively and rationally evaluating the adversarial robustness of DL models is crucial for building adversarially robust models. However, current methods for evaluating model adversarial robustness are relatively immature and can be broadly categorized into two types: adversarial evaluation and metric evaluation. The former proposes and improves adversarial attack and defense algorithms, repeatedly iterating through adversarial attacks to test the model's adversarial robustness. The latter, based on the perspective of adversarial examples, proposes a series of evaluation metrics, aiming to assess the model's adversarial robustness through comprehensive and reasonable metrics. Compared to the former, the latter offers the advantage of being able to measure the model's adversarial robustness in an objective and quantifiable manner.
[0005] Specifically, indicator evaluation methods can be broadly categorized into data-oriented and model-oriented metrics. Data-oriented metrics focus on the adequacy of model testing and the quality of test data. These metrics ensure the validity of the evaluation results and are often used to evaluate adversarial examples or measure perturbations other than adversarial perturbations (such as natural noise and degeneration). They can reflect the robustness of the model, but are not strictly adversarial robustness. Model-oriented metrics, on the other hand, can tap into more intrinsic information about the model and gain more implicit knowledge about its structure. They can assess the adversarial robustness of the model from a global perspective or based on the model's internal mechanisms, but they also have the drawback of being difficult to interpret. Summary of the Invention
[0006] The present invention provides an adversarial robustness evaluation method and terminal device based on a class activation map, which solves the difficult-to-explain technical problem in the prior art when evaluating the adversarial robustness of a model.
[0007] A first aspect of the present invention discloses a method for evaluating adversarial robustness based on class activation maps, comprising:
[0008] Use the adversarial attack algorithm to add perturbations to the original sample to obtain the adversarial sample;
[0009] Obtaining a class activation map of the deep learning model based on the original sample and a class activation map based on the adversarial sample;
[0010] Performing adversarial defense training on the deep learning model using the original sample and the adversarial sample to obtain a trained deep learning model;
[0011] Obtaining a class activation map of the trained deep learning model based on the original sample and a class activation map based on the adversarial sample;
[0012] The adversarial robustness of the deep learning model is evaluated based on the class activation maps of the original samples and the adversarial samples before and after training.
[0013] Preferably, evaluating the adversarial robustness of the deep learning model based on the class activation maps of the original samples and the adversarial samples before and after training specifically includes:
[0014] Determining a first average high-heat area overlap ratio based on the class activation maps of the original sample and the adversarial sample before training;
[0015] Determining a second average high-heat area overlap ratio according to the class activation maps of the original sample and the adversarial sample after training;
[0016] Evaluating the adversarial robustness of the deep learning model according to the first average high-heat area overlap ratio and the second average high-heat area overlap ratio;
[0017] The first average high-heat area overlap ratio and the second average high-heat area overlap ratio are both the ratio of the sum of the number of pixels with the same high-heat level at the same position to the total number of pixels in the class activation map of the original sample and the adversarial sample.
[0018] Preferably, the first average high-heat area overlap ratio and the second average high-heat area overlap ratio are both determined according to a first formula, which is:
[0019]
[0020] Where AHCR is the first average high heat area overlap ratio or the second average high heat area overlap ratio, k ij is the thermal level value of the pixel in row i and column j on the class activation map of the original sample, is the thermal level value of the pixel in the i-th row and j-th column on the class activation map of the adversarial sample, m and n are the total number of rows and columns of a class activation map, and N is the total number of pixels in the original sample and the adversarial sample. is the number of pixels in the class activation map of the original sample and the adversarial sample whose thermal level value is the same and greater than 4 in the i-th row and j-th column.
[0021] Preferably, evaluating the adversarial robustness of the deep learning model based on the class activation maps of the original samples and the adversarial samples before and after training specifically includes:
[0022] Determining a first average coverage overlap ratio based on class activation maps of the original sample and the adversarial sample before training;
[0023] Determining a second average coverage overlap ratio based on the class activation maps of the original sample and the adversarial sample after training;
[0024] Evaluating the adversarial robustness of the deep learning model according to the first average coverage overlap ratio and the second average coverage overlap ratio;
[0025] The first average coverage overlap ratio and the second average coverage overlap ratio are both the ratio of the sum of the number of pixels with the same thermal level at the same position to the total number of pixels in the class activation map of the original sample and the adversarial sample.
[0026] Preferably, the first average coverage overlap ratio and the second average coverage overlap ratio are both determined according to a second formula, which is:
[0027]
[0028] Where ACCR is the first average coverage overlap ratio or the second average coverage overlap ratio, k ij is the thermal level value of the pixel in row i and column j on the class activation map of the original sample, is the thermal level value of the pixel in the i-th row and j-th column on the class activation map of the adversarial sample, m and n are the total number of rows and columns of a class activation map, and N is the total number of pixels in the original sample and the adversarial sample. is the number of pixels with the same thermal level value in the i-th row and j-th column of the class activation map of the original sample and the adversarial sample.
[0029] A second aspect of the present invention discloses a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] The present invention provides a class activation map-based adversarial robustness assessment method and terminal device. Using SmoothGradCAM++ technology, the outputs of specific spatial locations in the final convolutional feature map of a deep neural network model are weighted by pixel, measuring the importance of the overall decision-making of the deep neural network model and outputting a class activation map. Indicators are designed based on the class activation map to measure the model's different responses to original samples and adversarial samples from the model's own perspective. By comparing the changes in the indicators before and after model training, the effectiveness of the indicators is verified, enabling adversarial robustness assessment based on the model's internal mechanisms while enhancing the interpretability of the assessment results. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 Flowchart of a method for evaluating adversarial robustness based on class activation maps according to an embodiment of the present invention. DETAILED DESCRIPTION
[0033] The technical solutions of the present invention will be described in further detail below with reference to the accompanying drawings and specific implementation examples. It should be understood that the following embodiments are merely exemplary illustrations and explanations of the present invention and should not be construed as limiting the scope of protection of the present invention. All technologies implemented based on the above content of the present invention are encompassed within the scope of protection intended by the present invention.
[0034] The first aspect of the present invention discloses a method for evaluating adversarial robustness based on class activation maps, such as Figure 1 As shown, including:
[0035] Step 1: Use the adversarial attack algorithm to add perturbations to the original sample to obtain the adversarial sample.
[0036] The original sample in the embodiment of the present invention is an image, the added disturbance is a small disturbance, and the counter-attack algorithm can be FGSM, DEEPFOOL or PGD attack algorithm, etc., which is not limited by the present invention.
[0037] Step 2: Obtain the class activation map of the deep learning model based on the original sample and the class activation map based on the adversarial sample.
[0038] In one embodiment, before step 2, the original samples and adversarial samples are subjected to preprocessing operations such as cropping and resizing to obtain a sample set with a pixel size of 224×224. The sample set includes the preprocessed original samples and the preprocessed adversarial samples.
[0039] The embodiment of the present invention uses SmoothGradCAM++ technology to generate a class activation map based on the original sample and a class activation map based on the adversarial sample of the deep learning model, wherein the deep neural network used by the deep learning model can be ResNet, Densenet, Squeezenet and VGG, etc., and the present invention does not limit this.
[0040] The greater the contribution to the prediction result in the class activation map, that is, the more important the information is, the "hotter" the color of the position is, and the closer it is to red.
[0041] Specifically, suppose there are N original samples of size m×n and N adversarial samples of size m×n. For each pixel with coordinates (i, j) on the sample image, there is a corresponding (H, S, V). According to the HSV color table, different heat levels are divided into 8 heat levels, with corresponding values 0 to 7, as shown in Table 1. The heat level value of each pixel on the class activation map of the original sample and the adversarial sample is k ij 、
[0042] Table 1 HSV color comparison table
[0043]
[0044]
[0045] Step 3: Use the original samples and adversarial samples to perform adversarial defense training on the deep learning model to obtain the trained deep learning model.
[0046] Step 4: Obtain the class activation map of the trained deep learning model based on the original sample and the class activation map of the adversarial sample.
[0047] This step is the same as step 2. SmoothGradCAM++ technology is used to generate the class activation map of the trained deep learning model based on the original sample and the class activation map based on the adversarial sample.
[0048] Step 5: Evaluate the adversarial robustness of the deep learning model based on the class activation maps of the original samples and adversarial samples before and after training.
[0049] In the embodiment of the present invention, when evaluating the adversarial robustness of a deep learning model, the evaluation indicators that can be used are the average high-heat area overlap ratio and the average coverage overlap ratio. The above two indicators can be used alone or in combination.
[0050] When the evaluation metric includes the average high-heat area overlap ratio, the adversarial robustness of the deep learning model is evaluated based on the class activation maps of the original samples and adversarial samples before and after training, including:
[0051] Determine the first average high-heat area overlap ratio based on the class activation maps of the original sample and the adversarial sample before training;
[0052] Determine the second average high-heat area overlap ratio based on the class activation maps of the original sample and the adversarial sample after training;
[0053] Evaluate the adversarial robustness of the deep learning model based on the first average high-heat area overlap ratio and the second average high-heat area overlap ratio;
[0054] The first and second average high-heat area overlap ratios are the ratios of the sum of the number of pixels with the same high-heat level at the same location in the class activation maps of the original and adversarial samples to the total number of pixels. This metric indicates the model's sensitivity to information in the original and adversarial samples that is relevant to the prediction results. A larger value indicates a model's robustness to adversarial scenarios.
[0055] The first average high-heat area overlap ratio and the second average high-heat area overlap ratio are both determined according to the first formula, which is:
[0056]
[0057] Where AHCR is the first average high heat area overlap ratio or the second average high heat area overlap ratio, k ij is the thermal level value of the pixel in row i and column j on the class activation map of the original sample, is the thermal level value of the pixel in the i-th row and j-th column on the class activation map of the adversarial sample, m and n are the total number of rows and columns of a class activation map, and N is the total number of pixels in the original sample and the adversarial sample. is the number of pixels in the class activation map of the original sample and the adversarial sample whose thermal level value is the same and greater than 4 in the i-th row and j-th column.
[0058] Specifically, when AHCR is the first average high heat area overlap ratio, k ij is the thermal level value of the pixel in row i and column j on the class activation map of the original sample before training, is the thermal level value of the pixel in the i-th row and j-th column on the class activation map of the adversarial sample before training, m and n are the total number of rows and columns of the class activation map before training, N is the total number of pixels in the original sample and the adversarial sample before training, is the number of pixels in the class activation map of the pre-trained original sample and the pre-trained adversarial sample whose thermal level value is the same and greater than 4 in the i-th row and j-th column.
[0059] When AHCR is the second average high heat area overlap ratio, k ij is the thermal level value of the pixel in row i and column j on the class activation map of the original sample after training, is the thermal level value of the pixel in the i-th row and j-th column on the class activation map of the adversarial sample after training, m and n are the total number of rows and columns of a class activation map after training, and N is the total number of pixels in the original sample and the adversarial sample after training. is the number of pixels in the class activation map of the trained original sample and the trained adversarial sample whose thermal level value is the same and greater than 4 in the i-th row and j-th column.
[0060] When the evaluation metric includes the average overlap ratio, the adversarial robustness of the deep learning model is evaluated based on the class activation maps of the original samples and adversarial samples before and after training, including:
[0061] Determine the first average coverage overlap ratio based on the class activation maps of the original sample and the adversarial sample before training;
[0062] Determine the second average coverage overlap ratio based on the class activation maps of the original sample and the adversarial sample after training;
[0063] Evaluate the adversarial robustness of the deep learning model based on the first average coverage overlap ratio and the second average coverage overlap ratio;
[0064] The first average overlap ratio and the second average overlap ratio are both the ratio of the sum of the number of pixels with the same thermal level at the same location in the class activation maps of the original and adversarial samples to the total number of pixels. This metric indicates the model's sensitivity to information related to the prediction results in the original and adversarial samples. A larger value indicates a stronger adversarial robustness.
[0065] The first average coverage overlap ratio and the second average coverage overlap ratio are both determined according to the second formula, which is:
[0066]
[0067] Where ACCR is the first average coverage overlap ratio or the second average coverage overlap ratio, k ij is the thermal level value of the pixel in row i and column j on the class activation map of the original sample, is the thermal level value of the pixel in the i-th row and j-th column on the class activation map of the adversarial sample, m and n are the total number of rows and columns of a class activation map, and N is the total number of pixels in the original sample and the adversarial sample. is the number of pixels with the same thermal level value in the i-th row and j-th column of the class activation map of the original sample and the adversarial sample.
[0068] Same as AHCR indicator, when ACCR is the first average coverage overlap ratio, k ij 、 m, n, N, etc. are the corresponding parameters before training. When ACCR is the second average coverage overlap ratio, k ij 、 m, n, N, The present invention will not elaborate on this in detail.
[0069] A second aspect of the present invention discloses a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above method are implemented.
[0070] The method and terminal device of the present invention utilize SmoothGradCAM++ technology to weight the output of specific spatial locations in the final convolutional feature map of a deep neural network model on a pixel-by-pixel basis, measure the importance of the overall decision-making of the deep neural network model, and output a class activation map. Indicators are designed based on the class activation map to measure the model's different responses to original samples and adversarial samples from the model's own perspective. By comparing the changes in the indicators before and after model training, the effectiveness of the indicators is verified, and adversarial robustness is evaluated based on the model's internal mechanisms, while also enhancing the interpretability of the evaluation results.
[0071] The method of the present invention will be described in detail below with more specific examples.
[0072] The experimental dataset used CIFAR10 as the original samples. Adversarial examples were generated using three mainstream adversarial attack algorithms: FGSM, DEEPFOOL, and PGD. Eleven deep neural networks with different architectures, including ResNet, DenseNet, SqueezeNet, and VGG, were used. PGD adversarial training was used for adversarial defense training. The experimental environment was Linux, the programming language was Python, and the libraries used in the project were Pytorch and Deepobust. Table 2 shows the changes in the four metrics before and after the model defense.
[0073] Table 2 Comparison of two indicators before and after model defense
[0074]
[0075] The data in Table 2 shows that the values of the two indicators, AHCR and ACCR, are improved after the model defense using different attack algorithms and different models. This verifies that the two proposed indicators can effectively evaluate the adversarial robustness of the model. Therefore, the adversarial robustness assessment method based on class activation maps proposed in this paper can evaluate the adversarial robustness of the model, combining the characteristics of evaluation from the perspective of the model's internal mechanism and interpretability.
[0076] The present invention can mine implicit knowledge in deep neural network models, evaluate adversarial robustness from the perspective of the model itself rather than direct model results, and introduce key position information to enhance the interpretability and accuracy of the evaluation results.
[0077] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as above in terms of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can, without departing from the scope of the technical solution of the present invention, use the technical content disclosed above to make many possible changes and modifications to the technical solution of the present invention, or modify it into an equivalent embodiment with equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention should fall within the scope of protection of the technical solution of the present invention.
Claims
1. A method for evaluating adversarial robustness based on class activation maps, characterized by: include: Use the adversarial attack algorithm to add perturbations to the original sample to obtain the adversarial sample; The original sample is an image; Obtaining a class activation map of the deep learning model based on the original sample and a class activation map based on the adversarial sample; Performing adversarial defense training on the deep learning model using the original sample and the adversarial sample to obtain a trained deep learning model; Obtaining a class activation map of the trained deep learning model based on the original sample and a class activation map based on the adversarial sample; The adversarial robustness of the deep learning model is evaluated based on the class activation maps of the original samples and the adversarial samples before and after training.
2. The method according to claim 1, wherein: Evaluate the adversarial robustness of the deep learning model based on the class activation maps of the original sample and the adversarial sample before and after training, specifically including: Determining a first average high-heat area overlap ratio based on the class activation maps of the original sample and the adversarial sample before training; Determining a second average high-heat area overlap ratio according to the class activation maps of the original sample and the adversarial sample after training; Evaluating the adversarial robustness of the deep learning model according to the first average high-heat area overlap ratio and the second average high-heat area overlap ratio; The first average high-heat area overlap ratio and the second average high-heat area overlap ratio are both the ratio of the sum of the number of pixels with the same high-heat level at the same position to the total number of pixels in the class activation map of the original sample and the adversarial sample.
3. The method according to claim 2, wherein: The first average high-heat area overlap ratio and the second average high-heat area overlap ratio are both determined according to a first formula, which is: Where, is the first average high-heat area overlap ratio or the second average high-heat area overlap ratio, is the class activation map of the original sample i Rank j The thermal level value of the column pixel, is the class activation map of the adversarial sample i Rank j The thermal level value of the column pixel, and are the total number of rows and columns of a class activation map, respectively. is the total number of pixels in the original sample and the adversarial sample, is the class activation map of the original sample and the adversarial sample i Rank j The number of pixels in the column with the same thermal level value greater than 4.
4. The method according to claim 1, wherein: Evaluate the adversarial robustness of the deep learning model based on the class activation maps of the original sample and the adversarial sample before and after training, specifically including: Determining a first average coverage overlap ratio based on class activation maps of the original sample and the adversarial sample before training; Determining a second average coverage overlap ratio based on the class activation maps of the original sample and the adversarial sample after training; Evaluating the adversarial robustness of the deep learning model according to the first average coverage overlap ratio and the second average coverage overlap ratio; The first average coverage overlap ratio and the second average coverage overlap ratio are both the ratio of the sum of the number of pixels with the same thermal level at the same position to the total number of pixels in the class activation map of the original sample and the adversarial sample.
5. The method according to claim 4, wherein: The first average coverage overlap ratio and the second average coverage overlap ratio are both determined according to a second formula, which is: Where, is the first average coverage overlap ratio or the second average coverage overlap ratio, is the class activation map of the original sample i Rank j The thermal level value of the column pixel, is the class activation map of the adversarial sample i Rank j The thermal level value of the column pixel, and are the total number of rows and columns of a class activation map, respectively. is the total number of pixels in the original sample and the adversarial sample, is the class activation map of the original sample and the adversarial sample i Rank j The number of pixels in a column with the same thermal level value.
6. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.