Explanatable method based on enhanced gradient and class activation mapping

Abstract

The invention discloses an interpretability method based on enhanced gradient and class activation mapping, which is used for performing post visual interpretation on a convolutional neural network and improving the credibility and applicability of the convolutional neural network. According to the main technical scheme, the method comprises the following steps: firstly, selecting a proper baseline, then extracting the baseline and an input activation graph from a specific convolutional layer of a convolutional neural network, secondly, using a Shapley value of each channel of an enhanced gradient effective approximation activation graph as a weight coefficient, then obtaining a class activation mapping graph through linear weighted combination, and finally obtaining the class activation mapping graph. And finally, performing post-processing including maximum and minimum value normalization, Relu truncation addition, bilinear interpolation up-sampling and mask to an original input image to obtain a saliency map capable of visually displaying the region of interest of the model in the input image. According to the method, the convolutional neural network interpretation model conforming to the axiom can be constructed, and the loyalty of the interpretable method is optimized. The method has the advantages of effectiveness, stability and applicability, can clearly and accurately position the decision basis of the convolutional neural network, has robustness to adversarial attacks, and is suitable for visual interpretation of various convolutional neural networks in complex application scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer science, specifically to the fields of deep learning and explainable artificial intelligence, and more particularly to the research and application of convolutional neural networks. This invention proposes a modeling method suitable for post-visual interpretation of convolutional neural networks. Background Art

[0002] With the rapid development of deep learning technology, convolutional neural networks (CNNs), with their excellent feature extraction and pattern recognition capabilities, have been widely used in fields such as image classification, object detection, semantic segmentation, caption generation, and visual question answering. However, despite their remarkable achievements in various fields, CNNs, as deep learning models, typically have a large number of parameters and a complex network structure, making their decision-making process difficult to understand and explain. This black-box nature limits human trust in CNNs, especially in fields that require a high degree of transparency in the model's decision-making process, such as medical diagnosis, financial risk assessment, and autonomous driving.

[0003] Interpretability methods for convolutional neural networks have become particularly important. They can help practitioners better understand the working principles and decision-making processes of convolutional neural networks, reveal the network's focus on input data and the basis for its decisions, and enhance the interpretability and credibility of the model. They can also help engineers further improve the performance and robustness of the model by identifying weaknesses and error patterns in the model to guide feature engineering, model selection, and hyperparameter tuning.

[0004] In the field of image processing, saliency maps are one of the most commonly used post-processing visual explanation methods. They highlight the key image regions on which model decisions are based. Researchers at home and abroad have conducted extensive work on developing visual explanation methods for convolutional neural networks. However, existing explanation methods have several problems:

[0005] (1) Explanation methods are not effective. Direct gradient-based methods are constrained by gradient problems, some modified backpropagation methods do not follow the chain rule, and some perturbation-based methods lack solid axiom guarantees. These methods may not effectively reflect the model's decision-making basis and reasoning behavior.

[0006] (2) Interpretation methods are not stable. Some direct gradient-based methods have been shown to be vulnerable to fluctuating data, and some perturbation-based methods are very sensitive to small changes in data distribution. These methods may lead to instability in model interpretation.

[0007] (3) Explanation methods are not applicable. Some explanation methods have strong assumptions about specific types of data or models, which limits their applicability in different scenarios. Some explanation methods have high computational complexity, which limits their application to large-scale datasets or complex models.

[0008] In summary, existing model interpretation methods need more research to address challenges in effectiveness, stability, and applicability to improve their usability and credibility in practical applications. Summary of the Invention

[0009] To overcome the shortcomings of existing methods, this paper proposes a more effective, stable, and applicable method for visual interpretation of convolutional neural networks, inspired by the Shapley value in cooperative game theory. By analyzing and visualizing the internal representation of a convolutional neural network, this method reveals its decision-making process and feature extraction methods. This method improves model interpretability, enhances user trust, and identifies model flaws, and is of great significance for the research and application of deep learning and explainable artificial intelligence.

[0010] To achieve the above objectives, the technical solution of the present invention is an interpretability method based on enhanced gradients and class activation maps: selecting an appropriate baseline, performing feature extraction through forward propagation, using enhanced gradients to approximate the Shapley value weight coefficients of each channel of the activation map, linearly weighted combination of each channel of the activation map to obtain the class activation map, and performing post-processing operations to obtain the final saliency map. The specific steps are as follows:

[0011] 1) Baseline selection: effectively model feature missingness, model feature importance as the degree to which the presence or absence of the current feature affects the change in the model's prediction score, and set a reference value for the interpretation method.

[0012] 2) Activation map extraction: Analyze the intrinsic representation of the convolutional neural network and obtain the category-related image feature information it extracts.

[0013] 3) Calculating weight coefficients: The class activation mapping method is modeled as an additive feature attribution model. The Shapley value from cooperative game theory is used as the weight coefficient of each channel of the activation map to uniquely conform to important axioms and ensure accurate measurement of the contribution of each feature.

[0014] 4) Efficient approximate calculation: Due to the exponential computational complexity of Shapley values, in order to balance computational overhead and accuracy, the present invention effectively approximates Shapley values ​​in practice by using enhanced gradients based on integral gradients or expected gradients while ensuring that the axioms are satisfied.

[0015] 5) Linear Weighted Combination and Post-Processing: A complete interpretation model is constructed by linearly weighting the activation map channels according to the weight coefficients, resulting in a class activation map. Post-processing is performed to obtain a saliency map that accurately locates the image region of interest, including minimum and maximum normalization, ReLU truncation, bilinear interpolation upsampling, and masking to the original input image.

[0016] As a further solution of the present invention, baseline selection specifically includes: humans perform attribution relying on counterfactual intuition, regarding the absence of causes as a benchmark for comparison results. When interpreting a model, it is necessary to understand which features, when missing, can most change the model output. The baseline is closely related to the concept of missingness in the feature space. Selecting a suitable baseline is crucial for interpreting the model's prediction results and feature importance, providing a standard for comparison and reference. This paper proposes a solution of selecting two baselines at two locations in the model, measuring the importance of a feature as the change in prediction score caused by the change of the feature from the baseline value to the current value, avoiding the problems of directly using the gradient.

[0017] As a further solution of the present invention, activation map extraction specifically includes: activation map refers to a feature map generated in each convolutional layer of a convolutional neural network, which shows specific features or patterns triggered in different levels of the network. By extracting the activation map, it is possible to deeply explore the internal workings of the model, understand the extracted and encoded image features, such as edges, textures or higher-level semantic features, and deepen the understanding of its image feature representation and pattern recognition capabilities. The class activation mapping method ultimately linearly weights the channels of the activation map, so it is necessary to obtain the activation maps of the input image and the baseline. The last convolution layer of the convolutional neural network is considered to be a good balance between high-level semantic information and detailed spatial information, so this layer is generally selected to extract the activation map. The present invention can select any convolution layer to extract the activation map.

[0018] As a further solution of the present invention, the calculation of the weight coefficient specifically includes: the weight coefficient is the essential difference between the various methods based on class activation mapping, which is used to measure the importance of each position in the image to a specific category. It is related to the performance of the interpretation method, affecting whether the interpretation model can faithfully reflect the network behavior and whether it can accurately locate the target area. In order to overcome the problems existing in previous methods, the present invention classifies the class activation mapping method into the additive feature attribution framework, and under the inspiration of the Shapley value in cooperative game theory, the average marginal benefit of each channel of the activation map to the model prediction score is used as the weight coefficient, thereby becoming the only effective solution that meets important axioms such as local accuracy, missingness and consistency.

[0019] As a further solution of the present invention, efficient approximate calculation specifically includes: although the Shapley value is the only attribution method that satisfies the above axioms, its exponential computational overhead makes it almost impossible to perform fast and accurate calculations in practical applications. The integral gradient and expected gradient are actually based on the Aumann-Shapley value, which is an extension of the Shapley value under a continuous setting and also conforms to the above axioms. If the model is approximated by a linear function between the baseline and the current explanatory input, and it is assumed that the features are independent, then the integral gradient and expected gradient will calculate the approximate Shapley value. The present invention solves the integral through Riemann sum, and calculates the enhanced gradient by selecting multiple baselines, calculating multi-step gradients and obtaining multiple inputs, avoiding the problems of gradient existence, and efficiently calculating the weight coefficients.

[0020] As a further aspect of the present invention, the linear weighted combination and post-processing specifically include: each channel of the activation map represents each feature extracted by the model, and based on the class activation mapping method, these feature maps are multiplied by the classifier weights and summed channel by channel to obtain the final class activation map. To obtain the final saliency map that can accurately locate the image area of ​​interest of the model, post-processing is required, including: using ReLU truncation, maximum and minimum normalization, bilinear interpolation upsampling, and masking to the original image. The beneficial effects of the present invention are as follows:

[0021] Compared with the prior art, the advantages of the present invention are:

[0022] 1) The proposed interpretation method is effective. It models the class activation mapping method as an additive feature attribution method and uses the Shapley values ​​of each channel of the activation map as weight coefficients. It conforms to the important axioms of the attribution method and can faithfully and effectively reflect the model behavior.

[0023] 2) The proposed modeling method is stable and uses enhanced gradients to efficiently approximate Shapley values ​​in practice. It is robust to adversarial attacks, can reduce noise and generate clearer saliency maps.

[0024] 3) The technical solution proposed in this invention is applicable and allows migration to other deep learning models for application in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings;

[0026] Figure 1 This is a schematic diagram of the overall framework process provided by an embodiment of the present invention;

[0027] Figure 2 It is a schematic diagram of calculating the weight coefficient by integrating the gradient at the input of the model;

[0028] Figure 3 It is a schematic diagram of calculating the weight coefficient by integrating the gradient at a specific convolutional layer; DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments and drawings in the present invention, all other embodiments and drawings obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Those skilled in the art should understand that the content described specifically below is illustrative rather than restrictive, and should not be used to limit the scope of protection of the present invention.

[0030] One embodiment of the present invention provides an interpretability method based on enhanced gradient and class activation mapping. The specific implementation process includes: selecting a suitable baseline, extracting an activation map through forward propagation, using enhanced gradient to approximate the Shapley value weight coefficient of each channel of the activation map, linearly weighting the activation map channels to obtain a class activation map, and performing post-processing operations to obtain a final saliency map. Figure 1 A schematic diagram depicting the overall process of the method.

[0031] In some optional implementations of this embodiment, the specific implementation of selecting the baseline further includes: for the input end of the model, a completely black image or a plurality of images randomly extracted from the background data set can be selected as the baseline; for a specific convolutional layer of the model, an all-zero vector with the same dimension as the activation map or activation maps corresponding to a plurality of images randomly extracted from the background data set can be selected as the baseline. Figure 2 and Figure 3 In the section on calculating intermediate interpolation, diagrams are listed for using a completely black image and a completely zero vector of the same dimension as the activation map as the baseline.

[0032] In some optional implementations of this embodiment, the specific implementation method of calculating the weight coefficient further includes: the essential difference between the class activation mapping method and the weight coefficient lies in the weight coefficient. The present invention refers to the Shapley value idea in cooperative game theory and the existing SHAP interpretation model, models the class activation mapping method as an additive feature attribution method, and provides a unique solution that conforms to the local accuracy, missingness and consistency axioms, that is, determining the weight coefficient of each channel of the activation map as the Shapley value.

[0033] In some optional implementations of this embodiment, a specific implementation of efficient approximate calculation further includes: balancing computational overhead and computational accuracy, the present invention effectively approximates the Shapley value by using an enhanced gradient based on the integral gradient or the expected gradient. This is because the Shapley value is an extension of the Shapley value in the continuous case, conforms to the above-mentioned important axioms, and satisfies the implementation invariance and symmetry axioms. In specific practice, the Riemann sum is used to approximate the integral, including calculations at two locations: the input end of the convolutional neural network and a specific convolutional layer. Figure 2 The diagram shows how weight coefficients are calculated by integrating gradients at the model input. A completely black image is chosen as the baseline, and intermediate interpolations are calculated equidistantly along the straight path between the baseline and the input. These are then input into the model to obtain the gradient for a specific category through backpropagation. Finally, the mean of the gradients is calculated and multiplied by the difference in activation maps to obtain the integrated gradient, effectively approximating the Shapley value. Figure 3 This diagram shows how weight coefficients are calculated using integrated gradients in a specific convolutional layer of the model. A zero vector of the same dimension as the activation map of the input image is chosen as the baseline. Intermediate interpolations are calculated at equal intervals along the straight path between the baseline and the input activation map. These are fed into the second half of the model to obtain the gradient for the specific class via backpropagation. Finally, the gradients are averaged and multiplied by the difference between the activation maps to obtain the integrated gradient, effectively approximating the Shapley value. A recommended interpolation step size is approximately 50. For approximating Shapley values ​​using expected gradients, the scheme is similar to that of the integrated gradient, except that both the baseline and the path are represented as expectations. A different baseline and interpolation coefficient are randomly selected for each intermediate interpolation. At the input, a random image is sampled from the background dataset; at the specific convolutional layer, the activation map of a random image is used as the baseline. A recommended interpolation step size is approximately 200. For both the integrated gradient and expected gradient, appropriate Gaussian noise is added to the original input image when calculating at the input, and to the activation map of the original input image when calculating at the specific convolutional layer. The level of Gaussian noise added should be determined according to the specific application scenario.

[0034] In some optional implementations of this embodiment, the specific implementation methods of the linear weighted combination and post-processing further include: after obtaining the weight map of the activation map, the importance of each element of the channel is accumulated to obtain the weight coefficient of the channel, and then the linear weighted combination is used to obtain the class activation map, as shown in Figure 1. In order to focus only on the image area that plays a positive role in classification, a Relu truncation operation can be added. If a counterfactual explanation is required, only the negative part of the class activation map can be retained. Through the maximum and minimum value normalization operation, the value range of the elements in the explanation map is limited, so that the contrast of each area is more obvious. By upsampling to the original input image size and masking to the original image through bilinear interpolation, an explanation map that can locate the image area of ​​interest of the model is obtained, which is easier for humans to observe.

Claims

1. An interpretability method based on enhanced gradient and class activation mapping, characterized by The class activation mapping method is modeled as an additive feature attribution model. The Shapley value in cooperative game theory is used as the weight coefficient of each channel of the convolutional neural network activation map. In practice, the gradient is enhanced to efficiently approximate the calculation. After linear weighted combination and post-processing operations, the saliency map is obtained. The specific steps include: (1) Baseline selection. Model missing features in the input, accurately measure the importance of features, and avoid the saturation problem of directly using gradients. (2) Activation map extraction: The activation maps corresponding to the baseline and input are extracted from the specific convolutional layer of the convolutional neural network through forward propagation to obtain information for model recognition. (3) Calculate the weight coefficient. Inspired by the Shapley value in cooperative game theory, the class activation mapping method is modeled as an additive feature attribution model. The weight coefficient of each channel is determined as the average marginal benefit of the model prediction score when the channel exists and does not exist in all possible channel combinations in the activation map. (4) Efficient approximate calculation. In order to be applicable to actual scenarios and balance computational overhead and accuracy, the present invention efficiently approximates the weight coefficients by enhancing the gradient based on the integral gradient and the expected gradient, and uses the Riemann sum to calculate the integral. Two locations and two methods are provided for approximate calculation of the Shapley value weight coefficients. The two locations are: the input end of the convolutional neural network and a specific convolution layer. The two calculation methods are: integral gradient and expected gradient. Because multiple calculations of intermediate interpolations are required, appropriate Gaussian noise can be naturally added to generate multiple copies, reducing the noise in the saliency map and obtaining a clearer and more coherent explanation map. (5) Linear weighted combination and post-processing. The obtained weight coefficients are multiplied by the corresponding activation maps and then linearly added to obtain the class activation map. The obtained class activation map is truncated with ReLU, normalized to the maximum and minimum values, upsampled to the original input image size through bilinear interpolation, and masked and superimposed on the original input image to obtain the final saliency map.

2. The interpretability method based on enhanced gradient and class activation mapping according to claim 1, characterized in that The need to select a suitable baseline further includes: for calculating intermediate interpolation at the model input, selecting a black image with the same dimension as the original image or randomly sampling multiple images from the background dataset as the baseline; for calculating intermediate interpolation at a specific convolutional layer of the model, selecting an all-zero vector with the same dimension as the activation map of the input image or the activation maps of multiple images randomly sampled from the background dataset as the baseline to model feature missing.

3. The interpretability method based on enhanced gradient and class activation mapping according to claim 1, characterized in that The class activation mapping method is established as an additive feature attribution model, using the Shapley value of each channel of the activation map as its linear weighting coefficient. In practice, it is efficiently approximated through integrated gradients or expected gradients, and can be optionally calculated at the model input or at a selected convolutional layer. It further includes: at the model input, the integrated gradient method calculates multiple intermediate interpolations at equal intervals on the straight path between the baseline image and the input image, and the expected gradient method randomly selects multiple baseline images from the background dataset and randomly determines the interpolation coefficients on the straight path between the baseline image and the input image to calculate multiple intermediate interpolations. At the same time, both methods can add appropriate Gaussian noise to the original input image to obtain multiple copies of the input. The obtained interpolations are then input into the convolutional neural network, and the gradient of the specific category is obtained through backpropagation. Finally, the gradients are averaged and multiplied by the difference between the activation map of the input image and the activation map of the baseline image to obtain the approximate Shapley value. For the selected convolutional layer, both methods treat the activation map as the input to the model's backend. The integrated gradient method computes multiple intermediate interpolations equidistantly along the straight line path between the input image's activation map and an all-zero vector of the same dimension. The expected gradient method similarly randomly selects multiple baseline images from the background dataset and randomly determines interpolation coefficients along the straight line path between the baseline image's activation map and the input image's activation map to compute multiple intermediate interpolations. Both methods add appropriate Gaussian noise to the original input image's activation map to produce multiple copies. The resulting interpolations are then input to the backend of the convolutional neural network, where class-specific gradients are obtained through backpropagation. Ultimately, the gradients are averaged and multiplied by the difference between the input image's activation map and the baseline image's activation map to approximate the Shapley value. This process of selecting multiple baselines, computing multiple intermediate interpolations, and then adding appropriate Gaussian noise to the input and then computing the gradient through backpropagation is called enhanced gradient. Its purpose is to efficiently approximate the Shapley value, reduce noise, and enhance robustness.

Images