Model decision interpretability method fusing integral gradient and class activation mapping

Abstract

The invention discloses a model decision interpretability method fusing integral gradient and class activation mapping, and belongs to the technical field of deep learning interpretability. Aiming at the problems of gradient noise interference, insufficient space positioning and the like existing in an existing single interpretability method, the method is optimized through five key steps: firstly, extracting a feature map set of the last convolutional layer of a deep learning model; secondly, calculating an integral gradient of the feature map to a target category based on a path integral idea; thirdly, obtaining a feature map weight through global average pooling; then, weighted summation is carried out, and an initial attribution thermodynamic diagram is generated through ReLU activation; and finally, a high-resolution thermodynamic diagram is obtained in combination with guided gradient optimization. According to the method, gradient stability of Integrating Gradients and spatial positioning advantages of Grad-CAM are fused, invariance and sensitivity axioms are realized, model adaptability is maintained, accuracy and robustness of interpretation results are remarkably improved, and the method is suitable for key fields such as signal processing and image classification.

Description

Technical Field

[0001] This invention relates to the field of deep learning model interpretability technology, specifically a hybrid interpretation method that integrates integrated gradients and class activation mapping (Grad-CAM). Background Technology

[0002] Deep learning models with convolutional layers have achieved groundbreaking progress in several important fields such as computer vision, signal processing, and healthcare, demonstrating their powerful application value. However, the inherent characteristics of deep learning models mean that their decision-making processes lack logical support that humans can understand. When these models are applied to critical fields with extremely high requirements for reliability and safety, such as medical diagnosis and autonomous driving, any decision-making bias can lead to serious consequences. Therefore, model interpretability technology has become a research hotspot in both academia and industry.

[0003] Currently, mainstream model interpretability methods are mainly divided into two categories: gradient-based interpretation methods and feature map visualization-based interpretation methods. Grad-CAM, as a representative method based on feature map visualization, generates attribution heatmaps by weighted summation of feature maps and gradient information, possessing good spatial localization capabilities and model adaptability. However, this method relies solely on a single backpropagation to calculate the gradient, making it highly susceptible to interference from local gradient noise. It struggles to effectively capture weak features in gradient-flat regions, resulting in numerous pseudo-feature regions in the heatmap and affecting interpretation accuracy. IntegratedGradients, as a representative gradient-based method, effectively solves the sensitivity loss problem of single gradients through path integral gradient calculation, strictly satisfying the realization invariance and sensitivity axioms. However, this method only outputs point-by-point importance scores, lacking spatial localization capabilities and failing to intuitively reflect the local feature regions of interest to the model, thus limiting its practical application scenarios. Furthermore, existing hybrid interpretation methods are mostly simple gradient weighted fusions, failing to fully leverage the complementary advantages of the two core methods, making it difficult to simultaneously achieve ideal results in localization accuracy and gradient stability. Therefore, there is an urgent need to develop an interpretability method that deeply integrates the advantages of both approaches to effectively solve the problems of noise interference and insufficient spatial positioning in the interpretation of complex data. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a model decision interpretability method that integrates integral gradient and class activation mapping. By replacing the traditional single gradient with path integral gradient calculation and combining feature map weighted visualization logic, it achieves dual optimization of gradient stability and spatial localization capability, thereby improving the accuracy and robustness of the interpretation results.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for interpretable model decisions that integrates integral gradients and class activation mappings includes the following steps: Step 1: Extract the feature map set output from the last convolutional layer of the deep learning model. ,in The total number of feature maps, The size is , , These represent the height and width of the feature map, respectively. The last convolutional layer was chosen because its feature map contains the most abstract and discriminative high-level semantic features, which can directly reflect the model's decision-making basis and avoid noise interference from shallow features.

[0006] Step 2: Define the baseline input and target input , build path ( Each feature map is computed using the Riemann sum approximation. For target category Path integral gradient Baseline Input Requires matching target input Consistent dimensions allow selection of either a zero-based vector or a completely black image that conforms to the data type distribution; path partitioning step size. Preferred To balance accuracy and efficiency.

[0007] Step 3: Aggregate the gradient information of the spatial dimension of the feature maps using global average pooling to obtain the gradient of each feature map. For target category Integral weights This operation effectively suppresses local gradient noise, improves the stability of weight calculation, and ensures that the weights reflect the global contribution of the feature map.

[0008] Step 4: Sum the integral weights with the corresponding feature maps using a weighted sum, and then apply the sum to the ReLU activation function to obtain the initial attribution heatmap. The ReLU activation function can filter out negative contribution regions, allowing the heatmap to focus on effective feature regions that have a positive effect on the prediction of the target category.

[0009] Step 5: Calculate the categories of the input data pairs The guiding gradient is used to upsample the initial attribution heatmap to the input data size, and then multiplied element-wise with the guiding gradient to obtain a high-resolution attribution heatmap. The guide gradient retains only the positive gradient, further suppressing negative contribution interference. Upsampling and fusion operations achieve precise alignment between the heatmap and the input data, improving detail accuracy.

[0010] Furthermore, the formula for calculating the path integral gradient in step two is as follows: in, For the model to class The original output score, Divide the path into step sizes.

[0011] Furthermore, the integration weights in step three The calculation formula is: in, It is a feature map exist The activation value of the location.

[0012] Furthermore, the formula for generating the initial attribution heatmap in step four is: Furthermore, the calculation formula for the high-resolution attribution heatmap in step five is as follows: in, To guide the gradient, This is the channel index for the input data.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. Significantly improved gradient stability: The path integral gradient is used instead of the single gradient calculation in traditional Grad-CAM. By accumulating gradient information on the path, the sensitivity loss problem of single gradient in flat areas is effectively avoided, the gradient calculation error is greatly reduced, and the accuracy of weak feature capture is significantly improved.

[0014] 2. Spatial positioning accuracy optimization: Relying on the feature map weighted visualization logic of Grad-CAM, accurate spatial positioning of target categories is successfully achieved. The spatial positioning accuracy is significantly improved compared with using IntegratedGradients alone, which completely solves the core defect of the pure gradient method lacking positioning capability.

[0015] 3. Enhanced credibility of the explanation: The explanation results strictly satisfy the implementation invariance and sensitivity axioms, are not affected by network implementation details, the proportion of pseudo-feature regions is significantly reduced compared with traditional Grad-CAM, the consistency with human visual judgment is greatly improved, and the credibility of the explanation is significantly enhanced.

[0016] 4. Excellent adaptability and practicality: It can be flexibly adapted to various deep learning models with convolutional layers without modifying the model architecture; the input data supports a variety of types such as high-dimensional time series signals, text feature maps, and medical images, with a wide range of application scenarios, while also taking into account real-time requirements, and has strong engineering practicality. Attached Figure Description

[0017] Figure 1 This is an overall flowchart of the method described in this invention.

[0018] Figure 2 This is a network structure diagram of the method described in this invention. Detailed Implementation

[0019] The present invention will be further described in detail below with reference to specific embodiments.

[0020] The present invention provides a method for interpretability of model decisions that integrates integral gradients and class activation mappings, comprising the following steps: Step 1: Extract the feature map set. Input the input data to be interpreted into the trained deep learning model with convolutional layers, and extract the feature map set output from the last convolutional layer. ,in For the number of channels, Size is .

[0021] Step 2: Calculate the path integral gradient. Define a baseline input with the same dimension as the input data. and target input Construct a linear path ( Set the path division step size Each feature map is computed using the Riemann sum approximation. For target category Path integral gradient The calculation formula is: in, For the model to target category The original output score.

[0022] Step 3: Calculate the integral weights. Global average pooling is used to process the path integral gradient to obtain each feature map. Integral weights The calculation formula is: in, For feature map exist The activation value of the location.

[0023] Step 4: Generate the initial attribution heatmap. The feature map is weighted and summed with its corresponding integral weights, then filtered for negative contribution regions using the ReLU activation function to generate the initial attribution heatmap. The calculation formula is as follows: Step 5: Generate a high-resolution attribution heatmap. The guided backpropagation algorithm is used to calculate the relationship between the input data and the target category. guiding gradient (Only positive gradients are retained), and the initial attribution heatmap is upsampled to the input data size using a bilinear interpolation algorithm. Then, multiply the result element-wise with the guiding gradient to obtain a high-resolution attribution heatmap. The calculation formula is as follows: in, This is the channel index for the input data.

[0024] In summary, this invention, by innovatively combining the advantages of integral gradient and class activation mapping, solves the defects of existing methods such as gradient noise interference and insufficient spatial localization, and is applicable to key fields requiring high-precision interpretation, such as signal processing and image classification.

Claims

1. A method for interpretable model decisions that integrates integral gradients and class activation mappings, characterized in that, Includes the following steps: Step 1: Extract the feature map set output from the last convolutional layer of the deep learning model. ,in The total number of feature maps, The size is , , These are the height and width of the feature map, respectively; Step 2: Define the baseline input and target input , build path ( Each feature map is computed using the Riemann sum approximation. For target category Path integral gradient ; Step 3: Aggregate the gradient information of the spatial dimension of the feature maps using global average pooling to obtain the gradient of each feature map. For target category Integral weights ; Step 4: Sum the integral weights with the corresponding feature maps using a weighted sum, and then apply the sum to the ReLU activation function to obtain the initial attribution heatmap. ; Step 5: Calculate the categories of the input data pairs The guiding gradient is used to upsample the initial attribution heatmap to the input data size, and then multiplied element-wise with the guiding gradient to obtain a high-resolution attribution heatmap. .

2. The model decision interpretability method integrating integral gradient and class activation mapping according to claim 1, characterized in that, The formula for calculating the path integral gradient in step two is: in, For the model to class The original output score, Divide the path into step sizes, and .

3. The model decision interpretability method integrating integral gradient and class activation mapping according to claim 1, characterized in that, Integral weights in step three The calculation formula is: in, It is a feature map exist The activation value of the location.

4. The model decision interpretability method integrating integral gradient and class activation mapping according to claim 1, characterized in that, The formula for generating the initial attribution heatmap in step four is: The ReLU operation sets the negative contribution region to 0, retaining only the region that contributes to the category. The positive contribution area.

5. The model decision interpretability method integrating integral gradient and class activation mapping according to claim 1, characterized in that, In step five, the gradient guide only retains the positive gradient. The formula for calculating the high-resolution attribution heatmap is: in, To guide the gradient, For the channel index of the input data, , , These represent the height, width, and number of channels of the input data, respectively.

6. The model decision interpretability method integrating integral gradient and class activation mapping according to claim 2, characterized in that, The path division step size .

7. The model decision interpretability method integrating integral gradient and class activation mapping according to claim 1, characterized in that, The deep learning model is a model containing convolutional layers.

8. The model decision interpretability method integrating integral gradient and class activation mapping according to claim 1, characterized in that, The baseline input A vector of all zeros or a completely black image that conforms to the distribution of the input data.

9. The model decision interpretability method integrating integral gradient and class activation mapping according to claim 1, characterized in that, The input data includes high-dimensional time-series signals, natural images, medical images, or text feature maps.

Images