Method and apparatus for deep neural network decision bias mining and visualization

Abstract

The application provides a deep neural network decision bias mining and visualization method and device, which has the following characteristics: step S1, inputting an image into a deep neural network model to obtain an output result; step S2, selecting a target feature layer; step S3, obtaining a weight vector according to the label of the image and the output result; step S4, setting a channel mask; step S5, obtaining a corresponding covered feature map through the channel mask; step S6, obtaining a covered output result according to all the covered feature maps; step S7, obtaining a loss function calculation result according to the covered output result, the output result and the channel mask; step S8, updating the channel mask according to the loss function calculation result; step S9, repeating steps S5 to S8 until a termination condition is reached, and then obtaining an attribution heat map according to all the channel masks, the feature maps and the weight vector. In summary, the method can obtain a visualized result of potential decision bias of a deep neural network.

Description

Technical Field

[0001] This invention relates to the field of intelligent image analysis, specifically to a method and apparatus for mining and visualizing decision biases in deep neural networks. Background Technology

[0002] Image analysis and classification systems based on deep neural networks have wide applications in various fields, including medical image analysis and autonomous driving. However, deep neural networks are typically complex black-box models, making it difficult to clearly explain their internal decision-making processes. Visualization techniques can make the model's decisions more transparent, helping users understand why the model produces specific outputs for different input data. This is crucial for reviewing the model's behavior, verifying its accuracy, and ensuring that the model meets specific criteria.

[0003] Visualization can also help researchers and engineers identify potential problems in models, such as vanishing gradients, overfitting, and underfitting. Through visualization, they can more easily identify areas for improvement and fine-tune the model. Visualization can also be used to verify the correctness and consistency of a model. It allows them to check whether the model performs the task as expected and whether it aligns with the knowledge of domain experts.

[0004] Furthermore, enhancing trust in decision-making is a crucial prerequisite for model application. In key areas such as medical diagnosis, model decisions must be trusted. Visualization can provide evidence, helping users understand how the model works and increasing trust in its decisions. The significance of neural network visualization lies in helping us understand and trust neural network models, making them more feasible and usable in practical applications. This is of great importance for promoting the development and application of deep learning technology.

[0005] To improve the interpretability and performance of image classification models, researchers proposed Grad-CAM, a deep learning-based technique used to visualize the responses of deep neural networks to input images. Grad-CAM analyzes the gradient information of specific layers in a Convolutional Neural Network (CNN) to determine the neural network's focus on the input image during classification decisions. This technique provides visual feedback on the contribution of different pixel regions in the image to the classification result, enabling operators to understand how the neural network makes classification decisions.

[0006] However, Grad-CAM has some problems, such as unstable pixel values ​​in the generated heatmaps, which makes the visualization less clear and susceptible to noise. The paper "Omeiza D, Speakman S, Cintas C, et al. Smooth grad-cam++: An enhanced inference level visualization technique for deep convolutional neural network models[J]. arXiv preprint arXiv:1908.01224,2019" discloses a method to reduce randomness and noise by adding small amounts of noise to the input image multiple times, then calculating the generated Grad-CAM heatmap each time, and finally averaging these heatmaps. This provides a smoother visualization result, but at the same time, it significantly increases the computational cost.

[0007] Discovering and quantifying decision bias is a prerequisite for addressing bias. Existing bias mining algorithms generally fall into the following categories: 1) Starting with the dataset: Data imbalance is a common cause of model bias. To address this, researchers use data augmentation techniques and resampling strategies to increase the number of samples for the minority class, thereby reducing the model's bias towards the majority class. 2) Adding fairness constraints to the training objective: Introducing fairness constraints during training to ensure that the model's predictions are fair across different groups. 3) Fair preprocessing and postprocessing: Several techniques are used to perform data preprocessing and postprocessing before model training or after predicting output to ensure fairness. This includes relabeling data, correcting output probabilities, and reordering the candidate list.

[0008] However, there is currently no method for visualizing the potential decision biases of deep neural networks. Summary of the Invention

[0009] This invention is made to solve the above-mentioned problems, and aims to provide a method and apparatus for mining and visualizing decision bias in deep neural networks.

[0010] This invention provides a method for mining and visualizing decision bias in deep neural networks, used to generate attribution heatmaps of images input to a deep neural network model. The method includes the following steps: Step S1, inputting the image into the deep neural network model to obtain the output result; Step S2, selecting a feature layer of the deep neural network model as the target feature layer; Step S3, performing backpropagation based on the image label and the output result to calculate the weight vector; Step S4, setting initialization masks for each channel of the target feature layer as corresponding channel masks; Step S5, for each channel, applying the corresponding channel mask... The feature map generated by this channel is masked to obtain the corresponding masked feature map; in step S6, all masked feature maps are used as the output of the target layer and processed by subsequent layers of the deep neural network model to obtain the masked output result; in step S7, for each channel mask, the loss function is calculated based on the masked output result, the output result and the channel mask to obtain the corresponding loss function calculation result; in step S8, all channel masks are updated by backpropagation based on all loss function calculation results; in step S9, steps S5 to S8 are repeated until the termination condition is met, and then the attribution heatmap is calculated based on all channel masks, feature maps and weight vectors.

[0011] The deep neural network decision bias mining and visualization method provided by this invention may also have the following feature: wherein, in step S3, the weight vector is a vector with dimension equal to the number of channels in the target feature layer, containing the weights of each channel in the target feature layer, and the formula for calculating the weight vector is: In the formula g t For the weight vector, For feature map l t The upper coordinate is (o, j), Z is the normalized term, f L This refers to the results corresponding to the labels in the output.

[0012] The deep neural network decision bias mining and visualization method provided by this invention may also have the following features: In step S9, the feature map corresponding to each channel is multiplied by the channel mask to obtain the corresponding distillation feature map; each weight in the weight vector is multiplied by the corresponding distillation feature map to obtain the weight feature map; and all weight feature maps are summed to obtain the attribution heatmap. The expression for the attribution heatmap is: CD-CAM = ReLU(g t ·m·l t In the formula, CD-CAM is the attribution heatmap, ReLU is the activation function, m is the channel mask, and l t This is the feature map corresponding to the channel mask.

[0013] The deep neural network decision bias mining and visualization method provided by this invention may also have the following feature: wherein, in step S7, the formula for calculating the loss function is: ||f mask -f||1+α||m||1+βTV(m), TV(m)=∑∑(|I(i,j)-I(i+1,j)|+|I(i,j)-I(i,j+1)|), where f mask To mask the output, f is the output, m is the channel mask, ||||1 is the L1 regularization term, I(i,j) is the value of the feature map with coordinate (i,j) generated by the channel corresponding to the channel mask, and α and β are both hyperparameters.

[0014] The deep neural network decision bias mining and visualization method provided by the present invention may also have the following feature: wherein, in step S2, the target feature layer is the last layer of the convolutional layer of the deep neural network model.

[0015] The deep neural network decision bias mining and visualization method provided by the present invention may also have the following features: wherein, in step S4, the initialization mask is a tensor with the same shape as the target feature layer and is initialized, and the initialization method includes Gaussian noise initialization and all-1 initialization.

[0016] The deep neural network decision bias mining and visualization method provided by the present invention may also have the following feature: in step S9, the termination condition is that the number of times the channel mask is updated reaches a preset number.

[0017] This invention also provides a deep neural network decision bias mining and visualization device for generating attribution heatmaps of images input to a deep neural network model. It comprises: an input module, a model storage module, an output result generation module, a target selection module, a weight vector generation module, a channel mask generation module, a masking feature map generation module, a masking output result generation module, a loss function calculation module, a channel mask update module, an update judgment module, an attribution heatmap generation module, and an output module. The input module is used to input images and corresponding labels; the model storage module is used to store the deep neural network model; the output result generation module is used to extract the deep neural network model from the model storage module and input the image into the deep neural network model to obtain the output result; the target selection module is used to select a feature layer of the deep neural network model as the target feature layer; the weight vector generation module is used to perform backpropagation based on the labels and output result to calculate the weight vector; and the channel mask generation module is used to generate the target feature map. The initialization mask corresponding to each channel in the feature layer is used as the channel mask. The masking feature map generation module is used to mask the feature maps generated for the corresponding channels according to the channel masks to obtain the corresponding masked feature maps. The masking output result generation module is used to use all masked feature maps as the output of the target layer. After processing by the subsequent layers of the deep neural network model, the masked output result is obtained. The loss function calculation module is used to calculate the loss function according to the masked output result, the output result and the channel mask to obtain the loss function calculation result for the corresponding channel. The channel mask update module is used to backpropagate and update all channel masks according to all loss function calculation results and send them to the update judgment module. The update judgment module stores a preset termination condition and judges whether the termination condition is met. If it is, the channel mask is sent to the attribution heatmap generation module. If not, the channel mask is sent to the masking feature map generation module. The attribution heatmap generation module is used to calculate the attribution heatmap according to all channel masks, feature maps and weight vectors. The output module is used to output the attribution heatmap.

[0018] The role and effect of invention

[0019] According to the deep neural network decision bias mining and visualization method and apparatus of the present invention, on the one hand, feature maps are masked using channel masks, and cognitive distillation is performed on the feature maps. Furthermore, the channel masks are continuously updated during iterative optimization using a loss function, thereby extracting feature channel information from important parts of the feature maps. On the other hand, the importance of each channel to the model's decision is obtained by calculating the weight vector of the target feature layer. Then, the feature maps after channel masking are integrated based on weights to obtain an accurate attribution heatmap. Therefore, the deep neural network decision bias mining and visualization method and apparatus of the present invention can obtain visualization results targeting potential decision biases in deep neural networks. Attached Figure Description

[0020] Figure 1 This is a block diagram of a deep neural network decision bias mining and visualization device in an embodiment of the present invention.

[0021] Figure 2 This is a flowchart illustrating the deep neural network decision bias mining and visualization method in an embodiment of the present invention;

[0022] Figure 3 This is a comparative schematic diagram of the Grad-CAM visualization method and the CD method in an embodiment of the present invention. Detailed Implementation

[0023] To make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, provide a detailed description of the deep neural network decision bias mining and visualization method and apparatus of the present invention.

[0024] The deep neural network decision bias mining and visualization method and apparatus of this embodiment are used to generate attribution heatmaps of images input to a deep neural network model.

[0025] This deep neural network model is a deep residual network model applied to the field of intelligent medical diagnosis. It obtains corresponding diagnostic results based on the input chest X-ray.

[0026] The deep neural network decision bias mining and visualization method and apparatus of this embodiment can generate an attribution heatmap based on the chest X-ray, thereby visually demonstrating the potential cognitive shortcuts of the deep residual network model.

[0027] This embodiment provides a deep neural network decision bias mining and visualization device for generating attribution heatmaps of images input to a deep neural network model.

[0028] Figure 1 This is a block diagram of a deep neural network decision bias mining and visualization device in an embodiment of the present invention.

[0029] like Figure 1 As shown, the deep neural network decision bias mining and visualization device 1 of this embodiment includes an input module 11, a model storage module 12, an output result generation module 13, a target selection module 14, a weight vector generation module 15, a channel mask generation module 16, a masking feature map generation module 17, a masking output result generation module 18, a loss function calculation module 19, a channel mask update module 20, an update judgment module 21, an attribution heatmap generation module 22, an output module 23, and a device control module 24 for controlling the above modules.

[0030] Input module 11 is used to input images and corresponding labels.

[0031] Model storage module 12 is used to store deep neural network models.

[0032] The output generation module 13 is used to extract the deep neural network model from the model storage module, input the image into the deep neural network model, and obtain the output result.

[0033] The target selection module 14 is used to select a feature layer of the deep neural network model as the target feature layer.

[0034] The weight vector generation module 15 is used to backpropagate based on the labels and output results to calculate the weight vector.

[0035] The channel mask generation module 16 is used to generate the initialization mask corresponding to each channel in the target feature layer as the channel mask.

[0036] The masking feature map generation module 17 is used to mask the feature maps generated for the corresponding channels according to the channel masks, so as to obtain the corresponding masking feature maps.

[0037] The masking output generation module 18 is used to take all masking feature maps as the output of the target layer, and then process them through subsequent layers of the deep neural network model to obtain the masking output results.

[0038] The loss function calculation module 19 is used to calculate the loss function based on the masking output result, the output result and the channel mask, and obtain the loss function calculation result of the corresponding channel.

[0039] The channel mask update module 20 is used to backpropagate and update all channel masks based on the calculation results of all loss functions, and then send the updated judgment module.

[0040] The update judgment module 21 stores preset termination conditions and determines whether the termination conditions are met. If so, the channel mask is sent to the attribution heatmap generation module; otherwise, the channel mask is sent to the masking feature map generation module.

[0041] The attribution heatmap generation module 22 is used to calculate the attribution heatmap based on all channel masks, feature maps and weight vectors.

[0042] Output module 23 is used to output the attribution heatmap.

[0043] The device control module 24 stores the control program that controls the operation of the above modules.

[0044] The following description, in conjunction with the accompanying drawings, illustrates the process of using the deep neural network decision bias mining and visualization device of this embodiment to perform deep neural network decision bias mining and visualization.

[0045] Figure 2 This is a flowchart illustrating the deep neural network decision bias mining and visualization method in an embodiment of the present invention.

[0046] like Figure 2 As shown, the deep neural network decision bias mining and visualization method of this embodiment includes the following steps:

[0047] Step S1: The output result generation module 13 takes the image input through the input module 11 and inputs it into the deep neural network model in the model storage module 12 to obtain the output result. In this embodiment, the output result is the Softmax score for each category.

[0048] Step S2: Select a feature layer of the deep neural network model as the target feature layer through the target selection module 14.

[0049] The target feature layer is the last convolutional layer in the deep neural network model.

[0050] Step S3: The weight vector generation module 15 performs backpropagation based on the image label and output result to calculate the weight vector.

[0051] The weight vector is a vector with a dimension equal to the number of channels in the target feature layer, containing the weights of each channel in the target feature layer. In this embodiment, the weights in the weight vector are used to quantify the importance of the corresponding channels to the model decision.

[0052] The formula for calculating the weight vector is:

[0053]

[0054] In the formula g t For the weight vector, For feature map l t The upper coordinate is (i,j), Z is the normalized term, f L This refers to the results corresponding to the labels in the output.

[0055] Step S4: The channel mask generation module 16 sets an initialization mask for each channel of the target feature layer as the corresponding channel mask. In this embodiment, each channel corresponds to a feature map, and each feature map captures different features in the input image, such as texture, shape, and edge.

[0056] In step S4, the initialization mask is a tensor with the same shape as the target feature layer and has been initialized. The initialization methods include Gaussian noise initialization and all-1 initialization. When Gaussian noise is initialized, the initialization mask corresponding to each channel is different. When all-1 is initialized, the initialization mask corresponding to each channel is the same. In this embodiment, all-1 initialization is used to obtain the corresponding initialization mask.

[0057] Step S5: For each channel, the feature map generated by the masking feature map generation module 17 masks the feature map generated for that channel according to the corresponding channel mask, thereby obtaining the corresponding masking feature map.

[0058] Step S6: The masking output result generation module 18 uses all masking feature maps as the output of the target layer, and processes them through subsequent layers of the deep neural network model to obtain the masking output result.

[0059] Step S7: For each channel mask, the loss function is calculated by the loss function calculation module 19 based on the masking output result, the output result and the channel mask, to obtain the corresponding loss function calculation result.

[0060] The formula for calculating the loss function is as follows:

[0061] ||f mask -f||1+α||m||1+βTV(m),

[0062] TV(m)=∑∑(|I(o,j)-I(o+1,j)|+|I(i,j)-I(i,j+1)|),

[0063] In the formula f mask To mask the output result, f is the output result, m is the channel mask, ||||1 is the L1 regularization term, I(i,j) is the value of the feature map with coordinates (o,j) generated by the channel corresponding to the channel mask, and α and β are both hyperparameters. In this embodiment, the goal of the optimization process is to minimize the loss function as much as possible.

[0064] In this embodiment, ||f mask -f||1 is used to make the masked feature map, containing as few features as possible, as similar as possible to the original result. ||m||1 is used to find a sparser mask, thereby removing features from the feature map that are not important to the decision. TV(m) is the total variation loss, used to smooth the feature mask so that the valuable parts in the sparse mask are as continuous as possible and as concentrated as possible in a specific region, thereby finding regional key information rather than scattered points. That is, the calculation of the total variation loss is based on the gradient information of the feature map and is used to penalize the discontinuity between adjacent feature elements in the feature mask.

[0065] Step S8: The channel mask update module 20 updates all channel masks by backpropagation based on the calculation results of all loss functions.

[0066] In step S9, the update judgment module 21 determines whether the termination condition has been met. If so, the attribution heatmap generation module 22 calculates the attribution heatmap based on all channel masks, feature maps, and weight vectors. If not, steps S5 to S8 are repeated. The termination condition is that the number of channel mask updates reaches a preset number.

[0067] In this embodiment, by repeating steps S5 to S8, cognitive distillation is performed on the feature map based on the channel mask, thereby removing unimportant feature channels from the feature map and extracting the feature channel information that is most important to the final output, revealing the hidden useful features and demonstrating the model's potential recognition mechanism for key information in the original input. Furthermore, this process is unsupervised, requiring no label information and no changes to the model's structure or parameters.

[0068] The process involves multiplying the feature map corresponding to each channel with the channel mask to obtain the corresponding distillation feature map, multiplying each weight in the weight vector with the corresponding distillation feature map to obtain the weight feature map, and summing all the weight feature maps to obtain the attribution heatmap. The expression for the attribution heatmap is as follows:

[0069] CD-CAM = ReLU(g) t ·m·l t ),

[0070] In the formula, CD-CAM is the attribution heatmap, ReLU is the activation function, m is the channel mask, and l t This is the feature map corresponding to the channel mask.

[0071] This embodiment compares the existing Grad-CAM visualization method with the deep neural network decision bias mining and visualization method, namely the CD method.

[0072] Figure 3 This is a comparative schematic diagram of the Grad-CAM visualization method and the CD method in an embodiment of the present invention.

[0073] like Figure 3As shown, (a) is the attribution heatmap generated by the Grad-CAM visualization method based on chest radiograph A and the deep residual network model; (b) is the attribution heatmap generated by the CD method based on chest radiograph A and the deep residual network model; (c) is the attribution heatmap generated by the Grad-CAM visualization method based on chest radiograph B and the deep residual network model; and (d) is the attribution heatmap generated by the CD method based on chest radiograph B and the deep residual network model. The areas within the white rectangles in each figure represent the key regions identified by each method. Therefore, it can be seen that the deep neural network decision bias mining and visualization method of this embodiment can generate a smaller range of key regions for model attribution compared to the existing Grad-CAM visualization method, that is, it can more accurately mine and display the potential cognitive shortcuts of the deep residual network model.

[0074] The role and effect of the embodiments

[0075] According to the deep neural network decision bias mining and visualization method and apparatus involved in this embodiment, on the one hand, the feature map is masked by a channel mask, and cognitive distillation is performed on the feature map. Furthermore, the channel mask is continuously updated during iterative optimization using a loss function, thereby extracting the feature channel information of important parts of the feature map. On the other hand, the importance of each channel to the model decision is obtained by calculating the weight vector of the target feature layer. Then, the feature map after channel masking is integrated based on weights to obtain an accurate attribution heatmap. In summary, this method can obtain visualization results for potential decision biases in deep neural networks.

[0076] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A method for mining and visualizing decision bias in deep neural networks, used to generate attribution heatmaps of images input to a deep neural network model, characterized in that, Includes the following steps: Step S1: Input the image into the deep neural network model to obtain the output result; Step S2: Select a feature layer of the deep neural network model as the target feature layer; Step S3: Perform backpropagation based on the image label and the output result to calculate the weight vector; Step S4: Set an initialization mask for each channel of the target feature layer as the corresponding channel mask; Step S5: For each channel, the feature map generated by the channel is masked using the corresponding channel mask to obtain the corresponding masked feature map; Step S6: Use all the masking feature maps as the output of the target feature layer, and process them through the subsequent layers of the deep neural network model to obtain the masking output result; Step S7: For each channel mask, calculate the loss function based on the masking output result, the output result, and the channel mask to obtain the corresponding loss function calculation result; Step S8: Backpropagate and update all channel masks based on the calculation results of all the loss functions; Step S9: Repeat steps S5 to S8 until the termination condition is met. Then, calculate the attribution heatmap based on all the channel masks, the feature map, and the weight vector.

2. The deep neural network decision bias mining and visualization method according to claim 1, characterized in that: in, In step S3, the weight vector is a vector with dimension equal to the number of channels in the target feature layer, containing the weights of each channel in the target feature layer. The formula for calculating the weight vector is: ， In the formula Let be the weight vector. For feature map The upper coordinate is The value, For the normalized terms, The result corresponding to the label in the output result.

3. The deep neural network decision bias mining and visualization method according to claim 2, characterized in that: in, In step S9, the feature map and channel mask corresponding to each channel are multiplied to obtain the corresponding distillation feature map. Each weight in the weight vector is multiplied by the corresponding distillation feature map to obtain the weight feature map. All weight feature maps are added together to obtain the attribution heatmap. The expression for the attribution heatmap is: ， In the formula For attribution heatmap, For activation function, The channel mask, This is the feature map corresponding to the channel mask.

4. The deep neural network decision bias mining and visualization method according to claim 1, characterized in that: in, In step S7, the formula for calculating the loss function is: ， ， In the formula The output result of the masking is... The output result is as described. The channel mask, for Regularization term, The coordinates of the feature map generated for the channel corresponding to the channel mask are: The value, and All of these are hyperparameters.

5. The deep neural network decision bias mining and visualization method according to claim 1, characterized in that: in, In step S2, the target feature layer is the last convolutional layer of the deep neural network model.

6. The deep neural network decision bias mining and visualization method according to claim 1, characterized in that: in, In step S4, the initialization mask is a tensor with the same shape as the target feature layer and is initialized. The initialization methods include Gaussian noise initialization and all-1 initialization.

7. The deep neural network decision bias mining and visualization method according to claim 1, characterized in that: in, In step S9, the termination condition is that the number of times the channel mask is updated reaches a preset number.

8. A device for mining and visualizing decision bias in deep neural networks, used to generate attribution heatmaps of images input to a deep neural network model, characterized in that, include: The module comprises an input module, a model storage module, an output result generation module, a target selection module, a weight vector generation module, a channel mask generation module, a masking feature map generation module, a masking output result generation module, a loss function calculation module, a channel mask update module, an update judgment module, an attribution heatmap generation module, and an output module. The input module is used to input the image and its corresponding label. The model storage module is used to store the deep neural network model. The output generation module is used to extract the deep neural network model from the model storage module, input the image into the deep neural network model, and obtain the output result. The target selection module is used to select a feature layer of the deep neural network model as the target feature layer. The weight vector generation module is used to calculate the weight vector by performing backpropagation based on the label and the output result. The channel mask generation module is used to generate the initialization mask corresponding to each channel in the target feature layer as the channel mask. The masking feature map generation module is used to mask the feature maps generated for the corresponding channels according to the channel masks, thereby obtaining the corresponding masking feature maps. The masking output generation module is used to take all the masking feature maps as the output of the target feature layer, and then process them through subsequent layers of the deep neural network model to obtain the masking output result. The loss function calculation module is used to calculate the loss function based on the masking output result, the output result, and the channel mask, to obtain the loss function calculation result for the corresponding channel. The channel mask update module is used to update all the channel masks based on the backpropagation of the calculation results of all the loss functions, and then send the updated mask update to the update judgment module. The update judgment module stores preset termination conditions and determines whether the termination conditions are met. If so, the channel mask is sent to the attribution heatmap generation module; otherwise, the channel mask is sent to the masking feature map generation module. The attribution heatmap generation module is used to calculate the attribution heatmap based on all the channel masks, the feature map, and the weight vector. The output module is used to output the attribution heatmap.

Images