A robust model-based feature attribution method with strong category specificity

Abstract

The present invention discloses a feature attribution method with strong category specificity based on a robust model, and belongs to the field of neural network interpretability methods. The present invention constructs a robust image classification model, obtains the initial feature activation mapping diagrams of all categories in the current image and the data set, calculates the correlation matrix between the initial feature activation map of each category and the target category, calculates the specific feature attribution map DFA, and finally obtains the attribution result. The present invention comprehensively considers the correlation between all categories in the data set, thereby improving the category specificity of the feature attribution result; and compared with the prior art, the interpretability of the feature attribution result is improved.

Description

Technical Field

[0001] The present invention provides a posterior interpretation method of a neural network, and specifically relates to a feature attribution method with strong category specificity based on a robust model, belonging to the field of neural network interpretability methods. Background Art

[0002] The posterior interpretability method of deep neural networks provides a proxy for model decision-making through rule extraction technology, explaining the behavior of deep neural networks in a way that is easy for users to understand, such as explaining the output of classification models through visualization of important visual features. Among them, the feature attribution algorithm calculates the importance of each input feature to the model's prediction of the target category after the model predicts the target category, and explains the decision basis of the model in the form of a heat map, which is more intuitive and easy to understand.

[0003] According to the different ways of calculating the importance of features, feature attribution algorithms can be divided into perturbation-based methods and gradient-based methods. The perturbation-based method perturbs the pixels of the input image or the output of the middle layer of the model, and measures the importance of the corresponding pixel features or feature maps by calculating the changes in the model's prediction scores on the target category. Representative methods include LIME (Local Interpretable Model-Agnostic Explanations) and Ablation Class Activation Maps (Ablation-CAM), which perturb the feature maps of the input image and the middle layer of the model, respectively. However, a major disadvantage of this type of method is that it requires multiple model forward passes and intensive calculations, and the algorithm efficiency is low.

[0004] Another type of gradient-based method obtains the importance of each pixel through the gradient of the model output to the input image or feature map. Since the gradient solution only requires one forward propagation and one backpropagation of the model, this type of algorithm has high computational efficiency. Representative ones include Integrated Gradients (IG) and Gradient Weighted Class Activation Mapping (Grad-CAM) and their subsequent improvements. However, existing research shows that these gradient-based feature attribution algorithms have serious class-invariant behavior, that is, "even if the target category of the model feature attribution is changed, the result of the model feature attribution remains almost unchanged." Therefore, in common fine-grained classification tasks and medical disease diagnosis tasks, since the features of objects of different categories to be classified are similar, the existing feature attribution algorithms can only locate the position of the object in the image due to poor class specificity, and cannot provide effective model decision explanations.

[0005] At the same time, existing feature attribution methods rarely consider the properties of the neural network being interpreted. The robust deep neural network obtained by adversarial training has shown significant advantages in model gradient interpretability and feature interpretability. Studies have shown that the model gradient of the adversarial robust model is more consistent with human visual perception. Wang et al. took advantage of the smoother decision boundary of the robust model and designed a boundary attribution method (BoundaryAttributions) to obtain a more interpretable model gradient. However, so far, there are few studies on designing feature attribution methods based on the characteristics of robust models, and they mainly focus on the ability of feature attribution results to locate the edges or pixels of objects. Therefore, there is still a problem of poor category specificity of feature attribution algorithms. Summary of the invention

[0006] In view of the problem that the existing gradient-based feature attribution algorithm has poor category specificity, the present invention provides a distinctive feature attribution algorithm (DFA) based on a robust model. Through the present invention, feature attribution interpretation with strong category specificity can be achieved in similar category image classification tasks.

[0007] The method of the present invention comprises the steps of:

[0008] A class-specific feature attribution method based on a robust model, such as Figure 1 As shown, the steps include:

[0009] A. Build a robust image classification model by:

[0010] A1. Dataset construction. Get training images, including the category label y corresponding to each image x. The constructed dataset is denoted as X:{(x i ,y i )|i∈[1,N]};

[0011] A2. Classification model construction. Select any existing deep learning model structure for image classification or create a new deep learning model structure. The available model architectures include: (1) convolutional neural network, such as ResNet, DenseNet and Xception; (2) Transformer image classification model with category token (cls token), such as VisionTransformer (ViT); (3) Transformer image classification model without category token, such as SwinTransformer. The constructed image classification model is called f Θ (·), where Θ is a trainable parameter;

[0012] A3. Acquisition of adversarial perturbation image data. The specific steps are to use the adversarial attack algorithm "projected gradient descent" (PGD), based on the image dataset constructed in A1 and the classification model built in A2, to obtain adversarial noise perturbation δ and the input image x′ after adversarial noise perturbation in each epoch:

[0013] x ′ =x+δ

[0014]

[0015] Among them, the PGD algorithm is an iterative algorithm, t is the number of algorithm iterations, α is the step size of the gradient descent for each iteration, Indicates that the strength of the adversarial noise perturbation is limited to within the strength ∈. δ represents the adversarial noise, δ t represents the adversarial noise at the t-th iteration step, sign(·) is the sign function, The loss function used for classification model training;

[0016] A4. Model training. In each epoch, first obtain a batch of training data, obtain the image x′ after adversarial noise perturbation based on the method in A3, and input it into the model to obtain the prediction result. According to the classification label y of the original image x, use the loss function Calculate batch loss and update model parameters using stochastic gradient descent;

[0017] A5. Repeat steps A3 to A4 until the batch loss of the model training converges to a stable minimum. The model parameter state at this time is the trained robust image classification model;

[0018] B. Input the image into the model constructed in step A to obtain the initial feature activation map M of the model for the current image and all categories in the dataset. The implementation method is:

[0019] B1. Get the model feature map. If the classification model built in A2 is a convolutional neural network, you can select the output result of any convolutional layer. As the feature map, C is the number of convolution kernels in the convolution layer (or the number of channels of the feature map), H and W are the height and width of the feature map, and Represents the feature map of the i-th channel; if it is a Transformer model containing category tokens, such as ViT, the feature vector shape output by the Transformer block of the model is generally in is the number of tokens input to the ViT model. The +1 part is the cls token set by the ViT model. The features of the cls token are discarded, and the shape of the feature vector is reorganized into L embedding ×N token ×N token , let C = L embedding , H=W=N token , and we also get If it is a Transformer model without category tokens, such as Swin Transformer, the feature shape of the Transformer block output of the model is generally H×W×C. The dimension order of the feature vector is changed to C×H×W, and we get

[0020] B2. For each category k∈{0,1,…,K-1} in the dataset, calculate the initial feature activation map m of the jth category in the dataset j ,j∈{0,1,…,K-1}, the specific calculation method is as follows:

[0021]

[0022] in is the minimum and maximum normalization function, and is calculated as follows:

[0023]

[0024] When the target category of feature attribution is k, the weight of the i-th channel feature map is calculated as follows:

[0025]

[0026] where δ jk is the Kronecker delta function, which takes the value 1 when j is equal to k and 0 otherwise. The result pair feature map A output by the model on the jth category i The coordinate position is (n x ,n y ) The partial derivative of the variable. The initial feature activation map of all categories in the data set is recorded as M = [m0,…,m j ,…,m K-1 ];

[0027] C. Calculate the correlation matrix S between the initial feature activation maps of all categories and the target category. Specifically, calculate each row S of the matrix S k . S krepresents the correlation matrix of the initial feature activation maps of all categories and the target category k, S k The calculation method is as follows:

[0028]

[0029] According to the above formula, calculate S for each target category k in the following way k , and finally get

[0030] D. Calculate the specific feature attribution graph DFA to obtain the final attribution result. That is, given the input image x and the target category k of feature attribution, the DFA is calculated as follows:

[0031] DFA(x,k)=ReLU(S k M T )

[0032] The obtained specific feature attribution map is then upsampled to the size of the input image x by bilinear interpolation to obtain the final attribution result.

[0033] F. Feature attribution result evaluation. The specific implementation steps are:

[0034] F.1 Test data preparation. The data used for testing is a small part of the training data that is not used for classification model training. The test data also contains the category label corresponding to each image.

[0035] F.2 Interpretability evaluation. On the one hand, after visualizing the feature attribution, we can make subjective judgments. We can use heat maps or threshold maps to visualize the feature attribution results. On the other hand, we use quantitative evaluation indicators to evaluate the performance of the feature attribution algorithm in explaining model decisions.

[0036] F.3 Evaluation of the category specificity of feature attribution results. On the one hand, after visualizing the feature attribution, subjective judgment can be made. The feature attribution results can be visualized using heat maps or threshold maps. On the other hand, quantitative evaluation indicators are used to evaluate the correlation of the feature attribution results calculated by the algorithm for the same image and different target categories. The lower the correlation, the higher the category specificity of the algorithm.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) The present invention extracts gradients and feature maps based on a robust image classification model, which improves the interpretability of feature attribution results compared with the prior art that generally extracts features based on models trained by stochastic gradient descent. The gradient of the robust model obtained through adversarial training is more consistent with our human visual perception, and can more sensitively capture the edge information of the object and locate the target object. The interpretability of the model gradient is significantly enhanced after adversarial training.

[0039] (2) The weight of the feature map in the present invention The calculation method comprehensively considers the gradients of all categories. Compared with existing feature attribution methods such as Grad-CAM and XGrad-CAM that only consider the gradients of the target category, it improves the specificity of the category feature activation map.

[0040] (3) The present invention significantly improves the category specificity of feature attribution results by calculating the correlation matrix between the initial feature activation map of each category and the target category. However, the existing feature attribution algorithm only considers the information of the target category and ignores the relationship between the categories in the data set, resulting in poor category specificity of feature attribution results in classification tasks with similar categories. On the contrary, the present invention comprehensively considers the correlation between all categories in the data set, thereby improving the category specificity of feature attribution results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the present invention.

[0042] Figure 2 It is a schematic diagram of a specific example of the present invention.

[0043] Figure 3 : is a schematic diagram of the specific feature attribution result of a specific embodiment of the present invention, wherein the black solid line area is the cell nucleus, and the dotted line area is the cytoplasm. The cell nucleus characteristics of eosinophils and neutrophils are similar, but the cytoplasm of eosinophils is obviously different from that of neutrophils. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention clearer, the specific implementation method of the present invention will be further described below in conjunction with the accompanying drawings.

[0045] Please refer to Figure 2, the present invention provides a feature attribution algorithm with strong category specificity based on a robust model. Specifically, the specific example of the present invention provides implementation steps on the leukocyte classification task. Leukocyte classification includes four categories, namely neutrophils, eosinophils, monocytes and lymphocytes. The image features of these four leukocytes are similar. For example, the nuclei of neutrophils and eosinophils are rod-shaped or divided into 2-5 lobes, which are connected by filaments. However, under staining conditions, there are some category-specific image features: the eosinophilic granules in the cytoplasm of eosinophils appear red or pink, while the cytoplasm of neutrophils is usually light pink or colorless. Monocytes are larger in size, and lymphocytes have very little cytoplasm, which is significantly different from neutrophils and eosinophils.

[0046] The specific implementation steps are:

[0047] A. Build a robust image classification model. The specific implementation steps are:

[0048] A1. Dataset construction. We use the publicly available open-source BCCD dataset for white blood cell image classification. The images in this dataset include four categories: neutrophils, eosinophils, monocytes, and lymphocytes. Each category has about 2,500 images. All data for each category are randomly divided into five parts, four of which are used as training sets and one as a test set. The size of each image is 640×480 pixels, with three RGB channels.

[0049] A2. Classification model construction. The model type selected in this example is a convolutional neural network, and the model architecture is a standard ResNet-34 model. The model parameters are set using the default random initialization method of the PyTorch library.

[0050] A3. Acquisition of adversarial perturbation image data. In this example, the PGD algorithm is used to obtain the adversarial noise perturbation required in the training phase. In each epoch, the adversarial noise perturbation δ and the perturbed input image x′ are obtained:

[0051] x ′ =x+δ

[0052]

[0053] Among them, the PGD algorithm is an iterative algorithm, t is the number of algorithm iterations, α is the step size of the gradient descent for each iteration, It means that the strength of the anti-noise perturbation is limited to within the strength ∈, sign(·) is the sign function, is the loss function used for classification model training; in the specific embodiment, the number of iterations t of the PGD algorithm is set to 5, and the size of the disturbance intensity limit ∈ is The intensity limit mode is l∞ norm, the step size α of gradient descent is The loss function used is the cross entropy loss function.

[0054] A4. Model training. In each epoch, first obtain a batch of training data, obtain the adversarial perturbation image x′ based on the method in A3, and input the model to obtain the prediction result. According to the classification label y of the original image x, use the loss function Calculate the batch loss and use the stochastic gradient descent method to update the model parameters. That is, add adversarial noise perturbations to the images in each batch of data. After obtaining the model output, calculate the cross entropy loss function of the current batch. After back propagation once, use the SGD optimizer to update the model parameters. The batch size is set to 32 and the learning rate is set to 1.0×10 -4 Keeping constant, the momentum parameter of the SGD optimizer is set to 0.9 and the weight decay coefficient is set to 5.0×10 -4 .

[0055] A5. Repeat steps A3 to A4 until the batch cross entropy loss converges. The model at this point is a trained robust white blood cell image classification model.

[0056] B. Input the image into the model obtained in step A, obtain the initial feature activation map M of the model for the current image and all categories in the data set, and sample the obtained feature activation map to the size of the input image x by bilinear interpolation to obtain the final attribution result. The implementation method is:

[0057] B1. Get the feature map extracted by the model corresponding to the input image. The model architecture used, ResNet-34, belongs to convolutional neural network.

[0058] B2. Specifically, the ResNet-34 network contains four residual blocks. In the embodiment of the present invention, the output feature map of the first residual block is particularly used, which is downsampled four times compared to the original input image. This feature map is generated by the last convolutional layer of the first residual block and is obtained after the skip connection of the residual block but before any pooling layer or subsequent convolutional layer.

[0059] B3.The model used in this example is a convolutional neural network.

[0060] B4. Calculate the initial feature activation map m for each category k∈{0,1,2,3} in the dataset j,j∈{0,1,2,3}. The category names corresponding to the k values ​​are eosinophils, neutrophils, monocytes and lymphocytes. The specific calculation method is as follows:

[0061]

[0062] in is the minimum and maximum normalization function, and is calculated as follows:

[0063]

[0064] is the weight of the i-th channel feature map when the target category is k, and is calculated as follows:

[0065]

[0066] The initial feature activation map of the four categories is recorded as M = [m0, m1, m2, m3], and the obtained m j The size is 160×120 pixels.

[0067] C. Calculate the correlation matrix S between the initial feature activation maps of all categories and the target category. The specific steps are:

[0068] C1. Calculate each row S of the matrix S k . S k represents the correlation between the initial feature activation map M and the target category k, S k The calculation method is as follows:

[0069]

[0070]

[0071] C2. Calculate S for each target category k according to C1. k , and finally get

[0072] D. Calculate the specific feature attribution map DFA. Given an input image x and a target category k of feature attribution, the DFA is calculated as:

[0073] DFA(x,k)=ReLU(S k M T )

[0074] The size of the DFA attribution result obtained is 160×120 pixels. The specific feature activation map obtained using the bilinear interpolation algorithm is upsampled back to 640×480 pixels to obtain the final attribution result.

[0075] F. Feature attribution result evaluation. The specific implementation steps are:

[0076] F.1 Test data preparation: Use the images of the test set divided in step A1 for testing.

[0077] F.2 Evaluation of model decision interpretation ability. In terms of subjective evaluation, the obtained feature attribution graph is normalized by the minimum and maximum value function Scale to between 0-1, Figure 3 The grayscale visualization results are given. In terms of quantitative evaluation, three indicators, keep positive mask (KPM), keep absolute mask (KAM) and remove absolute mask (RAM), are used to quantitatively evaluate the interpretability of feature attribution results. The test method of KPM indicator is as follows: given an input image The model's predicted probability p = softmax(f Θ (x)), the predicted probability of target category k is p k =softmax(f Θ (x) k , the feature attribution result of the target category calculate:

[0078]

[0079] in Indicates selection A x In the area where the value is greater than 0, the attribution value is greater than The position of A is assigned a value of 1, and the rest and all pixel positions with negative attribution values ​​are assigned a value of 0. The higher the KPM value, the more likely it is that the feature attribution result can capture the part that has the most positive impact on the model decision. The calculation method of KAM is similar to KPM. The difference is that KAM first calculates A x The absolute value is taken and then calculated in the same way as KPM:

[0080]

[0081] Similarly, the larger the KAM value, the more the feature attribution result captures the features that have the greatest impact on the model decision. Similarly, RAM is calculated as:

[0082]

[0083] RAM and KAM also evaluate the degree of influence of the features captured by the feature attribution results on the model decision results. The smaller the RAM value, the more interpretable the feature attribution results are. The values ​​of the above indicators KPM, KAM and RAM all range from 0 to 1.

[0084] F.3 Category-specific evaluation of feature attribution results. In terms of subjective evaluation, the obtained feature attribution graph is normalized by the minimum and maximum value function Scale to between 0-1, Figure 3 The grayscale visualization results are given. It can be seen that this method can give feature attribution results with strong category specificity when specifying different target categories, and can capture the importance of the eosinophil cytoplasm region and the importance of the neutrophil nucleus respectively. In terms of quantitative evaluation, the Spearman correlation coefficient between the feature attribution results calculated by the algorithm for different target categories of the same image is calculated and the average is taken. The Spearman correlation coefficient ranges from -1 to 1. The lower the correlation coefficient, the stronger the category specificity. All images in the test set are evaluated and the average of the results on all images is taken.

[0085] Table 1 Quantitative evaluation results of the interpretability of feature attribution algorithm results

[0086] The present invention Gradient IG Guided IG Blur IG Grad-CAM XGrad-CAM BagCAM KPM(↑) 0.1971 0.0240 0.0332 0.0037 0.0585 0.1584 0.1816 0.0943 KAM(↑) 0.8879 0.7776 0.7689 0.7953 0.7916 0.8487 0.7775 0.8254 RAM(↓) 0.8725 0.9567 0.9552 0.9604 0.9548 0.9155 0.8975 0.9445

[0087] Table 2 Quantitative evaluation results of category specificity of feature attribution algorithm results

[0088]

[0089] The above embodiments are only some preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any changes made by adopting the design principles of the present invention and performing non-creative work on this basis should fall within the protection scope of the present invention.

Claims

1. A feature attribution method with strong category specificity based on a robust model, the steps of which include: A. Build a robust image classification model by: A1. Dataset construction: Get training images, including the category label y corresponding to each image x, and construct a dataset denoted as X: {(x i ,y i )|i∈[1,N]}; A2. Classification model construction: Select any existing deep learning model structure for image classification or create a new deep learning model structure. The built image classification model is called f Θ (·), where Θ is a trainable parameter; A3. Acquisition of adversarial perturbation image data: The specific steps are to use the PGD algorithm to obtain the adversarial noise perturbation required in the training phase. Based on the image dataset constructed in A1 and the classification model built in A2, the adversarial noise perturbation δ and the perturbed input image x′ are obtained in each epoch: x′=x+δ Among them, the PGD algorithm is an iterative algorithm, t is the number of algorithm iterations, α is the step size of the gradient descent for each iteration, It means that the strength of the anti-noise perturbation is limited to within the strength ∈, sign(·) is the sign function, The loss function used for classification model training; A4. Model training: In each epoch, first obtain a batch of training data, obtain the adversarial perturbation image x′ based on the method of A3, and input the model to obtain the prediction result According to the classification label y of the original image x, use the loss function Calculate batch loss and update model parameters using stochastic gradient descent; A5. Repeat steps A3 to A4 until the batch loss of the model training converges to a stable minimum value. At this time, the model parameter state is a trained robust image classification model. B. Input the image into the model constructed in step A to obtain the model's initial feature activation map M for the current image and all categories in the dataset; C. Calculate the correlation matrix S of the initial feature activation maps of all categories and the target category; D. Calculate the specific feature attribution map DFA. Given an input image x and a target category k of feature attribution, the DFA is calculated as: DFA(x, k) = ReLU(S k M T ), and the final attribution result is obtained.

2. The feature attribution method with strong category specificity based on a robust model as claimed in claim 1, characterized in that: The deep learning model structure in step A2 includes: a convolutional neural network, a Transformer image classification model with category tokens, or a Transformer image classification model without category tokens.

3. The feature attribution method with strong category specificity based on a robust model as claimed in claim 1, characterized in that: The implementation method of step B is: B1. Obtain model feature map; B2. Calculate the initial feature activation map m for each category k∈{0, 1, ..., K-1} in the dataset j , j∈{0, 1, …, K-1}, the specific calculation method is as follows: is the weight of the i-th channel feature map when the target category is k, and is calculated as follows: where δ jk is the Kronecker delta function, which takes the value 1 when j is equal to k and takes the value 0 otherwise. The result pair feature map A output by the model on the jth category i The coordinate position is (n x , n y ) variable, and the initial feature activation map of all categories is M = [m0, ..., m j ,…,m K-1 ].

4. The method for attributing features with strong category specificity based on a robust model as claimed in claim 3, characterized in that: If the classification model built in A2 is a convolutional neural network, select the output result of any convolutional layer As the model feature map, C is the number of convolution kernels in the convolution layer, slices and W are the height and width of the feature map, and Represents the feature map of the i-th channel.

5. The feature attribution method with strong category specificity based on a robust model as claimed in claim 3, characterized in that: If the classification model built in A2 is a Transformer model with category tokens, the feature vector shape output by the Transformer block of the model is generally in is the number of tokens input to the model. The +1 part is the cls token set by the model. The features of the cls token are discarded, and the shape of the feature vector is reorganized into L embedding ×N token ×N token , let C = L embedding , H=W=N token , and we also get as the model feature map.

6. The method for attributing features with strong category specificity based on a robust model as claimed in claim 3, characterized in that: If the classification model built in A2 is a Transformer model without category tokens, the feature shape of the Transformer block output of the model is generally slice×W×C. The dimension order of the feature vector is changed to C×slice×W, and we get as the model feature map.

7. The method for attributing features with strong category specificity based on a robust model as claimed in claim 1, characterized in that: In step C, each row S of the matrix S is first calculated k , S k represents the correlation between the initial feature activation map M and the target category k, S k The calculation method is as follows: Calculate S for each target category k according to the above formula k , and finally get 8. The method for attributing features with strong category specificity based on a robust model as claimed in claim 1, characterized in that: After step D, the feature attribution result evaluation is further performed. The specific implementation steps are as follows: F.1 Test data preparation: The data used for testing is the part of the training data that is not used for image classification model training. The test data also contains the category label corresponding to each image; F.2 Interpretability evaluation: On the one hand, the feature attribution results are visualized and then judged subjectively. The feature attribution results are visualized using heat maps or threshold maps. On the other hand, quantitative evaluation indicators are used to evaluate the performance in explaining model decisions. F.3 Category-specific evaluation of feature attribution results: On the one hand, the feature attribution results are visualized and then judged subjectively. The feature attribution results are visualized using heat maps or threshold maps. On the other hand, quantitative evaluation indicators are used to evaluate the same image and different target categories and calculate the correlation of the feature attribution results.

Images