Evaluation method of image classification-oriented convolutional neural network interpretation method

Abstract

The invention relates to an evaluation method of an image classification-oriented convolutional neural network interpretation method, belongs to the field of artificial intelligence interpretability, and solves the technical problems that the method in the prior art depends on subjective judgment of people and evaluation is performed through a single index, so that the evaluation result has deviation and instability. According to the method, five quantifiable dimensions are selected based on the properties of the interpretation method to evaluate and select the interpretation method, the evaluation process can be ensured to be comprehensive and have a theoretical basis, the result is accurate, and the stability is good.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence interpretability, and in particular to an evaluation method for a convolutional neural network interpretation method for image classification. Background Art

[0002] In the current development of artificial intelligence, interpretability has become a crucial research topic. Although neural network models have demonstrated remarkable performance in improving algorithm generalization, their lack of interpretability often raises questions about their application in high-risk areas. For example, a black-box medical diagnostic model lacking interpretability can lead to incorrect treatments and even pose a serious threat to a patient's life. To address this issue, researchers have proposed a variety of interpretation methods, such as feature attribution and feature visualization, which aim to explain neural network models from different perspectives. These methods can enhance model understanding and improve interpretability. With the introduction of more interpretation methods, the options available to people are increasing. However, this also raises a new challenge: how to choose when different interpretation methods provide inconsistent explanations for the same neural network model. This is the problem of comparing and selecting interpretation methods, also known as the evaluation problem.

[0003] While research has been conducted on methods for evaluating interpretation methods from both qualitative and quantitative perspectives, most qualitative evaluations rely on subjective judgment, which can lead to unstable results. Furthermore, most quantitative evaluation methods rely on a single metric, which can lead to biased results and fail to fully reflect the actual effectiveness of the interpretation method. Summary of the Invention

[0004] In view of the above problems, the present invention provides an evaluation method for a convolutional neural network interpretation method for image classification. The present invention solves the technical problem that the existing method uses a single indicator to evaluate the interpretation method, resulting in deviation and instability in the evaluation results.

[0005] The present invention provides an evaluation method for a convolutional neural network interpretation method for image classification, which specifically includes the following steps:

[0006] Step 100: Obtain a convolutional neural network model for image classification to be explained and an input image set;

[0007] Step 101: Based on the convolutional neural network model for image classification to be interpreted and the input image set, obtain the index values ​​corresponding to the fidelity dimension, robustness dimension, localization dimension, complexity dimension, and sensitivity dimension of the convolutional neural network model under various interpretation methods;

[0008] Step 102: Normalize the index values of the loyalty dimension, robustness dimension, localization dimension, complexity dimension, and sensitivity dimension under each interpretation method obtained in Step 101 to obtain the normalization results of the five index values of each interpretation method of the model to be interpreted;

[0009] Step 103: Rank the normalization results of the different dimension indexes of each interpretation method obtained in Step 102, draw a radar chart according to the ranking, and obtain the comprehensive score of each interpretation method through the area of the radar chart;

[0010] Step 104: Analyze and compare the interpretation methods according to the comprehensive scores of each interpretation method obtained in Step 103, and select the optimal model interpretation method.

[0011] Optionally, the loyalty dimension is reflected by the relevant loyalty index Faithfulness(g,f,x,S), and the expression is:

[0012]

[0013] where g represents the interpretation method; f is the convolutional neural network model; x is the picture sample to be interpreted in the input image set; S is the feature subset, representing a set of randomly selected partial pixels from the sample x in the input image set; Faithfulness(.) represents the relevant loyalty value; g(f,x i ) is the interpretation result of the interpretation method, which is the attribution value of the i-th pixel x i of the sample x in the S feature subset; f(.) is the prediction value of the convolutional neural network model f; is the perturbed sample after perturbing the feature subset S of the sample x in the input image set, represents the average value of all pixels of the feature subset S; Correlation(.) represents the correlation coefficient, usually referring to the Spearman correlation coefficient.

[0014] Optionally, the robustness dimension is reflected by the maximum sensitivity value SENS MAX (g,f,x,r), and the expression is:

[0015]

[0016] where g represents the interpretation method; f is the convolutional neural network model; x is the picture sample to be interpreted in the input image set; y represents the sample after perturbing the sample to be interpreted x, r represents the neighborhood radius of the perturbation; g(f,x) represents the interpretation result of the interpretation method on the neural network model f under the sample x in the input image set, specifically the attribution values of all pixels of the sample x; ||·|| represents the function used to measure the difference, max represents the maximum value; ||y - x|| < r is the neighborhood constraint satisfied by the perturbed sample y.

[0017] Optionally, the positioning dimension is reflected by the relevant ranking accuracy RankAccuracy, which is expressed as:

[0018]

[0019] Among them, RankAccuracy represents the accuracy value of the relevant ranking; P topK ={p1,p2,…,p K ∣R p1 >R p2 >…>R pK} represents the set of pixels with the highest attribution values ​​selected from the first K pixels, R pi represents the attribution value of the i-th pixel; GT represents the true attribution value of the manually labeled pixel, |P tppK ∩GT| represents the number of intersections between the top K pixels with the highest attribution values ​​and GT, and |GT| represents the total number of pixels whose true attribution value GT is manually annotated.

[0020] Optionally, the complexity dimension is represented by Sparseness(v), which is expressed as:

[0021]

[0022] Among them, Sparseness(.) represents the sparsity value; v represents the pixel attribution vector of the sample to be explained; v (k) represents the kth element in the pixel attribution vector v of the sample to be interpreted, sorted from small to large; d represents the total number of pixels, and (2k-d-1) represents the weight assigned to the pixel attribution value related to its sorting position k.

[0023] Optionally, the sensitivity dimension is reflected by the model parameter randomization test index MPRT(g,f,x), which is expressed as:

[0024] MPRT(g,f,x)=SIM(g(f,x),g(f rand ,x));

[0025] Among them, MPRT(.) represents the interpretation result of the convolutional neural network model f and the neural network model f after randomization of model parameters. rand The similarity measure between the interpretation results of g represents the interpretation method; f represents the convolutional neural network model; x represents the image sample to be interpreted in the input image set; g(f,x) represents the attribution value of all pixels of the neural network model f under the sample x by the interpretation method; g(f rand ,x) represents the explanation method for the randomized neural network model f randAll pixel attribute values ​​under sample x; SIM(.) is a function used to measure the similarity between two explanation results.

[0026] Compared with the prior art, the present invention has at least the following beneficial effects:

[0027] The present invention selects five quantifiable dimensions to evaluate interpretation methods based on their properties. The criteria for measuring the quality of interpretation methods, which reveal the causal relationships behind things, include accuracy, fidelity, robustness, localization, complexity, comprehensibility, novelty, sensitivity, and representativeness. However, not all of these properties are suitable for quantification. Some properties, such as comprehensibility and representativeness, are difficult to measure using specific indicators. Therefore, the present invention identifies measurable and important dimensions as evaluation dimensions for interpretation methods, including fidelity, robustness, localization, complexity, and sensitivity. Fidelity refers to whether the explanation is consistent with the internal logic of the intelligent software. Users trust explanations that faithfully reflect the model's behavior more. Robustness refers to whether the explanation remains consistent even with small changes in the input data. Higher robustness reduces user confusion and provides reliable explanations in the presence of adversarial attacks, enhancing system security. Localization determines the ability of the explanation method to accurately locate the core areas that influence the model's decision-making. For image classification tasks, the explanation method needs to be able to clearly identify the most relevant feature areas. Complexity reflects the simplicity of the explanation method, indicating whether the intelligent software's prediction can be explained by a small number of key features. An explanation method with high sparsity can present the intelligent software's behavior in a concise and clear manner, thereby improving users' understanding of the intelligent software's decision-making process. Sensitivity determines and identifies the explanation method's response to small changes in the input, such as changes in the model parameters to be explained, revealing potential vulnerabilities or feature dependencies. Based on this, a specific calculation metric is proposed for each dimension to ensure that the evaluation process is both comprehensive and theoretically based, and the results are accurate and stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Flowchart of the evaluation method of the convolutional neural network interpretation method of the present invention;

[0029] Figure 2 This is an example of the interpretation results of the evaluation method of the convolutional neural network interpretation method of the present invention;

[0030] Figure 3 A radar chart comparing indicators of the evaluation method of the convolutional neural network interpretation method of the present invention. DETAILED DESCRIPTION

[0031] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited by the specific embodiments disclosed below.

[0032] A specific embodiment of the present invention, as Figure 1-3 , discloses an evaluation method for a convolutional neural network interpretation method for image classification, specifically comprising the following steps:

[0033] Step 100: Obtain a convolutional neural network model for image classification to be explained and an input image set.

[0034] It's understood that a convolutional neural network model refers to a model that relies on machine learning models to generate intelligent behavior. A convolutional neural network model for image classification is a deep learning model specifically designed for image tasks. It efficiently extracts spatial features through local perception and parameter sharing, and typically includes convolutional layers, pooling layers, and fully connected layers. Image sets serve as input data for convolutional neural network modeling.

[0035] Exemplarily, interpretation methods of convolutional neural network models include but are not limited to Grad-CAM, Saliency, Integrated Gradients, LIME, Guided Backpropagation, etc.

[0036] Step 101: Based on the convolutional neural network model for image classification to be interpreted and the input image set, five index values ​​corresponding to the fidelity dimension, robustness dimension, localization dimension, complexity dimension and sensitivity dimension of the convolutional neural network model under various interpretation methods are obtained.

[0037] Specifically, the loyalty dimension is reflected by the relevant loyalty indicator Faithfulness (g, f, x, S), which is expressed as follows:

[0038]

[0039] Where g represents the interpretation method; f is the convolutional neural network model; x is the image sample to be interpreted in the input image set; S is the feature subset, which represents a set of pixels randomly selected from the sample x in the input image set; Faithfulness(.) represents the relevant loyalty value; g(f,x i ) is the interpretation result of the interpretation method, and is the i-th pixel x of sample x in the S feature subset. iThe attribution value; f(.) is the predicted value of the convolutional neural network model f; is the perturbed sample of the feature subset S of the sample x in the input image set, represents the average value of all pixels of the feature subset S; Correlation(.) represents the correlation coefficient, usually referring to the Spearman correlation coefficient.

[0040] The correlation loyalty index adopted by this invention has important practical value in the evaluation of the interpretation method. Through this index, the consistency between the interpretation result and the prediction behavior of the neural network model can be quantified. If the value of the correlation loyalty index is positive and close to 1, it means that the interpretation method has a high loyalty, that is, the interpretation result is highly consistent with the decision-making mechanism inside the neural network model. On the contrary, if the value of the loyalty index is low or negative, it indicates that the interpretation method may not effectively capture the true behavior of the neural network model, and the interpretation result may be unreliable.

[0041] Specifically, the robustness dimension is reflected by the maximum sensitivity value SENS MAX (g, f, x, r), and the expression is:

[0042] [[ID=1s]]

[0043] Among them, g represents the interpretation method; f is the convolutional neural network model; x is the picture sample to be interpreted in the input image set; y represents the sample after perturbing the sample x to be interpreted, such as adding Gaussian noise to the image, r represents the neighborhood radius of the perturbation; g(f, x) represents the interpretation result of the interpretation method for the neural network model f under the sample x in the input image set, specifically the attribution value of all pixels of the sample x; ||·|| represents the function used to measure the difference, max represents the maximum value; ||y - x|| < r is the constraint satisfied by the perturbed sample y.

[0044] The maximum sensitivity index is of great significance in evaluating the robustness of the interpretation method. In practical applications, the interpretation method should maintain the stability of its interpretation result when the input data undergoes minor perturbations. By calculating the maximum sensitivity of different input points, the stability of the interpretation method in the face of input perturbations can be judged. A robust interpretation method should exhibit a small maximum sensitivity, that is, its interpretation result should not fluctuate significantly when the input changes slightly.

[0045] Specifically, the localization dimension is reflected by the correlation ranking accuracy Rank Accuracy, and the expression is:

[0046]

[0047] Among them, Rank Accuracy represents the correlation ranking accuracy value; P topK={p1,p2,…,p K ∣R p1 >R p2 >…>R pK} represents the set of pixels with the highest attribution values ​​selected from the first K pixels, R pi represents the attribution value of the i-th pixel; GT represents the true attribution value of the manually labeled pixel, |P topK ∩GT| represents the number of intersections between the top K pixels with the highest attribution values ​​and GT, and |GT| represents the total number of pixels whose true attribution value GT is manually annotated.

[0048] The correlation ranking accuracy metric of our invention effectively measures the ability of an interpretation method to identify the most important pixels in the neural network model's predictions. A high correlation ranking accuracy indicates that the interpretation method is able to accurately capture the most critical parts of the neural network model's predictions.

[0049] Specifically, the complexity dimension is reflected by Sparseness(v), which is expressed as:

[0050]

[0051] Among them, Sparseness(.) represents the sparsity value; v represents the pixel attribution vector of the sample to be explained; v (k) It represents the kth element in the pixel attribution vector v of the sample to be explained, sorted from small to large by attribution value; d represents the total number of pixels; (2k-d-1) represents the weight of the attribution value associated with the sort k.

[0052] The sparsity metric of this invention can be used to assess the performance of an interpretation method in terms of feature selection and feature contribution distribution. An interpretation method with high sparsity, where its interpretation results are concentrated on a small number of important features, generally indicates that the neural network model's predictions rely on these key features, reflecting lower complexity.

[0053] Specifically, the sensitivity dimension is reflected by the model parameter randomization test index MPRT(g,f,x), which is expressed as:

[0054] MPRT(g,f,x)=SIM(g(f,x),g(f rand ,x));

[0055] Among them, MPRT(.) represents the difference between the interpretation result of convolutional neural network model f and the randomized model f rand The similarity measure between the interpretation results; g represents the interpretation method; f is the convolutional neural network model; x is the image sample to be interpreted in the input image set; f rand represents the neural network model after randomization of model parameters, f randIt is a model generated by traversing the convolutional neural network model f layer by layer and randomizing the parameters; g(f,x) represents the attribution value of all pixels of the neural network model f under sample x by the interpretation method; g(f rand ,x) represents the explanation method for the randomized neural network model f rand All pixel attribution values ​​under sample x; SIM(.) is a function used to measure the similarity between two explanation results, usually using the Spearman correlation coefficient.

[0056] The model parameter randomization test index of the present invention can effectively verify whether the interpretation method really depends on the internal parameters of the model. The core assumption of the interpretation method (such as feature attribution) is: "The interpretation result (such as feature importance score) should directly reflect the actual calculation logic of the model parameters". In the image classification model, the interpretation method gives a high attribution value to certain pixels of the image, which should be due to the actual weighting of the model parameters to this area. By actively destroying the model parameters and observing the changes in the interpretable results, if the interpretation results change significantly with the parameters, it means that the interpretation method really depends on the parameters of the model, which means that its interpretation results are consistent with the actual working mechanism of the model, and the interpretation method is good; if the interpretation results remain stable after the model parameters are randomized, it may indicate that the interpretation method has not effectively captured the internal mechanism of the neural network model.

[0057] Repeat step 101 to calculate the five-dimensional index values ​​of all interpretation methods to be evaluated (each interpretation method)

[0058] Step 102: Normalization processing Step 101 obtains the index values ​​of the loyalty dimension, robustness dimension, positioning dimension, complexity dimension and sensitivity dimension of each interpretation method, and obtains the normalized results of the five index values ​​of each interpretation method of the model to be interpreted.

[0059] Specifically, the maximum value normalization is used for the scores of all dimensions except robustness, and the minimum value normalization is used for the robustness dimension score, so that the scores of each indicator are standardized to the range of 0 to 1.

[0060] Step 103: Rank the normalized results obtained under each interpretation method in step 102, draw a radar chart based on the ranking, and obtain the comprehensive score of each interpretation method through the area of ​​the radar chart.

[0061] Specifically, the ranked scores are mapped onto the various axes of the radar chart, forming polygons with different interpretation methods, which facilitates the evaluation of their overall performance. Finally, by calculating the area of ​​the radar chart, the multi-dimensional comprehensive performance is quantified into a single area value.

[0062] Step 104: Analyze and compare the interpretation methods based on the comprehensive scores of the interpretation methods obtained in step 103, and select the optimal model interpretation method.

[0063] Specifically, by comparing the areas of the radar charts, we can compare the comprehensive performance of different interpretation methods across multiple dimensions. The larger the area, the better the overall performance of the interpretation method across each evaluation dimension. This quantitative analysis can clarify the relative strengths and weaknesses of different interpretation methods, providing a scientific basis for selecting the most appropriate interpretation method in practical applications. If you want an interpretation method with good overall performance, choose the one with the largest radar chart area. If you have high requirements for a particular dimension, you can choose the interpretation method with the highest value in that dimension based on the evaluation results.

[0064] In addition, if all explanation methods have low scores on a certain dimension, the explanation methods should be replaced or further optimized. For generally low fidelity scores, it means that the explanation methods do not match the actual decision logic of the model. The explanation methods should be modified and, combined with the structural characteristics of convolutional neural networks, more compatible explanation methods should be designed to achieve better explanation fidelity. For generally low robustness scores, it means that the explanation methods are sensitive to input. An integration strategy can be used to combine multiple explanation methods to reduce the vulnerability of a single method. For generally low localization scores, the explanation results of the explanation methods can be post-processed, and pixel attribution values ​​can be threshold filtered or clustered to highlight significant areas. For generally low complexity scores, regularization terms can be introduced into the explanation methods to further enhance the sparsity of the explanation results. For generally low sensitivity scores, it means that the explanation methods are not responsive enough to changes in model parameters. Model gradients, parameters, and other information can be further introduced into the explanation methods to enhance the coupling between the explanation methods and model parameters.

[0065] In order to illustrate the effectiveness of the method proposed in the present invention, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) dataset is selected as the experimental dataset. The dataset contains 1,281,167 training images, 50,000 verification images, and 100,000 test images. Each image is labeled with at least one object category, covering a total of 1,000 categories, covering a wide range of categories, including animals, daily necessities, and transportation. Using the ResNet18 pre-trained model as the neural network model to be interpreted, the evaluation method process of the convolutional neural network interpretation method for image classification provided by the present invention is described. The specific implementation steps are as follows:

[0066] Step 1: Determine the neural network model to be interpreted, the corresponding data set and the interpretation method to be evaluated. In this example, the ResNet18 pre-trained model in the torchvision library is selected as the neural network model to be interpreted. ResNet18 is a classic model in the Residual Network (ResNet) series and is widely used in various computer vision tasks. The structure of ResNet18 is simple and effective, and is particularly suitable for image classification in resource-constrained environments. The ImageNetLarge Scale Visual Recognition Challenge (ILSVRC) dataset is selected as the experimental dataset. The four interpretation methods, LayerGradCam, GradientShap, Saliency and Integrated Gradients provided in the Captum library are used to implement the evaluation method of the neural network model interpretation method proposed in this invention.

[0067] Step 2: Calculate the evaluation indicators of the four interpretation methods for the ResNet18 neural network model in terms of fidelity, robustness, localization, complexity, and sensitivity. 1,000 images were randomly selected from the test set as experimental objects. The results of the four interpretation methods are shown in the following example: Figure 2 As shown in the figure, the hyperparameters for calculating each metric are set as follows: When calculating the relevant loyalty metric, the average of multiple loyalty values ​​is calculated, so the number of runs is set to 10 and the feature subset size is set to 224. When calculating the maximum sensitivity metric, the number of samples is set to 10, and the perturbation method is uniform noise perturbation. The scores for the five dimensions are obtained for different interpretation methods.

[0068] Step 3: Perform normalization and ranking calculation on the calculated results of each indicator.

[0069] First, for the scores of all dimensions except robustness, maximum normalization is applied, that is, each value is divided by the maximum value of the column so that the normalized value is between 0 and 1. For the robustness score, since lower values ​​indicate better performance, minimum normalization is used, and the minimum value of the column is divided by each value, so that smaller original values ​​correspond to larger normalized values.

[0070] On this basis, the normalized values ​​are ranked from small to large, with larger ranking values ​​indicating better interpretation methods. Then, the ranked scores are mapped onto a radar chart, with each dimension corresponding to an axis of the radar chart. The final scores form a pentagon for each interpretation method, as shown in the figure below. Figure 3As shown in the figure, the radar chart area of ​​the LayerGradCam interpretation method is 5.8252, the area of ​​the Saliency interpretation method is 25.4408, the area of ​​the GradientShap interpretation method is 16.4057, and the area of ​​the Integrated Gradients interpretation method is 13.9092.

[0071] Step 4: Analyze, compare, and select the interpretation methods based on the processing results. The size of the radar chart area can be used as a quantitative indicator of the performance of the interpretation method. The larger the area, the better the overall performance of the method in all dimensions.

[0072] from Figure 3 As can be seen from the results, the Saliency method has a significant advantage in overall performance, with its radar chart area reaching the largest, reaching 25.4408, far exceeding that of other methods. This result demonstrates that Saliency offers the most balanced performance in a comprehensive multi-dimensional evaluation, particularly in sensitivity and robustness. Although Saliency has some limitations in fidelity and complexity, these do not significantly impact its overall performance. Therefore, if the task requires high overall performance from the explanation method and does not prioritize any particular dimension, Saliency will be the optimal explanation method.

[0073] In contrast, while GradientShap and Integrated Gradients have certain advantages in individual dimensions, their overall performance falls short of Saliency. GradientShap has a strong advantage in complexity, but performs weakly in localization and robustness, with a radar chart area of ​​16.4057, ranking second. Integrated Gradients excels in fidelity, achieving the highest score, but significantly underperforms in localization, complexity, and sensitivity, with a radar chart area of ​​13.9092, ranking third. Therefore, while both methods have certain advantages when focusing on specific metrics, Saliency remains the most balanced and comprehensive interpretation method in terms of overall performance.

[0074] LayerGradCam's radar chart area is 5.8252, ranking lowest among all methods, indicating that its overall performance is inferior to other methods. Although LayerGradCam's localization ranking is high (the same as Saliency), and it can accurately locate the key feature areas in the neural network model's decision-making, it does not show sufficient advantages in terms of comprehensive performance across multiple dimensions. Therefore, unless the task has particularly high requirements for localization, LayerGradCam is not recommended as the preferred interpretation method in most scenarios.

[0075] In summary, without focusing on a particular dimension, Saliency is the optimal explanation method for the ResNet-18 pre-trained model on the ImageNet dataset. Not only does it perform well across multiple dimensions, but its robustness and sensitivity make it highly applicable in most application scenarios.

[0076] Based on the above description, the present invention constructs an evaluation index system for neural network model interpretation methods based on the various properties of interpretation methods. This system considers five aspects: whether the results of the interpretation methods are consistent with the decision-making process of the neural network model, whether they can resist small perturbations in the input data, whether they can accurately locate important features, the degree of simplicity, and whether they depend on model parameters. Five comparative dimensions are determined: fidelity, robustness, localization, complexity, and sensitivity, and specific indicators are proposed for each dimension. The values ​​of each indicator of different interpretation methods are calculated to form a radar chart. The effectiveness of the interpretation method is measured by calculating the area of ​​the radar chart of each interpretation method to solve the problem of how to choose when different interpretation methods give inconsistent interpretation results for the same neural network model.

[0077] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. An evaluation method for a convolutional neural network interpretation method for image classification, characterized in that: Specifically, it includes the following steps: Step 100: Obtain a convolutional neural network model for image classification to be explained and an input image set. Step 101: Based on the convolutional neural network model for image classification to be explained and the input image set, obtain the index values corresponding to the faithfulness dimension, robustness dimension, localization dimension, complexity dimension, and sensitivity dimension of the convolutional neural network model under each explanation method. Step 102: Normalize the index values of the faithfulness dimension, robustness dimension, localization dimension, complexity dimension, and sensitivity dimension obtained in Step 101 under each explanation method to obtain the normalization results of the five index values of each explanation method for the model to be explained. Step 103: Rank the normalization results of the different dimension indexes of each explanation method obtained in Step 102, draw a radar chart according to the ranking, and obtain the comprehensive score of each explanation method through the area of the radar chart. Step 104: Analyze and compare the explanation methods according to the comprehensive scores of each explanation method obtained in Step 103, and select the optimal model explanation method.

2. The evaluation method according to claim 1, wherein The faithfulness dimension is reflected by the relevant faithfulness index Faithfulness(g,f,x,S), and the expression is: Where g represents the interpretation method; f is the convolutional neural network model; x is the image sample to be interpreted in the input image set; S is the feature subset, which represents a set of pixels randomly selected from the sample x in the input image set; Faithfulness(.) represents the relevant loyalty value; g(f,x i ) is the interpretation result of the interpretation method, and is the i-th pixel x of sample x in the S feature subset. i The attribution value of ; f(.) is the predicted value of the convolutional neural network model f; is the perturbed sample after perturbing the feature subset S of the sample x in the input image set, Represents the average value of all pixels in the feature subset S; Correlation(.) represents the correlation coefficient, usually refers to the Spearman correlation coefficient.

3. The evaluation method according to claim 1, wherein: The robustness dimension is measured by the maximum sensitivity value SENS MAX (g,f,x,r) is expressed as: Among them, g represents the explanation method; f is the convolutional neural network model; x is the picture sample to be explained in the input image set; y represents the sample after perturbing the sample x to be explained, r represents the neighborhood radius of the perturbation; g(f,x) represents the explanation result of the explanation method for the neural network model f under the sample x of the input image set, specifically the attribution value of all pixels of the sample x; ||·|| represents the function used to measure the difference, max represents the maximum value; ||y - x|| < r is the neighborhood constraint satisfied by the perturbed sample y.

4. The evaluation method according to claim 1, wherein: The localization dimension is reflected by the relevant ranking accuracy RankAccuracy, and the expression is: Among them, RankAccuracy represents the relevant ranking accuracy value; represents the set of pixels with the highest attribution values ​​selected from the first K pixels, represents the attribution value of the i-th pixel; GT represents the true attribution value of the manually labeled pixel, |P top K ∩GT| represents the number of intersections between the top K pixels with the highest attribution values ​​and GT, and |GT| represents the total number of pixels whose true attribution value GT is manually annotated.

5. The evaluation method according to claim 1, wherein: The complexity dimension is reflected by the sparsity Sparseness(v), and the expression is: Among them, Sparseness(.) represents the sparsity value; v represents the pixel attribution vector of the sample to be explained; v (k) represents the kth element in the pixel attribution vector v of the sample to be interpreted, sorted from small to large; d represents the total number of pixels, and (2k-d-1) represents the weight assigned to the pixel attribution value related to its sorting position k.

6. The evaluation method according to claim 1, wherein: The sensitivity dimension is reflected by the model parameter randomization test index MPRT(g,f,x), and the expression is: MPRT(g,f,x)=SIM(g(f,x),g(f rand ,x)); Among them, MPRT(.) represents the interpretation result of the convolutional neural network model f and the neural network model f after randomization of model parameters. rand The similarity measure between the interpretation results of g represents the interpretation method; f represents the convolutional neural network model; x represents the image sample to be interpreted in the input image set; g(f,x) represents the attribution value of all pixels of the neural network model f under the sample x by the interpretation method; g(f rand ,x) represents the explanation method for the randomized neural network model f rand All pixel attribute values ​​under sample x; SIM(.) is a function used to measure the similarity between two explanation results.

Images