A method for explaining classification decisions of convolutional neural networks for image recognition

Abstract

The present invention discloses an interpretation method for convolutional neural network image recognition classification decision, firstly, an image perturbed by a feature map is input into a neural network model, and a neuron activation value is obtained through a classification output layer; then, the neuron activation value is divided by a penalty factor corresponding to the feature map, and then input into a softmax layer to obtain a score of a target class as a weight of the feature map, and finally, an interpretation map is obtained according to the weights of a plurality of the feature maps. The present invention realizes the difference between the average values ​​of feature maps as required by a parameter, adjusts the importance of the softmax function to enlarge / reduce features, greatly reduces the amount of calculation, and the method can generate a heat map and an interpretation map according to the standard desired by the user, has very good applicability and very high interpretation credibility, and in addition, the method can be used for any convolutional neural network without restrictions on the model structure.

Description

Technical Field

[0001] The present invention relates to the technical field of visual image recognition, and more specifically to a method for explaining convolutional neural network image recognition classification decisions. Background Art

[0002] In recent years, neural networks have achieved good application results in more and more fields, especially convolutional neural networks have achieved excellent results in areas such as image recognition and classification. However, as the tasks to be processed become more and more complex, the structure of convolutional neural networks has become more and more complex. At present, the network structure of most convolutional neural networks is too complex, making it impossible for people to intuitively understand the decision-making logic rules of these convolutional neural networks. Therefore, most convolutional neural networks are currently a "black box" for people, which greatly reduces the credibility of convolutional neural networks. Since people cannot understand the decision logic of neural networks, they cannot judge whether the neural network makes decisions according to the expected logical rules, which greatly reduces the possibility of developers discovering errors in neural networks, and then the neural network may make wrong decisions when applied. This kind of neural network that people cannot intuitively understand and may make wrong decisions will cause people's distrust, and may also bring problems such as inability to guarantee security, resulting in the inability of neural networks to be used in some fields with high reliability and security requirements, and the scope of application is greatly limited.

[0003] Therefore, explaining the classification decision basis of the convolutional neural network can greatly improve the application effect of the convolutional neural network. The class activation map (CAM) visualization explanation method is a good explanation method. This method generates a heat map by linearly weighting the feature map output by a convolutional layer in the convolutional neural network, and then superimposes it with the test image to generate an explanation map, which can better show the decision basis of the convolutional neural network classification, such as Figure 1 As shown in the figure, the explanation map and heat map mark the importance score of each pixel in the image in the classification decision. The higher the score, the more important the pixel is. Through the heat map and explanation map, we can know which parts of the image the convolutional neural network mainly relies on to make classification decisions.

[0004] Among the specific CAM interpretation methods, the Score-CAM method is a very good interpretation method, but it still has the following problems in use:

[0005] 1) Some feature maps get higher weights, not because they are more important, but because they highlight more pixels, because highlighting more pixels means more pixels contribute. This makes the Score-CAM method inaccurate in measuring the weight of feature maps;

[0006] 2) The softmax function will amplify the gap between feature map weights due to exponential operation, and will incorrectly amplify / reduce the importance of certain features on some input images. This will also make the Score-CAM method's measurement of feature map weights inaccurate, resulting in erroneous interpretation results.

[0007] Therefore, providing a method for explaining convolutional neural network image recognition classification decisions that can overcome the above-mentioned defects is an issue that needs to be urgently addressed by those skilled in the art. Summary of the invention

[0008] In view of this, the present invention provides an interpretation method for convolutional neural network image recognition classification decisions. The method adds a penalty factor on the basis of the Score-CAM method, which makes up for the defects of the Score-CAM method and makes the average interpretation performance better. The method can generate the final heat map and interpretation map according to the standards desired by the user, and has better applicability.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] A method for explaining classification decisions of convolutional neural networks for image recognition, including

[0011] S1. Get the weight of the test image feature map:

[0012] The test image is input into the neural network model after being perturbed by its feature map, and the neuron activation value is obtained through the classification output layer;

[0013] After dividing the neuron activation value by the penalty factor corresponding to the feature map, the result is input into the softmax layer to obtain the score of the target class as the weight of the feature map, wherein the penalty factor is determined according to the pixel average value of the feature map of the test image;

[0014] S2. According to the weights of the multiple feature maps, linearly weighted sum is performed on the multiple feature maps, upsampled to the size of the test image, and after ReLU function operation and normalization, a heat map is obtained. The heat map is superimposed on the test image to obtain the test image interpretation map.

[0015] Preferably, the feature map is obtained by inputting a test image into the neural network model and outputting it through a target convolutional layer.

[0016] Preferably, a forward hook is registered in the target convolutional layer.

[0017] Preferably, the test image is perturbed by its feature map by upsampling the feature map to the size of the test image, normalizing it, and then performing dot multiplication with the test image.

[0018] Preferably, the penalty factor corresponding to the feature map is a multiple of the average value of the feature map pixels,

[0019] The value of a is obtained through the following steps:

[0020] S11. Traverse the parameter a at a certain interval within a certain range. For each parameter a traversed within this range, repeat steps S12 - S14 until the traversal ends;

[0021] S12. Determine the penalty factor according to the current value of the parameter a. After dividing the neuron activation value by the penalty factor, input it into the softmax layer to obtain the score of the target class, which is used as the weight of the feature map.

[0022] S13. Linearly weighted sum multiple feature maps according to the weights of multiple feature maps, and after upsampling, ReLU function operation, and normalization processing, obtain a heat map.

[0023] S14. Determine whether the heat map is more reasonable. If it is more reasonable, take the current value of the parameter a as the optimal solution.

[0024] Preferably, the method for determining whether the heat map is more reasonable is:

[0025] S141. According to the heat map, respectively retain the top x%, 2x%, 3x%,... nx% pixels with the highest scores in the test image, and set the remaining pixels to 0 to obtain n images,

[0026] where m - x < nx <= m, and m is the number of pixels of the original input image;

[0027] S142. Input the n images into the neural network model to obtain n target class scores, sum the n target class scores to obtain a total score, and determine whether it is more reasonable according to the total score.

[0028] Preferably, take the solution of the parameter a traversed in S11 as the initial point of simulated annealing, and use the simulated annealing method to execute steps S12 - S14 to solve the optimal solution of the parameter a.

[0029] Through the above technical solutions, it can be seen that compared with the prior art, the method disclosed in the present invention can, through the parameter a, achieve magnifying / reducing the gap between the average values of the feature maps as needed, and at the same time can adjust the importance degree of magnifying / reducing the features by the softmax function, that is, adjust the softmax to change the gap between the weights of the feature maps.

[0030] The present invention uses one parameter to simultaneously make up for two problems in the Score-CAM method, greatly reducing the amount of calculation, and the method can generate heat maps and explanation maps according to the standards desired by the user, with very good applicability and very high interpretation credibility. In addition, the method can be used for any convolutional neural network without any restrictions on the model structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0032] Figure 1 The accompanying drawing shows the process of generating a heat map and an explanation map for a target class of "tiger cat" by the CAM class explanation method provided by the present invention;

[0033] Figure 2 The accompanying drawing is a flow chart of the interpretation method of the convolutional neural network image recognition classification decision provided by the present invention;

[0034] Figure 3 The accompanying drawings are a thermal diagram (above) and an explanation diagram (below) generated by the method provided by the present invention;

[0035] Figure 4 The accompanying drawings are a comparison of the explanation graphs and thermal maps provided by the present invention and those generated by other explanation methods. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The embodiment of the present invention discloses a method for explaining convolutional neural network image recognition classification decision. In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the following is combined with the attached Figure 1 The present invention is further described in detail with reference to the accompanying drawings and specific embodiments.

[0038] First, select the trained convolutional neural network model that needs to be explained, determine the target convolution layer in the model, the input image, i.e. the test image, and the target category corresponding to the image;

[0039] Download and use the VGG16 pre-trained model in the Pytorch library as the experimental model. The target convolution layer in the model selects the last convolution layer. Download and use the ILSVRC2012 validation set in ImageNet from the public dataset as the experimental dataset.

[0040] Then refer to Figure 2 , preprocess the images in the original image dataset (required by the vgg16 pre-training model input, if not required, you can skip it); for the input image, crop it to 3×224×224 size, and normalize and standardize it. The mean used for standardization is [0.485, 0.456, 0.406], and the standard deviation is [0.229, 0.224, 0.225];

[0041] Register the forward hook in the target convolutional layer of the model to hook the feature map output by the convolutional layer;

[0042] Input the test image into the trained convolutional neural network model to obtain the feature map of the target convolutional layer;

[0043] The obtained feature map is upsampled to the input image size, normalized, and finally dot-multiplied with the original image to obtain the perturbed image.

[0044] The perturbed image is input into the trained neural network model to obtain the neuron activation value of the classification output layer.

[0045] Then determine the parameter a, and then determine the penalty factor corresponding to each feature map. Divide the target class neuron activation value obtained above by the corresponding penalty factor, and then input it into the softmax layer. The obtained target class score is used as the weight of the feature map.

[0046] Finally, the feature maps are linearly weighted combined to obtain a heat map, which is then superimposed on the test image to obtain an explanation map.

[0047] The process of determining parameter a is as follows:

[0048] 1) traversing parameter a at a certain interval within a certain range. In one embodiment, traversing parameter a at an interval of 0.2 within the range of [0.001, 40.001], for each parameter a traversed within the range, looping through steps 2) to 5) until the traversal is completed;

[0049] 2) Determine the value of the penalty factor (a times the average value of the feature map pixels) according to the value of the current parameter a, and divide the neuron activation value obtained by the classification output layer by the corresponding penalty factor;

[0050] 3) Input the result calculated in 2) into the softmax layer, obtain the score of the target class according to the output of the softmax layer, and use this score as the weight of the feature map;

[0051] 4) Linearly weighted sum multiple feature maps by combining the feature map weights obtained in 3), then upsample to the size of the original input image, perform ReLU function operation, and then perform normalization to obtain the heat map.

[0052] 5) According to the heat map, respectively retain the top x%, 2x%, 3x%,... nx% pixels with the highest scores in the test image, and set the remaining pixels to 0 to obtain n images,

[0053] where m - x < nx <= m, m is the number of pixels of the original input image. In one embodiment, n is 10. Then input the obtained 10 images into the model, and sum the obtained target class scores (scores output after passing through the softmax layer) to obtain the total score. If this total score is higher, set the optimal solution to the value of the current parameter a.

[0054] In one embodiment, if a more accurate solution is required, use the solution obtained in step 1) as the initial point, run the simulated annealing method twice in the range of [0.001, 40.001], the initial transfer step size is 80, the step size decay coefficient for the first time is set to 0.9, and the second time is set to 0.95. The end condition is that the transfer step size is less than or equal to 0.0001, and loop to execute 2) to 5) to obtain the optimal solution of parameter a.

[0055] Finally, combine the obtained parameter a to determine the penalty factor corresponding to each feature map, divide the obtained target class neuron activation value by the corresponding penalty factor, then input it into the softmax layer, and use the obtained target class score as the weight of the feature map.

[0056] According to the weights of multiple said feature maps, linearly weighted sum multiple said feature maps, upsample to the size of the test image, perform ReLU function operation, and perform normalization to obtain the heat map, and superimpose the heat map on the test image to obtain the interpretation map.

[0057] To further illustrate the beneficial effects of the present invention, it is verified through the following embodiments:

[0058] In the embodiment, the hardware used is CPU: Intel(R) Core(TM) i5-7300HQ CPU @ 2.50GHz, GPU: GeForceRTX1050Ti, video memory 4G, memory: 16GB, hard disk: 1T. The operating system is Windows 10. The software is CUDA (11.4.153), Python (3.8.3), torch (1.11.0), torchvision (0.12.0), numpy (1.22.3), opencv-python (4.5.5.64).

[0059] The method for explaining the classification decision of convolutional neural network image recognition described in the present invention generates a heat map and an explanation map in this example as shown in the attached figure. Figure 3 As shown,

[0060] It can be found that the heat map and explanation map generated by the method described in the present invention can clearly show people the classification decision basis of the convolutional neural network. Figure 3 We can then intuitively understand which pixel regions in the image the convolutional neural network used to classify the input image as "Bullmastiff" and "Tiger Cat" respectively.

[0061] This example also compares the effects of different interpretation methods (Grad-CAM method, Score-CAM method, etc.) and the interpretation graphs and thermal maps produced by the present invention. Figure 4 shown.

[0062] pass Figure 4 , it can be found that the heat map and explanation map generated by the method of the present invention are more concentrated on the positioning of the target object in the image compared with other explanation methods, and the heat map and explanation map have less random noise.

[0063] At the same time, this example also compares the performance of the present invention and other interpretation methods in terms of interpretation confidence. We randomly selected 2000 images from the ILSVRC2012 validation set as the experimental data set. The evaluation method is to retain only the x% of pixels with the highest scores in the input image based on the generated heat map, set the remaining pixel values ​​to 0, and then input them into the model and evaluate them using the following indicators:

[0064] 1) AverageDrop%: Its definition is shown in the following formula,

[0065]

[0066] Where N is the number of images, is the score of target class c obtained after the i-th original image is input into the model, It is the score of target class c obtained by inputting the model after retaining only the x% of pixels with the highest scores according to the heat map of target class c. The effect of this indicator in measuring the confidence of explanation is also relatively intuitive. If the important features annotated by the heat map are relatively more correct, then the retained pixels should be more important to the target class, and the deleted pixels are insignificant to the target class, then the impact on the target class score should be small, and the target class score should drop less. Therefore, the smaller this indicator is, the higher the confidence of the explanation method is;

[0067] 2) AverageIncrease%: The definition of this indicator is as follows,

[0068]

[0069] The value of the Sign() function is 0 or 1. When the input condition is True, If true, 1 is returned, otherwise 0 is returned. The basis for this indicator to measure the confidence of the explanation is that if the deleted pixels are insignificant to the target class c, or even features of other classes, then the image will be free of interference from features of other classes, and the probability of being classified as the target class c should increase. Therefore, we count the percentage of images in which the probability of the target class is increased after the original input image is perturbed according to the heat map among a total of N images. The higher the percentage, the more accurate the explanation of the target class by the heat map generated by this method, and the higher the confidence of the explanation;

[0070] 3) Win / Lose%: This metric compares the performance of different methods on the same image and is defined as follows:

[0071]

[0072] in The score of target class c obtained after perturbing the original input image (retaining the first x% of pixels) by explaining the heat map generated by method a and inputting it into the model, Similarly. This indicator compares the accuracy of heat maps generated by different methods on the same image. If one method is better than another method, then after perturbing the original input image according to the generated heat map, the method should obtain a higher target class score on most images than the target class score obtained by the other method. Then the Win% indicator should be higher and the Lose% should be lower. In this experiment, we compare the Win / Lose% indicators of the present invention and other interpretation methods respectively.

[0073] In this example, x is 30 and 50 respectively, and the experimental results are shown in Tables 1 and 2, where Win% and Lose% indicators are the results compared with the method of the present invention (Win% indicates the proportion of images with better performance of the present invention).

[0074] Table 1 Experimental results of retaining 30% pixels

[0075]

[0076] Table 2 Experimental results of retaining 50% pixels

[0077]

[0078] From the above experimental results, it can be found that the performance of the present invention is better than other interpretation methods in any index and has a higher interpretation confidence.

[0079] Furthermore, this example also compares the positioning ability of the target object in the image with the method described in the present invention and other interpretation methods. We evaluate how many values ​​in the generated heat map fall into the target object box. The larger the value, the more concentrated the positioning ability of the target object. The formal definition is as follows:

[0080]

[0081] in is the pixel value at position (i, j) in the heat map of target class c.

[0082] We randomly selected 200 images in which the target object box occupies no more than 60% of the pixels as experimental data (a target object box that is too large will affect the evaluation results, narrowing the gap between evaluation results). The experimental results are shown in Table 3. From the results, it can be found that the method proposed in the present invention has better positioning ability.

[0083] Table 3 Comparison of the positioning ability evaluation results of the present invention and other interpretation methods

[0084]

[0085] Based on the above experimental results, it can be found that the interpretation method proposed in the present invention has better average interpretation performance.

[0086] Beneficial effects of the present invention:

[0087] 1) It does not rely on gradients or specific model structures, and the generated explanation maps and heat maps are more focused on the location of the target object and have less random noise;

[0088] 2) Higher confidence in the interpretation of convolutional neural networks;

[0089] 3) Thermal maps and explanation maps can be generated according to the standards required by the user, which has better applicability;

[0090] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0091] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An interpretation method for convolutional neural network image recognition classification decision-making, characterized in that S1. Obtain the weights of the test image feature maps: The test image is input into the neural network model after being perturbed by its feature maps, and the neuron activation values are obtained through the classification output layer; After dividing the neuron activation values by the penalty factors corresponding to the feature maps, they are input into the softmax layer to obtain the scores of the target classes, which are used as the weights of the feature maps. The penalty factors are determined according to the pixel average values of the test image feature maps; The penalty factor corresponding to the feature map is a multiple value of a times the average value of the feature map, The a is obtained through the following steps: S11. Traverse the parameter a at a certain interval within a certain range. For each parameter a traversed within the range, repeat S12 - S14 until the traversal ends; S12. Determine the penalty factor according to the value of the current parameter a. After dividing the neuron activation values by the penalty factor, input them into the softmax layer to obtain the scores of the target classes, which are used as the weights of the feature maps, S13. Linearly weighted sum multiple feature maps according to the weights of the multiple feature maps, and after upsampling, ReLU function operation and normalization processing, obtain a heat map; S14. Determine whether the heat map is more reasonable. If it is more reasonable, use the value of the current parameter a as the optimal solution; The method for determining whether the heat map is more reasonable is: S141. According to the heat map, respectively retain the top x%, 2x%, 3x%,... nx% pixels with the highest scores in the test image, and set the remaining pixels to 0, obtaining n images, where m - x < nx <= m, and m is the number of pixels of the original input image; S142. Input the n images into the neural network model to obtain n target class scores, sum the n target class scores to obtain the total score, and determine whether it is more reasonable according to the total score; S2. Linearly weighted sum multiple feature maps according to the weights of the multiple feature maps, upsample them to the size of the test image, and after ReLU function operation and normalization, obtain a heat map. Superimpose the heat map on the test image to obtain the test image interpretation map.

2. The method for explaining a convolutional neural network image recognition classification decision according to claim 1, characterized in that: The feature maps are obtained by inputting the test image into the neural network model and outputting through the target convolutional layer.

3. The method for explaining a convolutional neural network image recognition classification decision according to claim 2, characterized in that: A forward hook is registered in the target convolutional layer.

4. The method for explaining convolutional neural network image recognition classification decision according to claim 1, characterized in that: The perturbation of the test image by its feature maps is to upsample the feature maps to the size of the test image, normalize them, and then perform dot multiplication with the test image respectively.

5. The method for explaining convolutional neural network image recognition classification decision according to claim 1, characterized in that: Use the solution of the parameter a traversed in step S11 as the initial point of simulated annealing, and use the simulated annealing method to execute steps S12 - S14 to solve the optimal solution of the parameter a.

Images