Black box interpretable method and device for positioning problem in target detection model

Abstract

The invention discloses a black box interpretable method and device for a positioning problem in a target detection model, and the method comprises the steps: training a target detection model through original image data, and obtaining a trained target detection model; an image to be explained is input into the trained target detection model, a plurality of target frames to be explained are output, and each target frame comprises vertex coordinates, the category of an object and the confidence coefficient of the category; the method comprises the following steps: generating a plurality of masks by adopting Monte Carlo sampling, disturbing an image, inputting the disturbed image into a network, generating a candidate frame, and carrying out weight assignment on a disturbed area according to whether the candidate frame and a to-be-explained target frame are in the same category, the same object and the same edge or not; and performing weighted summation on the masks by using the weight corresponding to each disturbance mask to obtain a final explaining heat map. According to the method, the output change is accurately captured by disturbing the original image, so that the thermodynamic diagram is generated for explanation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence explainable technology, and in particular to a black box explainable method and device for positioning problems in a target detection model. Background Art

[0002] Existing explanation methods can be divided into black-box methods and white-box methods. White-box methods need to access the inside of the network to obtain information such as the feature map and gradient of the hidden layer to make explanations; black-box methods only need to obtain the output results of the model. The latter is also a model-independent method because it does not require internal information. Similarly, according to the "Causal Ladder", explainable methods can be reclassified into correlation methods with weaker causal relationships and intervention methods with stronger causal relationships. The stronger the causal relationship, the higher the fidelity of the method.

[0003] For image classification networks, the relevant methods include Gradient, IG (Integrated Gradient Method) series, CAM (Class Activation Mapping) series, LRP (Layer-by-Layer Relevance Propagation), etc. Gradient is the simplest and most direct interpretation method. By calculating the gradient of the output to the input, the contribution of the input to the final output is obtained. Deconvolution and GuidedBackprop are extensions and optimizations of Gradient. Deconvolution modifies the ReLU (Rectified Linear Unit) in the original network to become a gradient ReLU, thereby filtering out negative gradients and focusing only on features that contribute positively to the output.

[0004] GuidedBackprop filters negative gradients based on the original network ReLU, focusing only on activated features that contribute positively to the output, which can reduce salient map noise. The IG gradient integration method improves problems such as gradient explosion by integrating the gradient, and subsequent IG methods are derived from this. The working principle of CAM, GradCAM (gradient weighted class activation mapping), and GradCAM++ is to use the activation map and weight linear addition of the last convolutional layer to obtain a heat map for interpretation. The weights of the three are different. CAM uses the parameters of the fully connected layer, GradCAM uses the gradient of the category score, and GradCAM++ assigns gradient weights to each pixel in the activation map. The LRP correlation score is propagated layer by layer. As the name suggests, the correlation score is passed forward layer by layer from the final output result to obtain the correlation of each pixel. et al. improved LRP and applied it to the transformer model, and the scalability was improved.

[0005] However, the above methods cannot be directly applied to the interpretation of target detection and positioning problems. Even if some of them can be successfully migrated, the effect is not good. Recently, the correlation method ODAM (Object Detection Activation Mapping) was proposed to explain the target detection and positioning problem. It explains the positioning problem of the upper, lower, left, and right sides of the target box by calculating the gradient of the regression. In addition, GradCAM and GradCAM++ can be migrated to the interpretation task of the positioning problem based on ODAM, but their fidelity and scalability are not as good as the method proposed in the present invention.

[0006] Interventional methods include Occlusion, RISE (Random Input Sampling Explanation), LIME (Model-Independent Local Interpretability), Sobol (Sobol Index-based Explanation), HSIC (Hilbert Schmidt Independence Criterion-based Explanation), EVA (Verified Perturbation Analysis Explanation), ScoreCAM (Gradient-free Class Activation Mapping), etc. Occlusion, by masking the input of a specific position, calculates the change in the model output to obtain the "contribution" of the position, traverses all inputs, and obtains a saliency map. RISE uses the Monte Carlo method to weight the random mask (the weight is the output of the masked model) and averages the final saliency map. Subsequent improvements to Rise include Sobol, HSIC, and LIME. Although ScoreCAM is a CAM series, this method belongs to the white-box perturbation method. ScoreCAM abandons the gradient as the weight of the feature map, and instead uses the feature map to guide the perturbation, using its output as the weight, avoiding problems such as gradient explosion caused by the gradient. EVA, also a white-box perturbation method, perturbs the image more comprehensively by estimating the space of the activation map of the intermediate layer.

[0007] The above methods cannot distinguish different objects in the same category when explaining the localization of object detection. D-RISE (Explanation of Random Input Sampling for Object Detection) is inspired by RISE and completes the explainability of object detection, but this method mixes the explanation of localization and classification. Summary of the invention

[0008] In view of this, the present invention proposes a black box interpretable method and device for the positioning problem in the target detection model, which can explain various attributes of the target box according to user needs, including the attributes output by the model (i.e., the four edges of the box, such as top, bottom, left, right, etc.), and the attributes that the user wants to know, such as width, height, proportion, and center point. At the same time, the method should be model-independent and model-loyal, and can reveal the positioning strategy of the target detection model.

[0009] The present invention discloses a black box interpretable method for positioning problems in target detection models, which comprises:

[0010] Step 1: Use the original image data to train the target detection model to obtain a trained target detection model;

[0011] Step 2: Input the image to be interpreted into the trained object detection model, and output multiple object boxes to be interpreted. Each object box includes vertex coordinates, object category and category confidence.

[0012] Step 3: Monte Carlo sampling is used to generate multiple masks to perturb the image. The perturbed image is input into the network to generate candidate boxes. The perturbed area is weighted according to whether the candidate box is of the same category, the same object, or the same edge as the target box to be interpreted.

[0013] Step 4: Using the weight corresponding to each perturbation mask, perform weighted summation on the masks to obtain the final explanation heatmap.

[0014] Furthermore, the step 1 comprises:

[0015] The target detection models are FCOS and Faster RCNN. Both use ResNet-50 as the feature extraction backbone network and combine it with the feature pyramid network FPN. The MS COCO training set is used as the original image training model to obtain a model that performs well on the MS COCO test set.

[0016] Furthermore, the step 2 comprises:

[0017] The jth candidate box includes:

[0018] P j =(l j ,t j ,r j ,b j ,c j ,s j )

[0019] Among them, P j represents the jth candidate box, which includes two vertex coordinates (l j ,t j ) and (r j ,b j ), the category c of the object in the candidate box j , and the confidence of the category s j ;

[0020] The NMS algorithm selects one or more target boxes P from multiple proposals t ;P t Contains two vertex coordinates (l, t) and (r, b), the category of the object c, and the confidence s of the category; that is:

[0021] Pt =NMS(f(I))=(l,t,r,b,c,s)

[0022] Among them, the image to be interpreted is I, and f is the target detection model.

[0023] Furthermore, the step 3 comprises:

[0024] For the upper edge of the explained target box, Monte Carlo sampling is used to generate N binary masks, and the image to be explained is perturbed to obtain the weight Weight:

[0025] Weight(I⊙M i ,P t )=func(Top j ,Score j ,IOU j )

[0026] Among them, M i is the mask generated by the i-th sample, Top j Used to quantify the similarity of the upper edge, Score j Used to quantify category similarity, IOU j Used to quantify object similarity; func is a function used to screen candidate boxes whose edge similarity, category similarity, and object similarity of the candidate boxes meet the preset similarity threshold, and assign a weight to the current disturbance according to the edge similarity, category similarity, and object similarity of the candidate boxes.

[0027] Furthermore, the func function is expressed as:

[0028]

[0029] Among them, P j is the jth candidate box;

[0030] The maximum value of the product of the candidate box edge similarity, category similarity, and object similarity is selected to quantify the weight of the current perturbation.

[0031] Furthermore, after the i-th interference, the image is fed into the target detection model, and its output is:

[0032]

[0033] Among them, P j i represents the jth candidate box generated after the i-th perturbation; the top of the j-th candidate box j 、Score j and IOU j Defined as:

[0034]

[0035] in, Indicates the confidence that the jth candidate box is of category c;

[0036]

[0037] Among them, IoU is the intersection over union ratio, which is used to measure the overlap between two objects.

[0038] Furthermore, in step 4, the heat map is explained as:

[0039]

[0040] Where λ is a pixel on the mask; M i (λ) is the value of the mask at the λ pixel position; H I,f It is a heat map, which is the final interpretation result of the target detection model f and the image to be detected I.

[0041] The present invention also discloses a black box interpretable device for the positioning problem in the target detection model, which is used to implement the above-mentioned method, and comprises:

[0042] A training module is used to train the target detection model using the original image data to obtain a trained target detection model;

[0043] The candidate box acquisition module is used to input the image to be interpreted into the trained object detection model and output multiple object boxes to be interpreted. Each object box includes vertex coordinates, object category and category confidence.

[0044] The assignment module is used to generate multiple masks using Monte Carlo sampling, perturb the image, input the perturbed image into the network, generate candidate boxes, and assign weights to the perturbed areas based on whether the candidate boxes are of the same category, the same object, and the same edge as the target box to be interpreted;

[0045] The explanation heat map acquisition module is used to use the weight corresponding to each perturbation mask to perform weighted summation on the masks to obtain the final explanation heat map.

[0046] Due to the adoption of the above technical solution, the present invention has the following advantages:

[0047] 1. The present invention is an interpretable method for target detection and positioning, which can explain everything about positioning (it can explain the top, bottom, left, right, width, height and center of the box). By perturbing the original image, the output changes are accurately captured, thereby generating a heat map for interpretation, so it has model independence and model loyalty. Compared with the latest methods, the present invention has higher loyalty. In addition, through this method, the model's positioning strategy and the differences in positioning strategies of different models can be discovered.

[0048] 2. In specific applications, the present invention can be used in scenarios with high security requirements such as face recognition payment and autonomous driving, to improve the robustness of artificial intelligence models in the above scenarios and reduce the occurrence of security issues. For example, designers can use the present invention to detect whether the face recognition system really pays attention to a person's facial features, rather than making false judgments based on features such as hairstyle and glasses. Effectively ensure the reliability of the model. The present invention is a black box method and is also an intervention method, so it is applicable to any model and is more loyal to the model.

[0049] 3. The present invention is a black box intervention method, which is applicable to any model and is more faithful to the model. It can explain the target detection and positioning problem solely through the model output. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0051] Figure 1 A schematic diagram of a flow chart of an explainable method for locating a target bounding box in a target detection model according to an embodiment of the present invention;

[0052] Figure 2 This is a diagram showing the interpretation results of the Faster RCNN and FCOS models according to an embodiment of the present invention;

[0053] Figure 3 Schematic diagram of the expansion result of Faster RCNN according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described in conjunction with the accompanying drawings and embodiments, and the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present invention.

[0055] like Figure 1As shown, the embodiment of the present application provides a black box interpretable method for the positioning problem in the target detection model, which may specifically include:

[0056] Step S1: Use the original image data to train the FCOS and Faster RCNN target detection models to obtain a target detection model that performs well on a normal dataset.

[0057] This embodiment takes the large-scale detection benchmark dataset MS COCO dataset as an example to illustrate the specific implementation process in detail.

[0058] The FCOS and Faster RCNN used in this embodiment use ResNet-50 as the feature extraction backbone network and combine it with the feature pyramid network FPN. The model is trained using the training set of MS COCO to obtain a model that performs well on its test set.

[0059] Step S2: Convert the original image Input into the FCOS and Faster RCNN object detection model trained in step 1, the model outputs multiple candidate boxes P, each of which includes two vertex coordinates, the object category and the category confidence. For example, the jth proposal includes:

[0060] P j =(l j ,t j ,r j ,b j ,c j ,s j )

[0061] Among them, P j represents the jth candidate box, which includes two vertex coordinates (l j ,t j ) and (r j ,b j ), the category c of the object in the candidate box j , and the confidence of the category s j .

[0062] The NMS algorithm selects one or more target boxes P from multiple proposals t .P t It also contains two vertex coordinates (l, t) and (r, b), the category of the object c, and the confidence s of the category. That is:

[0063] P t =NMS(f(I))=(l,t,r,b,c,s)

[0064] Among them, the image to be interpreted is I, and f is the target detection model

[0065] In this embodiment, the MS COCO test set image is used as the original image I and input into the trained FCOS and FasterRCNN target detection models, and the model outputs the prediction result P t The attributes in the prediction result are used to subsequently generate the heat map H corresponding to the target bounding box boundary U,f (λ).

[0066] Step S3: Taking the upper edge of the target box as an example, Monte Carlo sampling is used to generate N binary masks to perturb the image. The obtained weight function is expressed as Weight:

[0067] Weight(I⊙M i ,P t )=func(Top j ,Score j ,IOU j )

[0068] Among them, M i is the mask generated by the i-th sample, Top j Used to quantify the similarity of the upper edge, Score j Used to quantify category similarity, IOU j Used to quantify the similarity of objects. There are many forms of the function func, which is mainly used to screen out candidate boxes with high similarity among the three, and assign the weight of the current disturbance according to the similarity of the three.

[0069] In this embodiment, the following method is used as the func function:

[0070]

[0071] Among them, P j is the jth candidate box;

[0072] This embodiment selects the maximum value of the product of three similarities to implement the func function, which can ensure that the weight of the current disturbance is reasonably quantized when all three are high.

[0073] After the i-th interference, the image is sent to the detector and the output is as follows:

[0074]

[0075] in, represents the jth candidate box generated after the i-th perturbation;

[0076] in Represents the jth candidate box generated after the i-th perturbation. The top of the jth candidate box j 、Score j and IOUj The definition is as follows:

[0077]

[0078] in, Indicates the confidence that the jth candidate box is of category c.

[0079]

[0080] Among them, IoU is the intersection over union ratio, which is used to measure the overlap between two objects.

[0081] Step S4: Using the weights calculated in step S3, perform weighted summation on the random masks to obtain the final explanation heat map H I,f (λ):

[0082]

[0083] Where λ is a pixel on the mask; M i (λ) is the value of the mask at the λ pixel position; H I,f It is a heat map, which is the final interpretation result of the target detection model f and the image to be detected I.

[0084] The present invention also provides a black box interpretable device for the positioning problem in the target detection model, which is used to implement the embodiment of the method described above, and includes:

[0085] A training module is used to train the target detection model using the original image data to obtain a trained target detection model;

[0086] The candidate box acquisition module is used to input the image to be interpreted into the trained object detection model and output multiple object boxes to be interpreted. Each object box includes vertex coordinates, object category and category confidence.

[0087] The assignment module is used to generate multiple masks using Monte Carlo sampling, perturb the image, input the perturbed image into the network, generate candidate boxes, and assign weights to the perturbed areas based on whether the candidate boxes are of the same category, the same object, and the same edge as the target box to be interpreted;

[0088] The explanation heat map acquisition module is used to use the weight corresponding to each perturbation mask to perform weighted summation on the masks to obtain the final explanation heat map.

[0089] The Faster RCNN and FCOS models are explained using the present invention. The specific explanation results are as follows: Figure 2As shown. The red box in the figure represents the predicted boundary box, the red area represents that the area is important for the model decision, and the blue area represents that the area is not important for the model decision. The first and second rows show the interpretation results of the present invention on the Faster RCNN and FCOS models, respectively. In the figure, the redder the color, the more important the area is in model positioning, and the bluer the color, the less important the area is. Similarly, this method can be extended to explain the center point, aspect ratio, length, width of the target box. The specific interpretation results are as follows Figure 3 shown. Figure 3 The red box in each image is the location of the detected object. The first to fourth columns are the result images explaining the center point, scale, height and width of the target box respectively.

[0090] The heat map generated by the interpretable method sometimes has the following situation: the highlighted area of ​​the heat map is indeed the key area in the eyes of humans, but for the model, the area may not be the key area. This situation is called the interpretable method is not loyal to the model. In order to ensure that the interpretable method proposed in the present invention is loyal to the model, this embodiment will use the two indicators of Deletion and Insertion to check whether the method proposed in the present invention is loyal to the model, and compare the current mainstream interpretable methods GradCAM and GradCAM++, as well as the latest interpretable method ODAM. The results are shown in Table 1.

[0091] Deletion and Insertion are indicators for evaluating whether the interpretable method is reliable. The Deletion method evaluates the importance of the deleted areas by successively deleting important areas in the heat map and then observing the impact of this deletion on the model output. If the output of the model changes dramatically, it means that the deleted area is more important. On the contrary, the Insertion method usually refers to inserting certain areas into the input of the model and then observing the changes in the model output to evaluate the importance of the inserted area.

[0092] Note that the lower the Deletion index, the more faithful the interpretable method is; the higher the Insertion index, the more faithful the interpretable method is.

[0093] Table 1. Deletion and Insertion metrics of different methods. The best results are in bold.

[0094]

[0095] The above experiments and related result analysis verify the effectiveness of the interpretation method proposed in this invention.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A black-box interpretable method for the localization problem in target detection models, characterized in that: include: Step 1: Use the original image data to train the target detection model to obtain a trained target detection model; Step 2: Input the image to be interpreted into the trained object detection model, and output multiple object boxes to be interpreted. Each object box includes vertex coordinates, object category and category confidence. Step 3: Monte Carlo sampling is used to generate multiple masks to perturb the image. The perturbed image is input into the network to generate candidate boxes. The perturbed area is weighted according to whether the candidate box is of the same category, the same object, or the same edge as the target box to be interpreted. Step 4: Using the weight corresponding to each perturbation mask, perform weighted summation on the masks to obtain the final explanation heatmap.

2. The method according to claim 1, characterized in that: The step 1 comprises: The target detection models are FCOS and Faster RCNN. Both use ResNet-50 as the feature extraction backbone network and combine it with the feature pyramid network FPN. The MS COCO training set is used as the original image training model to obtain a model that performs well on the MS COCO test set.

3. The method according to claim 1, characterized in that The step 2 comprises: The jth candidate box includes: P j =(l j ,t j ,r j ,b j ,c j ,s j ) Among them, P j represents the jth candidate box, which includes two vertex coordinates (l j ,t j ) and (r j ,b j ), the category c of the object in the candidate box j , and the confidence of the category s j ; The NMS algorithm selects one or more target boxes P from multiple proposals t ;P t Contains two vertex coordinates (l, t) and (r, b), the category of the object c, and the confidence s of the category; that is: P t =NMS(f(I))=(l,t,r,b,c,s) Among them, the image to be interpreted is I, and f is the target detection model.

4. The method according to claim 1, characterized in that: The step 3 comprises: For the upper edge of the explained target box, Monte Carlo sampling is used to generate N binary masks, and the image to be explained is perturbed to obtain the weight Weight: Weight(I⊙M i ,P t )=func(Top j ,Score j ,IOU j ) Among them, M i is the mask generated by the i-th sample, Top j Used to quantify the similarity of the upper edge, Score j Used to quantify category similarity, IOU j Used to quantify object similarity; func is a function used to screen candidate boxes whose edge similarity, category similarity, and object similarity of the candidate boxes meet the preset similarity threshold, and assign a weight to the current disturbance according to the edge similarity, category similarity, and object similarity of the candidate boxes.

5. The method according to claim 4, characterized in that The func function is represented as: Among them, P j is the jth candidate box; The maximum value of the product of the candidate box edge similarity, category similarity, and object similarity is selected to quantify the weight of the current perturbation.

6. The method according to claim 4, characterized in that After the i-th disturbance, the image is fed into the target detection model, and its output is: in, represents the jth candidate box generated after the i-th perturbation; the top of the j-th candidate box j 、Score j and IOU j Defined as: in, Indicates the confidence that the jth candidate box is of category c; Among them, IoU is the intersection over union ratio, which is used to measure the overlap between two objects.

7. The method according to claim 1, characterized in that In step 4, the heat map is explained as: Where λ is a pixel on the mask; M i (λ) is the value of the mask at the λ pixel position; H I,f It is a heat map, which is the final interpretation result of the target detection model f and the image to be detected I.

8. A black box interpretable device for positioning problems in target detection models, used to implement the method of any one of claims 1 to 7, characterized in that: include: A training module is used to train the target detection model using the original image data to obtain a trained target detection model; The candidate box acquisition module is used to input the image to be interpreted into the trained object detection model and output multiple object boxes to be interpreted. Each object box includes vertex coordinates, object category and category confidence. The assignment module is used to generate multiple masks using Monte Carlo sampling, perturb the image, input the perturbed image into the network, generate candidate boxes, and assign weights to the perturbed areas based on whether the candidate boxes are of the same category, the same object, and the same edge as the target box to be interpreted; The explanation heat map acquisition module is used to use the weight corresponding to each perturbation mask to perform weighted summation on the masks to obtain the final explanation heat map.

Images