A priority testing method for object detection system based on instance-level analysis

Through instance-level analysis and hidden confidence distribution maps and other technologies, detection errors and omissions in the target detection system are identified and given priority, and the problem of inefficient testing in the existing technology is solved, and efficient allocation of resources and effective detection errors are achieved. deal with.

CN118331883BActive Publication Date: 2024-11-19NANJING UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410566350.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-09
Publication Date
2024-11-19
Estimated Expiration
2044-05-09

AI Technical Summary

Technical Problem

The existing technology is difficult to effectively solve the inefficiency of the testing of the target detection system in the detection error and missed detection problems. Especially under limited budgets, the traditional image-based test priority strategy is not efficient enough, resulting in unstrategic resource allocation, which increases. Label costs.

Method used

The priority testing method of the object detection system based on instance-level analysis is adopted. By identifying detection errors and detection omissions, the suspicion score of the instance is calculated, and the hidden confidence distribution map and two-dimensional cardan algorithm are used to identify potential detection omission areas, and priority processing is given. Test cases that may cause model errors to achieve accurate and efficient allocation of resources.

Benefits of technology

It significantly improves the testing efficiency of the target detection system in detection errors and missed detection problems, reduces the labeling cost, makes resource allocation more strategic, and improves the process efficiency of detection model errors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118331883B_ABST
    Figure CN118331883B_ABST
Patent Text Reader

Abstract

The present invention discloses a target detection system priority testing method based on instance-level analysis, which is designed for instance-level object detection systems and can help testers identify detection errors and omissions in these systems. For detection errors, the present invention assigns a suspicion score to each predicted bounding box based on classification and positioning capabilities. In addition, the present invention overcomes the shortcomings of existing priority sorting methods that only focus on prediction results, and introduces a novel algorithm to assign suspicion scores to potential detection omission areas in each input. By summarizing the scores of two types of model errors, false positives and false negatives can be identified at the same time. The suspicion value elements are prioritized, sorted in descending order according to their values, the corresponding images are grayed, and the error areas are highlighted. The task of the tester is to focus on the highlighted areas. Compared with similar technologies, the present invention significantly saves budget and can maximize the role of testers.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a target detection system priority testing method based on instance-level analysis, and belongs to the fields of artificial intelligence and software engineering. Background Art

[0002] Object detection systems based on deep neural networks (DNNs) play a pivotal role in advancing fields such as autonomous driving, face recognition, and robotics. These systems are tasked with identifying and localizing objects in images or videos, facilitating countless applications. Despite this, the inherent weaknesses of DNNs pose serious quality and reliability issues.

[0003] DNN-based systems are based on a data-centric programming paradigm that requires large amounts of annotated data for training and validation. The data annotation process is labor-intensive and costly, especially the manual delineation of bounding boxes for objects, which exacerbates this challenge. Existing research emphasizes that the cost of accurately annotating objects is an order of magnitude higher than that of simple tasks such as multiple-choice questions. This cost increases exponentially when there are a large number of objects in an image. While leveraging all collected data for annotation, testing, and model refinement can improve performance and robustness, it is an inefficient strategy given that the annotation cost is disproportionate to the marginal benefit obtained from detecting good instances.

[0004] To alleviate this problem, test prioritization methods have been proposed to filter out the data that is most likely to enhance the model from the massive amount of unlabeled data. This selective annotation method can save budget while improving model performance and robustness. New prioritization techniques such as DeepGini, DeepState, and PRIMA have been introduced for classification tasks. However, object detection tasks are fundamentally different from classification tasks because they involve instance-level evaluation - each object detected by the model constitutes an instance - a granularity problem that these classification-focused methods cannot address. In addition, the error output by the object detection system includes not only the misidentification of the object, but also the possibility that the system missed the object - another aspect that classification-focused methods do not take into account.

[0005] Object detection models are increasingly being deployed in many security-sensitive software systems. However, if an object detection system is not thoroughly tested, it may behave abnormally in actual applications, which can cause great and unpredictable damage to users. Given the high cost of labeling object detection tasks, the need for testing to ensure the accuracy and reliability of the system is particularly urgent. This paper focuses on how to quickly and effectively prioritize the test data of an object detection system with the help of automated tools.

[0006] Although many test prioritization strategies have been proposed recently to reduce the high cost burden of test case annotation for deep learning systems, most existing methods fail to adapt to the complexity of the object detection task. Object detection systems need to identify and classify all potential objects in an image, and testers must also prioritize detection omissions and errors for annotation and analysis, which poses new challenges to the design of prioritization methods. Therefore, the present invention aims to create an automated and accurate data error prioritization mechanism, which is a key problem to be solved in the field of deep learning. Summary of the invention

[0007] Purpose of the invention: In view of the problems and shortcomings in the prior art, the present invention proposes a target detection system priority test method based on instance-level analysis, which is used to improve the test efficiency of the target detection system in dealing with detection errors and missed detection problems. Although existing intelligent software testing technologies have developed a variety of test data priority sorting schemes to identify test samples that are most likely to cause system errors, these technologies are mainly aimed at image classification models, and use input images as independent evaluation units, assigning each image a priority score so that testers can identify and analyze potential model errors without fully annotating the data set.

[0008] However, for object detection systems, prioritizing test data based on the entire image is a time-consuming task because the labeling needs to be done instance by instance in the image. Although most instances are often correctly identified by the model with high confidence, there are still a few instances that may be detected incorrectly. Traditional methods are not efficient in using limited labeling budgets. Taking Google's data labeling service as an example, the billing unit for object detection is calculated by instance, charging $63 for every 1,000 instances, rather than by image. This means that images with a large number of instances will incur higher labeling costs. In comparison, the labeling of classification data sets only costs $35 to process 1,000 images.

[0009] In order to make more rational use of the limited testing budget, a practical solution is to focus resources on labeling those key instances, that is, those instances that the model has not yet been able to process with high confidence, rather than those parts that the model can already accurately identify. The present invention proposes an innovative instance-level test priority sorting method for target detection models, which is more accurate and efficient in allocating labeling budgets. Compared with traditional image-based test priority strategies, the instance-level sorting strategy of the present invention has obvious advantages in cost-effectiveness and economy. It makes resource allocation more strategic and is expected to significantly improve the process efficiency of detecting model errors.

[0010] Technical solution: A priority test method for target detection system based on instance-level analysis. First, some important concepts in the target detection task are explained. Consider a picture I∈R W×H The test data set composed of

[0011]

[0012] For a given image I and threshold τ, the target detection model can be determined The detected instance set is represented as satisfy Each element p in consists of three parts, defined as follows:

[0013]

[0014] Each instance Contains a triple (b,c,s), representing the bounding box b, category c, and confidence score s of the detected instance. The bounding box b consists of a coordinate pair that identifies the upper left and lower right corners of the rectangle surrounding the detected object. In a system using a threshold τ, if the confidence score s of instance p is p ≥τ, then the instance is considered valid, where s p Represents the confidence s of instance p in the equation.

[0015] The threshold τ plays a crucial role in regulating the number of instances produced by the model, acting as a filter to eliminate unwanted detections. For example, high-precision applications such as autonomous driving usually require a higher threshold τ to ensure the authenticity of the detected instances. On the contrary, for applications such as pedestrian counting systems, a lower threshold may be more appropriate in order to adopt a more relaxed detection method.

[0016] The set of real instances corresponding to each image is represented as It provides a standard reference for evaluation:

[0017]

[0018] Each real instance Contains a pair (b,c), representing the actual bounding box and category of the object respectively. In the context of the test scenario, where the test images are newly collected and lack preset true values, the goal of the priority test is to identify data that may induce model errors. These selected data are then sent to the annotator for annotation before formal testing.

[0019] In the field of object detection, the accuracy of a model is not only determined by the instances it correctly identifies. It is also important to consider the mistakes it makes. Therefore, the errors output by object detection models are usually divided into two categories: detection errors and detection omissions, each reflecting a different type of problem.

[0020] (1) Detection error, which refers to instances where the model incorrectly detects an object. This inaccuracy may be caused by the inconsistency between the detected object location (bounding box) and the actual object location, or by the misclassification of the object category. The set of detection errors is defined as E1:

[0021]

[0022] Here, the Intersection over Union (IoU) metric measures the difference between the detected bounding box b and the true bounding box The error is quantified by the overlap between express The real instance with the maximum IoU with this instance. If the IoU is less than 0.5, or the detected category c is consistent with this real category If it does not match, the instance is considered a detection error. Therefore, E1 constitutes a set of false positives, which is A subset of .

[0023] (2) Missed detections, which occur when the model completely misses an object that should have been detected. The error set E2 represents these misses:

[0024]

[0025] in represents the model output instance with the highest IoU with this real instance. Therefore, set E2 contains those from the set real instances that have no corresponding model output instances with an IoU greater than 0.5. This definition of missed detections is consistent with the common standard adopted by most object detection systems. An object is considered detected if it has at least one matching model output instance. Therefore, E2 intuitively represents the set of real instances that are ignored by the model.

[0026] Identifying missed detections is critical to evaluating the robustness of object detection systems. Because missed detections indicate that the system failed to detect the object, if we fail to diagnose these missed detections during the testing phase, we will be in a state of uncertainty. In object detection techniques, the accuracy of a model does not depend solely on the instances it correctly identifies. It is equally important to consider the mistakes that the model makes. Therefore, the errors output by object detection models are often categorized into two types: detection errors and missed detections, each of which reflects a different problem.

[0027] By prioritizing test cases, object detection systems can be tested with minimal effort. Our goal is to prioritize test cases that will cause model prediction errors. Then, let the testers annotate them and provide them to the model improvement process. Considering the impracticality of annotating every test data to establish the basic truth set G, prioritized testing aims to sort the test data so that those data that are most likely to contain errors are reviewed and annotated first. This approach aims to maximize the detection of errors with minimal manual effort. Specifically, the method includes the following steps.

[0028] 1) Find the detection error. Quantifying the detection error requires evaluating each output of the target detection system. Each output p should be assigned a suspicion score. Independent evaluation methods are introduced to measure the object detection system's ability to classify and locate instance p. These methods aim to profile and analyze the system's overall capabilities to ensure a detailed understanding of its performance. We then combine these two metrics to form the sk-score. The following is their design:

[0029] (1) Classification capability: the object detection system generates a confidence score s for a given instance p p , which reflects the confidence of the model in classifying the bounding box content as a specific category; the indicator of instance classification ability is defined as:

[0030] C=1-s p

[0031] (2) Localization capability. The initial proposal of an object detection system is an intermediate state when the object detection model performs regression, where the model regresses to accurately approximate a rough proposal of the object. From pioneering research on object detection models such as Faster R-CNN and SSD, we observe that the accuracy of object localization is significantly improved in well-trained models, which are able to effectively optimize the initial rough proposals. In contrast, the outputs of under-trained models tend to be closely aligned with these initial proposals. Our methodology is inspired by these findings. In this paper, the precise location of an object output by an object detection system is called the instance location, which is represented by its bounding box b. p Indicates that the initial proposal area of ​​the target generated by the system during the prediction process is called the model view, which is represented by the bounding box v p Representation. The model view reflects the model's initial assessment of where an object might be located. To quantitatively measure this concept, we introduce a new metric:

[0032] L=IoU(v p ,b p )

[0033] In this function, the value of L is determined by the IoU metric, and a higher value of L indicates that the model’s ability to locate instances is reduced. p According to the nature of the target detection model, v can be traced back through the regression parameters output by the model p Specifically, the regression parameters [d x (v),d y (v),d w (v),d h (v)] is used to adjust v p To get the instance bounding box:

[0034] b p =[p x ,p y ,p w ,p h ]

[0035] p x ,p y represents the center point of the box, p w ,p h Represents the length and width of the box. From the initial proposal v of the target detection system p =[v x ,v y ,v w ,v h ] to b p =[p x ,p y ,p w ,p h The conversion of ] is given by:

[0036]

[0037] Among them, d x (v),d y (v),d w (v),d h (v) represents the regression parameter output by the target detection system. Furthermore, the model view v can be inferred from the above formula: p for:

[0038]

[0039] The above formula ensures that no matter what the model architecture is, as long as it outputs regression parameters, the model view v can be obtained p This reverse calculation allows us to evaluate the localization capability by determining the corresponding model view of the output result.

[0040] In order to comprehensively evaluate the object detection system's ability to classify and localize a specific object, these are combined to form a suspicion score sk for instance p. The suspicion elements are then used to identify detection errors:

[0041] (t,sk)=(p,L·C)

[0042] The sk-score reflects the proficiency of the model in detecting instances, with higher values ​​indicating weaker detection capabilities.

[0043] 2) Diagnose missed detections. An algorithm is designed to generate regions containing missed detections and assign appropriate suspicion scores. In the process of exploring the reasoning mechanism of the object detection model, we found an interesting pattern. After using the Faster-RCNN model with a confidence threshold of 0.7 on the COCO validation set, 93% of the instances were excluded after threshold filtering, leaving only 7% of the instances for output. We found that the number of low-confidence instances is increasing. The number of low-confidence instances filtered out by the threshold is large, and we believe that these low-information parts may carry enough information to diagnose missed detections. Inspired by this, we found that due to the nature of the object detection model, some missed detections will be roughly predicted when the threshold is lowered. Therefore, a natural idea is that when a region has no output instances but has a dense set of potential low-information predicted bounding boxes, the region is more likely to have potential missed detections. Based on the above viewpoints, we interpret the calculation method of the hidden confidence distribution map. According to the calculated distribution map, we can deduce the potential region These potential areas are then aggregated into priority queues to identify missed detections.

[0044] Now we describe this process. For an image I∈R W×H , we introduce the term hidden confidence map, which is initially a full zero matrix M0∈R n×m , expressed as:

[0045]

[0046] Here, n and m are used to describe the shape of the matrix, which conform to the constraints:

[0047]

[0048] Among them, W and H represent the length and height of the image input to the target detection system. Assume that the target detection system sets an initial threshold τ. When the threshold is relaxed to 0, represents the set of output instances of image I. Each represents a single instance, let (x1, y1) and (x2, y2) be the bounding box b p Operation This is defined as:

[0049]

[0050] function By adding the bounding box b p Mapping to the corresponding matrix segment in M0 effectively updates the hidden confidence distribution map and assigns the confidence score of p to the corresponding position in the matrix. Then, we get M p , which represents the hidden confidence distribution of p:

[0051]

[0052] In this way, the hidden confidence distribution map under the system threshold τ is obtained, which includes all instance confidences omitted from the output by the object detection system. Therefore, this hidden confidence distribution map represents the low-information part of the image after model inference. For convenience of calculation, the median concentration matrix M is used τ :

[0053] M′ τ =M τ -median(M τ )

[0054] After the data is concentrated near zero, M τ ′ becomes a hidden confidence distribution map for diagnosing missed detection. Therefore, the median is used to convert M τ ′ is placed in the center to facilitate the execution of the following two-dimensional Cardan algorithm; moving the center point of the data to 0 helps the algorithm find the real low information density area. The reason for using the median instead of the average is to make the center point insensitive to outliers, because outliers often mean potential detection omissions. Based on the above insights, we need to find areas where potential low information prediction boxes are dense. With the hidden confidence distribution map M τ ′, then the problem of finding instances missed by the object detection system can be transformed into finding M τ ′ is a submatrix with a larger sum of element values. This submatrix is ​​in M τ The range in ′ is the potential region we need to find. Let sum(z) represent the sum of all elements in the region, and use the two-dimensional Kardané algorithm to find M τ ′ where the maximum value of sum(z) is the first N regions. To represent the top N largest areas:

[0055]

[0056] in Representing a collection An element of , function The function is to use the two-dimensional Cardan algorithm to output N regions z. The two-dimensional Cardan algorithm will be executed N times on the matrix. When the algorithm is executed once, the algorithm will output a region with the maximum sum(z) value, and then it will convert M τ The z regions in ω are set to zero and the next execution is performed until N regions are output. Then, z is scaled to the original image according to the ρ value in the matrix constraint above. Therefore, z represents the location box of potential regions where there is a detection omission.

[0057] At the same time, calculate the average value of the elements in z and use it as the suspicion score sk. This score summarizes the quantitative suspicion of the missed detection. Therefore, the suspicion elements of the missed detection are:

[0058]

[0059] Finally, the above process of diagnosing missed detections is repeated for all images in the dataset, merging potential high-confidence suspicious regions in each image into the set middle.

[0060] 3) Priority sorting execution. After getting all the suspicion elements (t, sk), we need to build a set It is used to store all these suspicious elements and facilitate subsequent analysis. Therefore, how the present invention efficiently and preferentially processes suspicious elements is now described in detail. This helps testers to conduct effective testing and data annotation. The complete workflow of test prioritization is introduced. The input of this part includes the object detection model M, the threshold τ determined by the target detection system, and the annotation budget k. The output result is a prioritized dataset TestData, which consists of the original dataset D, re-sorted according to the suspicion score, and highlights those instances with higher suspicion scores for review. First, initialize an empty set To represent the suspected element. Then, we find the detection error and diagnose the detection omission, respectively, as described below. In the process of finding the detection error, for each image I in the dataset D, we use the object detection model M and the threshold τ to perform reasoning, and then segment the result into different instances, using the set For each instance Define element t as p. Then, calculate the classification capability C of the object detection model for the instance. Calculate the model view and then determine the model's localization capability L for the instance. Then, derive the suspicion score sk and append the derived suspicion element (t, sk) to the set For the missing part of the diagnostic detection, we first initialize an all-zero matrix M0 to represent the hidden confidence distribution map. Then we use the object detection model M with a threshold of 0 for reasoning, and split the obtained instances to remove The relevant subset is obtained Obtain the comprehensive hidden confidence distribution map M τ , and center it, perform a two-dimensional Kadane algorithm on the distribution map to isolate the top N potential regions with the largest sum of confidences, and then calculate the average confidence value for each region Then these regions z and Unified into suspicion elements (t, sk) and included in the set Then, yes The suspicion score sk in is normalized and the sk value is adjusted proportionally to the range of [0,1]. The elements in are prioritized and sorted in descending order according to their sk values. For each t, ​​the corresponding image I is grayscaled, these grayscaled images are added to the TestData set, and the area represented by t is highlighted. The tester's task is then to look at the images in the TestData set in turn, focusing on the highlighted areas. The highlighted area t represents a detection error or a detection omission. This approach significantly saves budget and maximizes the role of the tester.

[0061] Further, the specific steps of the above step 1) are as follows:

[0062] Step 1)-1: Starting state;

[0063] Step 1)-2: Input dataset and test model Model threshold τ, test budget k;

[0064] Step 1)-3: Traverse Each I in;

[0065] Step 1)-4: Execution Split the model's prediction results into instances, splitintoinstance() is the instance splitting function;

[0066] Step 1)-5: Traversal For each p in;

[0067] Step 1)-6: Execute t←p to assign values;

[0068] Step 1)-7: Execute C=1-ps to calculate the classification ability of the model for the instance, where ps represents the confidence value output by the target detection system;

[0069] Step 1)-8: Execute pv=getBoxV(pb,d(v)), which means executing the above steps of calculating the model view, thereby obtaining the model view corresponding to the instance p, and getBoxV() is a function for calculating the model view;

[0070] Step 1)-9: Execute L=getIoU(pv,pb), which means calculating the IoU value between the model view pv and the prediction box pb, so as to obtain the model's ability to locate the instance. getIoU() is a function for calculating the IoU of two elements.

[0071] Step 1)-10: Execute sk=L·C to obtain the suspicion score;

[0072] Step 1)-11: Execution This operation means adding the suspicion score to the set, add() is the addition operation;

[0073] Step 1)-12: Judgment Check if the traversal is completed. If so, proceed to the next step. If not, proceed to steps 1)-5.

[0074] Step 1)-13: Judgment Check if the traversal is completed. If so, proceed to the next step. If not, proceed to steps 1)-3.

[0075] Step 1)-14: End state.

[0076] Further, the specific steps of the above step 2) are as follows:

[0077] Step 2)-1: Starting state;

[0078] Step 2)-2: Traverse Each I in;

[0079] Step 2)-3: Initialize the all-zero matrix M0=O∈R n×m ;

[0080] Step 2)-4: Execute This operation means splitting the prediction results of the target detection system at the threshold τ into instances, and then deleting the parts that have been predicted in steps 1)-4

[0081] Step 2)-5: Execution Indicates the calculation of the hidden confidence matrix of image I, getHCDmap() is the function for calculating the hidden confidence matrix;

[0082] Step 2)-6: Execute M τ ′=M τ -median(Mτ ), which means to convert M τ The value of each element of the matrix is ​​subtracted from the median of all elements to achieve centralization. median() is the function for calculating the median.

[0083] Step 2)-7: Execute Use the two-dimensional Kardane algorithm to isolate the top N potential regions with the largest sum of confidences;

[0084] Step 2)-8: Traversal For each z in

[0085] Step 2)-9: Execute Indicates the calculation of M′ τ The mean of all elements covered by the potential area z in is used as the suspicion degree, and getMean() is the function for calculating the mean;

[0086] Step 2)-10: Execute Add the suspicion score to the set;

[0087] Step 2)-11: Judgment Check if the traversal is completed. If yes, proceed to the next step. If not, proceed to step 2)-8.

[0088] Step 2)-12: Judgment Check if the traversal is completed. If yes, proceed to the next step. If not, proceed to step 2)-2.

[0089] Step 2)-13: End state.

[0090] Further, the specific steps of the above step 3) are as follows:

[0091] Step 3)-1: Starting state;

[0092] Step 3)-2: Execute Will Regularization, normaxlize() is the regularization function;

[0093] Step 3)-3: Execute Perform priority sorting, reverse_sort() is the priority sorting function;

[0094] Step 3)-4: Traverse For each t in;

[0095] Step 3)-5: execute image=Grayed_out(I) to grayscale, Grayed_out() is a grayscale processing function;

[0096] Step 3)-6: If Then execute step 3)-7, otherwise execute step 3)-9;

[0097] Step 3)-7: Execute TestData.add(image) to add image to TestData;

[0098] Step 3)-8: Execute TestData.image = highlight(TestData.image, t) to highlight, highlight() is the highlighting function;

[0099] Step 3)-9: Execute k=k-1;

[0100] Step 3)-10: If k≤0, execute step 3)-12, otherwise execute step 3)-11;

[0101] Step 3)-11: Judgment Check if the traversal is completed. If so, proceed to the next step. If not, proceed to step 3)-4.

[0102] Step 3)-12: Return TestData as the final result;

[0103] Step 3)-13: End state.

[0104] Beneficial effects: The target detection system priority testing method based on instance-level analysis provided by the present invention is used to improve the test efficiency of the target detection system in dealing with detection errors and missed detection problems. Resources are concentrated on labeling those key instances, that is, those instances that the model has not yet been able to process with high confidence, rather than those parts that the model can already accurately identify. The present invention is more accurate and efficient in allocating labeling budgets. Compared with traditional image-based test priority strategies, our instance-level sorting strategy has obvious advantages in cost-effectiveness and economy. It makes resource allocation more strategic and is expected to significantly improve the process efficiency of detecting model errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 is an overall flow chart of the method of an embodiment of the present invention;

[0106] Figure 2 for Figure 1 Flowchart of detection errors found in;

[0107] Figure 3 for Figure 1 Flowchart of missed diagnostic tests;

[0108] Figure 4 for Figure 1 Flowchart of test priority execution. DETAILED DESCRIPTION

[0109] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0110] Figure 1 The overall framework diagram of a target detection system priority testing method based on instance-level analysis implemented in the present invention.

[0111] An overall framework of a priority testing method for an object detection system based on instance-level analysis includes the following steps.

[0112] Step 1: Discover detection errors: Split the model’s prediction results into multiple instances, and then calculate the classification and positioning capabilities of each instance to obtain a suspicion score for discovering detection errors.

[0113] Step 2: Diagnose detection omissions: By executing the two-dimensional Kadan algorithm on the defined hidden confidence distribution map, the area where detection omissions may exist can be obtained, thereby achieving accurate diagnosis of detection omissions of the target detection model.

[0114] Step 3 Test Priority Execution: Allows testers to view images in the TestData collection in sequence and focus on the highlighted areas. This approach significantly saves budget and maximizes the role of testers.

[0115] Figure 2 Flow chart for finding detection errors. The specific steps are as follows: Step 1: Starting state; Step 2: Input data set and test model Model threshold τ, test budget k; Step 3: Traverse Each I in; Step 4: Execute Split the model's prediction results into instances; Step 5: Traverse For each p in ; Step 6: Execute t←p to assign a value; Step 7: Execute C=1-ps to calculate the classification ability of the model for the instance; Step 8: Execute pv=getBoxV(pb,d(v)) to obtain the model view; Step 9: Execute L=getIoU(pv,pb) to obtain the model's positioning ability for the instance; Step 10: Execute sk=L·C to obtain the suspicion score; Step 11: Execute Add the suspicion score to the set; Step 12: Judge Check if the traversal is complete. If yes, proceed to the next step. If not, proceed to step 5. Step 13: Determine Check whether the traversal is completed. If so, proceed to the next step. If not, proceed to step 3. Step 14: End state.

[0116] Figure 3 This is a flowchart for diagnosing missed detection. The specific steps are as follows: Step 1: Starting state; Step 2: Traversal Each I in; Step 3: Initialize the all-zero matrix M0 = O∈R n×m ; Step 4: Execute The model's prediction results are split into instances and the predicted parts are extracted; Step 5: Execute Calculate the hidden confidence matrix; Step 6: Execute M′ τ =M τ -median(M τ ) τ Matrix Centering; Step 7: Execution Use the two-dimensional Kadane algorithm to separate the top N potential regions with the largest sum of confidences; Step 8: Traverse For each z in; Step 9: Execute Calculate the mean as the suspicion level; Step 10: Execute Add the suspicion score to the set; Step 11: Judge Check if the traversal is complete. If yes, proceed to the next step. If not, proceed to step 8. Step 12: Determine Check whether the traversal is completed. If so, proceed to the next step. If not, proceed to step 2. Step 13: End state.

[0117] Figure 4 The flowchart for test priority execution. The specific steps are as follows: Step 1: Starting state; Step 2: Execution Will Regularization; Step 3: Execution Prioritize; Step 4: Traverse For each t in; Step 5: execute image = Grayed_out (I) for grayscale conversion; Step 6: If Then execute step 7, otherwise execute step 9; Step 7: execute TestData.add(image) to add image to TestData; Step 8: execute TestData.image=highlight(TestData.image,t) to highlight; Step 9: execute k=k-1; Step 10: if k≤0, execute step 12, otherwise execute step 11; Step 11: judge Check if the traversal is completed. If yes, proceed to the next step. If not, proceed to step 4. Step 12: Return TestData as the final result. Step 13: End state.

[0118] In summary, the present invention proposes a method to perform priority testing on target detection system test data based on instance-level analysis, while taking into account two types of errors inherent in target detection systems, namely detection errors and detection omissions, thus filling the gap in current testing methods in the field of target detection systems.

Claims

1. A method for testing the priority of an object detection system based on instance-level analysis, characterized in that: The following steps are involved: Step 1, discovering detection errors: split the prediction results of the object detection model into multiple instances, and then calculate the classification and localization capabilities of each instance to obtain a suspicion score for discovering detection errors; Step 2, diagnose detection omissions: By executing the two-dimensional Kadan algorithm on the defined hidden confidence distribution map, the area where detection omissions may exist is obtained, thereby achieving accurate diagnosis of detection omissions of the target detection model; Step 3, test priority execution: enables the tester to view the images in the TestData collection in sequence and focus on the highlighted areas; In the step of finding detection errors, the granularity of object detection data is multiple instances in the image, and quantification of detection errors requires evaluating each output of the object detection system. For the set of detected instances, each output p is assigned a suspicion score; two indicators, classification ability and localization ability, are designed to evaluate the model's classification and localization capabilities for specific instances, and a comprehensive score is calculated to determine whether an instance has a detection error; Given an instance p, generate a confidence score s p , which reflects the confidence of the model in classifying the bounding box content as a specific category; the indicator for evaluating the model's ability to classify this instance is defined as: C=1-s p ; A higher value of C means that the model's ability to classify a particular instance decreases, whereas a lower value indicates a higher ability. The location of the object output by the object detection system is called the instance location, which is represented by its bounding box b p Indicates that the initial proposal area of ​​the target generated by the system during the prediction process is called the model view, which is represented by the bounding box v p Representation; the model view reflects the model's preliminary assessment of where the object may be located. In order to quantitatively measure this concept, the positioning capability indicator is introduced: L=IoU(v p ,b p ); In the step of diagnosing missed detection, for image I∈R W×H , introduce a hidden confidence distribution map, which is initially a full zero matrix M0∈R n×m , n and m are used to describe the shape of the matrix, and n and m meet the constraints: ρ≤1; W and H represent the length and height of the image input to the object detection system; The target detection system sets an initial threshold τ. When the threshold is relaxed to 0, represents the set of output instances of image I, each represents a single instance, let (x1, y1) and (x2, y2) be the bounding box b p The coordinates of the operation definition: function By adding the bounding box b p Mapped to the corresponding matrix segment in M0, the hidden confidence distribution map is updated, and the confidence score of p is assigned to the corresponding position in the matrix; then, M p , which represents the hidden confidence distribution map of p, then the comprehensive hidden confidence distribution map M under the system threshold τ τ It is expressed as: In this way, we get the comprehensive hidden confidence distribution map under the system threshold τ, which includes all instance confidences omitted from the output by the object detection system; the median concentration matrix M τ :M′ τ =M τ -median(M τ ), after concentrating the data around zero, M′ τ It becomes a hidden confidence distribution map for diagnosing missed detections, finding areas where potential low-information prediction boxes are densely populated, using sum(z) to represent the sum of all elements in the area, and using the two-dimensional Kadan algorithm to find M′ τ The maximum value of sum(z) is in the first N regions, using the set To represent them: in Representing a collection An element of , function The function of is to use the two-dimensional Kadan algorithm to output N regions z; z represents potential regions, which are areas where detection is missed; Calculate the average of the elements in z This score is used as the suspicion score sk, which summarizes the quantitative suspicion of the missed detection. The elements of suspicion for missed detection are: Merge potential high-confidence suspicious regions in each image into a set middle; In the step of test priority execution, All suspicion scores sk are normalized and the sk values ​​are proportionally adjusted to the range of [0,1], and the set Used to store the suspicion element; after normalization, Prioritize the elements in , and sort them in descending order according to their sk values; for each suspected element t, grayscale the corresponding image I, add it to the test set, and highlight the area represented by t; The tester's task is then to look at the images in the test set in turn, focusing on the highlighted areas; the highlighted areas represent detection errors or missed detections.

2. The method for testing the priority of target detection systems based on instance-level analysis according to claim 1, characterized in that: In the step of diagnosing and detecting missed detection, missed detection is discovered by generating potential areas on the implicit confidence distribution map. The specific steps are to calculate the hidden confidence matrix M τ It is centered and the two-dimensional Kadan algorithm is used to separate the top N potential regions with the largest sum of confidences. The average confidence of the potential regions is used to diagnose the missed areas.