A data error scoring method for object detection
By fine-grained error scoring and sorting the object detection data set, the problem of defects in the existing technology of large-grained error scoring and cross-over threshold setting is solved, and higher-precision data cleaning and object detection effects are achieved.
Patent Information
- Application Number
- CN202210011101.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-05
- Publication Date
- 2025-04-22
- Estimated Expiration
- 2042-01-05
AI Technical Summary
In the existing object detection technology, the granularity of the error score is too large, and the error in each bounding box cannot be accurately evaluated, and there are defects in the setting of the crossover threshold, resulting in insufficient data cleaning.
A data error scoring method for object detection is proposed. By marking the targets in multiple pictures, target detection data is generated and divided into training sets and test sets. Use the training model to reason about the test set, calculate the error scores of real examples, false positive examples, and false counterexamples, and process the pictures based on these scores, optimize the error score sort, and locate them in each box.
Quantitative error evaluation of the data set to be cleaned is realized, the uncertainty of the model output is taken into account, the error score sorting is optimized, and the accuracy and cleaning accuracy of each box are improved.
Smart Images
Figure CN114511001B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of automotive electronics technology, and in particular to a data error scoring method for target detection. Background Art
[0002] In target detection, bounding boxes are used to mark targets. In training, the bounding box coordinates of the image and the true value target are input to the algorithm, and the algorithm model is finally obtained through various parameter adjustment methods; in reasoning, the image data is input to the model to obtain the result, which includes the category, bounding box coordinates, and confidence of the target inside the image. By comparing the inference result with the true value, true positive examples, false positive examples, and false negative examples are obtained. Before comparison, all targets with confidence less than a certain threshold in the inference result are generally deleted and no longer compared with the true value, which is equivalent to manually setting this part as true negative examples. In addition, the intersection-and-union ratio threshold must be set, that is, the intersection-and-union ratio of the true value bounding box and the inference bounding box must be greater than the threshold to be considered a match. According to the above evaluation methods and rules, data noise can be found in the data set, wrong annotations can be located, so that the data can be cleaned and the data accuracy can be improved.
[0003] In the current method, the obtained true positives, false positives, and false negatives are generally plotted on the original image, and then audited and cleaned. In addition, the image-level error score is also performed based on the precision and recall rate of each image or the number of true positives and false positives, and the images with error scores exceeding the threshold are selected for subsequent cleaning.
[0004] In the above method, the inference result includes the category, bounding box coordinates, and confidence of the target. The above method directly gives three types of box results without considering the confidence. The confidence reflects the pseudo probability of the target. The larger the probability, the more reliable the model believes the output is. The larger the difference between the confidence and 1, the greater the model uncertainty. In addition, the error sorting of the data set generally uses the precision and recall of a picture or calculates the number of true positives and false positives to score the errors, and then sorts and selects the pictures that exceed the error score threshold. The granularity of the error score of this method is a picture, and the box is not scored, so the accuracy is not enough. At the same time, the intersection-over-union ratio threshold is also a manually set value, which takes the value [0,1]. It is generally believed that if the intersection-over-union ratio of two boxes is greater than 0.5, it means that the two boxes match. However, there is a value range of 0.5-1 in the boxes (true positives) that match the inference result and the true value result. There is a defect in setting the intersection-over-union ratio of 0.5 and 1 to the same weight. Summary of the invention
[0005] In order to solve the problems in the prior art, the present application provides a data error scoring method for target detection.
[0006] A data error scoring method for target detection, the method comprising:
[0007] Marking targets in multiple images to generate target detection data; and dividing the target detection data into a training set and a test set;
[0008] Performing reasoning on the test set through the training model to generate reasoning results;
[0009] Pairing the inference result with the true value of the training set to obtain true positive examples, false positive examples, and false negative examples;
[0010] Calculating error scores of the true positive examples, false positive examples, and false negative examples respectively, and processing the multiple images according to the error scores;
[0011] The training model is obtained by training the training set using a target detection algorithm.
[0012] Optionally, the marking of objects in the multiple images includes:
[0013] Mark the objects in the image with bounding boxes, annotate the object category of the current bounding box, and generate object detection data.
[0014] Optionally, the ratio of the training set to the test set is 8:2 or the ratio of the training set to the test set is 7:3.
[0015] Optionally, the target detection algorithm includes any one of the YOLO series algorithms and the RCNN series algorithms.
[0016] Optionally, the inference result includes the target category, bounding box coordinates, and confidence level of each target detection data.
[0017] Optionally, pairing the inference result with the true value of the training set to obtain true positive examples, false positive examples, and false negative examples includes:
[0018] Arranging the plurality of inference results in descending order according to confidence;
[0019] Calculate the intersection and union ratio of all inference results and training sets;
[0020] According to the intersection-union ratio and the confidence of the inference result, true positive examples, false positive examples and false negative examples are obtained.
[0021] Optionally, according to the intersection-over-union ratio and the confidence of the inference result, true positive examples, false positive examples and false negative examples are obtained.
[0022] Determine the inference result whose confidence is less than a second threshold as a true counterexample;
[0023] Select the inference result whose intersection-over-union ratio is greater than the first threshold and whose confidence is the largest, and form a true example with the current training set;
[0024] The training sets whose intersection-over-union ratio is less than or equal to the first threshold are judged as false negative examples;
[0025] The inference results without pairing are judged as false positives.
[0026] Optionally, calculating an error score according to the true positive example comprises:
[0027] According to the intersection-and-union ratio of the inference result and the training set bounding box and the confidence of the inference result, the first error score is calculated, and the formula is: mistakenness1 = (1-score+10-3)*(1-iou+10-3), where score is the confidence of the inference result and iou is the intersection-and-union ratio.
[0028] Optionally, calculating an error score according to the false positive example comprises:
[0029] The second error score is calculated by the confidence of the inference result, and the formula is: mistakenness2=0.5*score+0.4, where score is the confidence of the inference result.
[0030] Optionally, calculating an error score according to the false negative example comprises:
[0031] If no matching inference result is found in the training set, the third error score is mistakenness3=1;
[0032] If the training set finds an inference result with an intersection-over-union ratio greater than 0.5, and the target categories of the training set and the inference result are different, the third error score calculation formula is: mistakenness3=0.5*score+0.5, where score is the confidence of the inference result.
[0033] Optionally, the processing of the plurality of images according to the error scores includes:
[0034] According to the first error scores, the second error scores, and the third error scores of the same picture, a target error score of the current picture is calculated;
[0035] Sort multiple images according to the target error score, and select images with target error scores greater than a preset threshold for inspection and modification.
[0036] Optionally, calculating a target error score of the current picture according to the plurality of first error scores, second error scores, and third error scores of the same picture includes:
[0037] sorting the plurality of first error scores, second error scores, and third error scores to form an error score sequence;
[0038] Select some or all of the error scores in the error score sequence, and sum or average them to obtain a target error score for the current image.
[0039] The present application discloses a data error scoring method for target detection, which has the beneficial effect of quantifying the errors of the data set to be cleaned, comprehensively considering the uncertainty of the model output, and optimizing the error score sorting. At the same time, the optimization target is positioned at each box, with smaller granularity and higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 The method flow of the embodiment of this application Figure 1 .
[0041] Figure 2 This is a system block diagram of an embodiment of the present application.
[0042] Figure 3 The method flow of the embodiment of this application Figure 2 . DETAILED DESCRIPTION
[0043] The preferred embodiments of the present application are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present application can be more easily understood by those skilled in the art, thereby making a clearer definition of the protection scope of the present application.
[0044] The same or similar numbers in the drawings of the embodiments of the present application correspond to the same or similar parts; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "top", "bottom", "inside", "outside", etc. indicate the orientation or position relationship, it is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as a limitation on this patent.
[0045] In addition, if there are terms such as "first" and "second", they are only used for descriptive purposes, mainly to distinguish different devices, elements or components (the specific types and structures may be the same or different), and are not used to indicate or imply the relative importance and quantity of the indicated devices, elements or components, and cannot be understood as indicating or implying relative importance.
[0046] Embodiment 1,
[0047] In such Figure 1-2In the illustrated embodiment, the present application discloses a data error scoring method for target detection, the method comprising:
[0048] 100, mark the objects in the multiple images to generate object detection data; and divide the object detection data into a training set and a test set; In step 100, mark the objects in the multiple images, including: mark the objects in the images by bounding boxes, and mark the object category of the current bounding box to generate object detection data. The ratio of the training set to the test set is 8:2 or the ratio of the training set to the test set is 7:3.
[0049] 200, the test set is inferred through the training model to generate an inference result; in step 200, before the test set inference is performed, the training model can be trained through the training set through the YOLO series algorithm and the RCNN series algorithm to obtain a training model. Then the test set is inferred through the training model to obtain the test result. The inference result includes the target category, bounding box coordinates, and confidence level of each target detection data.
[0050] 300, pair the inference results with the true values of the training set to obtain true positive examples, false positive examples, and false negative examples; In step 300, pair the inference results with the true values of the training set to obtain true positive examples, false positive examples, and false negative examples, including: arranging multiple inference results in descending order according to confidence; calculating the intersection-and-union ratio of all inference results and the training set; obtaining true positive examples, false positive examples, and false negative examples according to the intersection-and-union ratio and the confidence of the inference results. According to the intersection-and-union ratio and the confidence of the inference results, true positive examples, false positive examples, and false negative examples are obtained, and the inference results with confidence less than the second threshold are judged as true negative examples; the inference results with the intersection-and-union ratio greater than the first threshold and the largest confidence are selected to form true positive examples with the current training set; the training set with the intersection-and-union ratio less than or equal to the first threshold is judged as a false negative example; the inference results without pairing are judged as false positive examples.
[0051] 400, respectively calculating the error scores of true positive examples, false positive examples, and false negative examples, and processing multiple images according to the error scores; in step 400, calculating the error score according to the true positive example includes: calculating the first error score according to the intersection of the inference result and the training set bounding box, and the confidence of the inference result, and the formula is: mistakenness1 = (1-score + 10-3) * (1-iou + 10-3), where score is the confidence of the inference result, and iou is the intersection of the inference result. Calculating the error score according to the false positive example includes: calculating the second error score according to the confidence of the inference result, and the formula is: mistakenness2 = 0.5 * score + 0.4, where score is the confidence of the inference result. Calculate the error score based on the false negative example, including: if the training set does not find a matching reasoning result, the third error score is mistakenness3=1; if the training set finds a reasoning result with an intersection-over-union ratio greater than 0.5, and the target category of the training set and the reasoning result is different, then the third error score calculation formula is: mistakenness3=0.5*score+0.5, where score is the confidence of the reasoning result. Process multiple images based on the error score, including: according to multiple first error scores, second error scores, and third error scores of the same image, calculate the target error score of the current image; sort multiple images according to the target error score, and select images with a target error score greater than a preset threshold for inspection and modification. Among them, the training model is obtained by training the training set with a target detection algorithm.
[0052] This application quantifies the errors of the data set to be cleaned, comprehensively considers the uncertainty of the model output, and optimizes the error score ranking. At the same time, the optimization target is positioned in each box, with smaller granularity and higher accuracy.
[0053] In some embodiments, the targets in the multiple images are marked, including: marking the targets in the images with bounding boxes, marking the target category of the current bounding box, and generating target detection data. In this embodiment, the application can mark the images manually or by machine. Among them, the targets in the images are marked with bounding boxes, and the target category of the rectangular box is marked, and the marked targets are the target detection data.
[0054] In one embodiment of the above embodiment, the ratio of the training set to the test set is 8:2. In this embodiment, multiple target detection data of multiple images can be selected as the training set, and another part of the multiple images can be the test set. The training model is trained with the training set, and reasoning is performed by inputting the test set into the training model.
[0055] In one embodiment of the above embodiment, the ratio of the training set to the test set is 7:3. In this embodiment, multiple target detection data of multiple images can be selected as the training set, and another part of the multiple images can be the test set. The training model is trained with the training set, and reasoning is performed by inputting the test set into the training model.
[0056] In some embodiments, the target detection algorithm includes any one of the YOLO series algorithms and the RCNN series algorithms. In this embodiment, the target detection algorithm of the present application can select a suitable open source algorithm. The YOLO series, namely You Only LookOnce, is the beginning of the one-stage algorithm in the target detection algorithm. The meaning of You Only Look Once is to complete the target detection process in only one step. The core idea is to use the entire image as the input of the network, and directly regress the position of the bounding box and the category to which the bounding box belongs in the output layer. The RCNN series is a simple and scalable target detection algorithm. RCNN contains three main modules: generating category-independent region proposals to define candidate regions for the perceptron; CNN extracts fixed-length features; and a linear SVM classifier performs category classification. The present application can train the training model using any one of the YOLO series algorithms and the RCNN series algorithms.
[0057] In some embodiments, the inference result includes the target category, bounding box coordinates, and confidence level of each target detection data. 300, pairing the inference result with the true value of the training set to obtain true positive examples, false positive examples, and false negative examples, including:
[0058] 310, multiple inference results are arranged in descending order according to confidence; in step 310, the present application outputs multiple test set quantities to the training model for inference, outputs multiple inference results, and after the inference of the training sets is completed, the inference results are arranged in descending order according to the confidence level.
[0059] 320, calculate the intersection and union ratio of all inference results and the training set; In step 320, in this application, this embodiment calculates the intersection and union ratio of each inference result according to the device reliability level with the true value of the training set.
[0060] 330, according to the intersection-and-union ratio and the confidence of the inference result, obtain true positive examples, false positive examples, and false negative examples. In step 330, according to the intersection-and-union ratio and the confidence of the inference result, obtain true positive examples, false positive examples, and false negative examples, and judge the inference result with a confidence less than the second threshold as a true negative example; select the inference result with an intersection-and-union ratio greater than the first threshold and the largest confidence to form a true positive example with the current training set; judge the training set with an intersection-and-union ratio less than or equal to the first threshold as a false negative example; judge the unpaired inference result as a false positive example. In this embodiment, the application pairs the inference result with the true value of the training set according to the intersection-and-union ratio and the confidence of the inference result, wherein the true value of the training set includes the bounding box coordinates and the target category. The first threshold and the second threshold can be set as required. In this embodiment, the first threshold can be 0.5 and the second threshold can be 0.01.
[0061] In some embodiments, the error score is calculated based on the true example, including: calculating the first error score based on the intersection-and-union ratio of the inference result and the training set bounding box and the confidence of the inference result, and the formula is: mistakenness1=(1-score+10-3)*(1-iou+10-3), where score is the confidence of the inference result and iou is the intersection-and-union ratio.
[0062] Calculating the error score according to the false positive example includes: calculating the second error score according to the confidence of the reasoning result, and the formula is: mistakenness2=0.5*score+0.4, wherein score is the confidence of the reasoning result.
[0063] Calculate error scores based on false negative examples, including:
[0064] If no matching inference result is found in the training set, the third error score is mistakenness3=1; if the training set finds an inference result with an intersection-over-combination ratio greater than 0.5, and the target category of the training set is different from that of the inference result, the third error score calculation formula is: mistakenness3=0.5*score+0.5, where score is the confidence of the inference result.
[0065] In some embodiments, 400, processing a plurality of images according to the error scores includes:
[0066] 410, counting a target error score of the current image based on multiple first error scores, second error scores, and third error scores of the same image; in step 410, counting a target error score of the current image based on multiple first error scores, second error scores, and third error scores of the same image, including: sorting the multiple first error scores, second error scores, and third error scores to form an error score sequence; selecting some or all of the error scores in the error score sequence, and summing or averaging the target error score of the current image.
[0067] At 420, the plurality of images are sorted according to the target error scores, and images having target error scores greater than a preset threshold are selected for inspection and modification.
[0068] In this embodiment, the present application checks and modifies the image by processing multiple error scores.
[0069] Embodiment 2,
[0070] In such Figure 2 In the illustrated embodiment, the present application further provides a data error scoring system for target detection, the system includes the method described in Example 1, and the system includes the following steps:
[0071] Step 1: Mark the object inside the image with a bounding box and annotate the object category of the rectangular box. Divide the labeled object detection data into training sets and test sets. The ratio of training sets to test sets is usually 8:2 or 7:3.
[0072] Step 2: Select a suitable open source target detection algorithm according to your needs, such as the YOLO series and the RCNN series; use the training set to train the algorithm to obtain the training model model.
[0073] Step 3: Use the training model to infer the test set to obtain the inference result. Here, other data with true value labels can be used for inference and the inference result is output. The inference result includes the category, bounding box coordinates, and confidence of each rectangular box target.
[0074] Step 4, by comparing the inference results with the true values, we can get the true positive examples TP, false positive examples FP, and false negative examples FN. The specific comparison method is: arrange the inference results in descending order according to the confidence, calculate the intersection and union ratio of each true value with all the inference results, and select the pair with the intersection and union ratio higher than 0.5 and the largest confidence as the true positive example. The true value of the training set with the intersection and union ratio iou<0.5 is the false negative example, and the remaining inference results are the true negative examples. Before the comparison, all the targets in the inference results with a confidence less than a certain threshold, such as 0.01, are generally deleted, and are no longer compared with the true value, which is equivalent to manually setting this part as the true negative example TN. It is worth mentioning that the true value and the inference value are the only match and cannot be matched multiple times. In addition, the intersection and union ratio threshold must be set, that is, the intersection and union ratio iou of the true value rectangle and the inference rectangle is greater than the threshold to be considered a match. The categories of the two are not compared here. This evaluation method refers to the public COCO evaluation method, and other special cases are also subject to it. Thus, the three results of the batch processing of the output of the previous step of true positive examples TPs, true negative examples FPs, and false negative examples FNs are obtained.
[0075] Step 5: For TPs, design a formula to calculate the error score mistakenness using the iou of the truth box and the reasoning box and the confidence score of the reasoning box. The formula is mistakenness = (1-score + 10-3) * (1-iou + 10-3); for FPs, the calculation formula is mistakenness = 0.5 * score + 0.4; for FNs, there are two cases to discuss: one is that the truth box does not find a matching reasoning box, and the other is that the truth box finds a reasoning box with an iou greater than 0.5, but the categories of the two boxes are different. In the first case, condition one is met, mistakenness = 1; in the second case, condition two is met, mistakenness = 0.5 * score + 0.5.
[0076] Step 6: Use the formula to calculate the result. For a graph, you can select the first n items in the mistakenness sequence and use the sum or average to represent the mistakenness of the graph.
[0077] Step 7. Finally, sort the images according to their mistakenness. The data inspector selects k images with a value greater than the threshold. K is the image in the test set that needs to be checked and modified the most because it has the largest error.
[0078] Step 8: You can also sort the mistakenness of each box. Then, the box with mistakenness = 1 must be checked and corrected. You can also set the mistakenness threshold of other boxes to select k boxes for inspection and correction.
[0079] Embodiment 3,
[0080] The present application also discloses an operation terminal, and the operation terminal runs a program for executing a data error scoring method for target detection in the above-mentioned embodiment 1 or embodiment 2. The present application quantifies the errors of the data set to be cleaned, comprehensively considers the uncertainty of the model output, and optimizes the error score sorting. At the same time, the optimization target is positioned in each box, with smaller granularity and higher accuracy.
[0081] The implementation modes of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in the field without departing from the purpose of the present application.
Claims
1. A data error scoring method for target detection, characterized in that: The method comprises: Marking targets in multiple images to generate target detection data; and dividing the target detection data into a training set and a test set; The test set is inferred through a training model to generate an inference result, wherein the training model is obtained by training the training set through a target detection algorithm, and the inference result includes a target category, a bounding box coordinate, and a confidence level of each target detection data; Pairing the inference result with the true value of the training set to obtain true positive examples, false positive examples, and false negative examples; Calculating error scores of the true positive examples, false positive examples, and false negative examples respectively, and processing the multiple images according to the error scores; The step of pairing the inference result with the true value of the training set to obtain true positive examples, false positive examples, and false negative examples includes: Arranging the plurality of inference results in descending order according to confidence; Calculate the intersection and union ratio of all inference results and training sets; According to the intersection-over-union ratio and the confidence of the inference result, true positive examples, false positive examples and false negative examples are obtained, specifically including: judging the inference result with a confidence less than a second threshold as a true negative example; selecting the inference result with the intersection-over-union ratio greater than the first threshold and the largest confidence to form a true positive example with the current training set; judging the training set with an intersection-over-union ratio less than or equal to the first threshold as a false negative example; judging the unpaired inference result as a false positive example; the first threshold is greater than the second threshold.
2. A data error scoring method for target detection according to claim 1, characterized in that: The marking process of the objects in the multiple images includes: Mark the objects in the image with bounding boxes, annotate the object category of the current bounding box, and generate object detection data.
3. A data error scoring method for target detection according to claim 2, characterized in that: The ratio of the training set to the test set is 8:2 or the ratio of the training set to the test set is 7:
3.
4. The data error scoring method for target detection according to claim 1, characterized in that: The target detection algorithm includes any one of the YOLO series algorithms and the RCNN series algorithms.
5. The data error scoring method for target detection according to claim 1, characterized in that: An error score is calculated based on the true positives, including: According to the intersection-and-union ratio of the inference result and the training set bounding box and the confidence of the inference result, the first error score is calculated, and the formula is: mistakenness1=(1-score+10-3)*(1-iou+10-3), where score is the confidence of the inference result and iou is the intersection-and-union ratio.
6. A data error scoring method for target detection according to claim 5, characterized in that: An error score is calculated based on the false positive examples, including: The second error score is calculated by the confidence of the inference result, and the formula is: mistakenness2=0.5*score+0.4, where score is the confidence of the inference result.
7. A data error scoring method for target detection according to claim 6, characterized in that: An error score is calculated based on the false negative example, including: If no matching inference result is found in the training set, the third error score is mistakenness3=1; If the training set finds an inference result with an intersection-over-union ratio greater than 0.5, and the target category of the training set is different from that of the inference result, the third error score calculation formula is: mistakenness3=0.5*score+0.5, where score is the confidence of the inference result.
8. A data error scoring method for target detection according to claim 7, characterized in that: The processing of the plurality of images according to the error scores comprises: According to the first error scores, the second error scores, and the third error scores of the same picture, a target error score of the current picture is calculated; Sort multiple images according to the target error score, and select images with target error scores greater than a preset threshold for inspection and modification.
9. A method for scoring data errors for target detection according to claim 7, characterized in that: The calculating a target error score of the current picture according to the first error scores, the second error scores, and the third error scores of the same picture includes: sorting the plurality of first error scores, second error scores, and third error scores to form an error score sequence; Select some or all of the error scores in the error score sequence, and sum or average them to obtain a target error score for the current image.