Labeling error defect detection method based on data disassembly
By performing data disassembly and robust classifier training on the data set, suspicious scores are calculated to locate labeling errors, the problem of inaccurate or missing labeling in large-scale data sets is solved, and the data set quality and model performance are improved.
Patent Information
- Application Number
- CN202411743594.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-30
- Publication Date
- 2025-04-18
AI Technical Summary
In large-scale datasets, some instances may have inaccurate or missing labeling problems, which affects the training effect of the model and its performance in actual applications.
A method of detecting labeling error defects based on data disassembly is proposed. By disassembling the original data set into a foreground data set and a background data set, the foreground and background classifiers are trained respectively, the suspicious score is calculated to locate the labeling errors, and manual inspection and correction are performed.
Effectively detect the labeling accuracy of the foreground and background areas in the dataset, reduce the impact of noise data on the model, quickly locate and repair problem labeling in the dataset, and improve the quality of the target detection dataset.
Smart Images
Figure CN119851104A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of target detection, and in particular relates to a method for detecting labeling errors based on data decomposition. Background Art
[0002] In the field of object detection, the performance of the model is highly dependent on the quality of the dataset. Object detection tasks in application scenarios such as autonomous driving, security monitoring, and smart medical care usually require the model to be able to accurately and reliably detect, identify, and locate objects in complex environments. Therefore, the quality of the annotations in the dataset is crucial: incorrect annotations or missing annotations will directly affect the training effect of the model and even cause safety hazards in actual applications. In order to ensure the high reliability of the object detection system, automated annotation error detection and missing label detection of the dataset has become a key requirement.
[0003] In order to efficiently and automatically detect labeling errors and missing labels in target detection datasets, the following background technology solutions are currently available:
[0004] 1. Detection method based on label noise
[0005] This method automatically identifies label noise in the data by analyzing the difference between the model prediction and the annotated label. By calculating the confidence difference between the annotated label and the predicted label, if the prediction confidence of an instance is inconsistent with the label, the instance is marked as a potential noisy label. The label noise detection method can effectively detect incorrect labeling, especially the case of incorrect label category and mismatched bounding boxes.
[0006] 2. Loss-based methods
[0007] Using classification loss and localization loss to detect data failure is another common method. Commonly used loss functions include cross entropy loss, smooth L1 loss, focal loss, and generalized intersection loss (GIoU Loss). By setting a threshold, if the loss value of an instance is too large, it may be due to incorrect labeling or missing labels, resulting in the model failing to make accurate predictions.
[0008] 3. Active learning methods
[0009] Active learning methods use uncertainty metrics to prioritize samples with uncertain annotation quality, thereby guiding manual review. Common metrics include edge metrics and entropy metrics. Edge metrics calculate the difference between the first and second high confidence categories in the instance prediction. The smaller the difference, the greater the uncertainty and the sample may need to be checked. Entropy metrics calculate the entropy value of the predicted category. The higher the entropy, the higher the uncertainty, which may be a potential annotation error or missed instance. Active learning methods focus annotation resources more on samples with high uncertainty, which can effectively improve data quality.
[0010] Although the existing object detection data fault localization technology has improved the quality of the dataset to a certain extent, it still has some key defects, which limits its application effect in large-scale and complex datasets.
[0011] 1. Detection methods based on label noise rely on the difference between model predictions and labels to identify data problems. Therefore, when the model itself is trained with bias or has a lot of noise, the accuracy of detection is often affected. These methods are prone to misclassifying normal labeled instances as noise instances, especially when data distribution is uneven, resulting in more misclassifications and missed detections, affecting the accuracy of fault location.
[0012] 2. Although loss-based algorithms can help identify high-error instances, they are less robust to high-noise data. Samples with large loss values may be due to real labeling errors or errors caused by the model's failure to fully learn the data distribution. Since the loss function itself cannot distinguish between the two, the algorithm is easily affected by noisy data, thereby affecting the detection results. In addition, a single loss threshold is difficult to adapt to the complex data distribution in different scenarios, which can easily lead to a high false alarm rate and increase the burden of manual review.
[0013] 3. Although active learning methods guide manual review through uncertainty measurement, their core is to improve the learning efficiency of the model on unlabeled data rather than focusing on detecting labeling errors. In labeling quality control, active learning methods usually regard instances with high uncertainty as samples that need to be labeled, but these samples are not necessarily instances with incorrect or missing labels, so they cannot effectively locate faults in the data set. In addition, active learning relies on the model's judgment of uncertainty, so when the model itself lacks the ability to recognize specific categories or specific scenarios, its uncertainty measurement will also lose accuracy, thereby reducing the reliability of fault location. Summary of the invention
[0014] 1. Technical issues to be resolved
[0015] The technical problem to be solved by the present invention is how to provide a method for detecting labeling errors based on data disassembly, so as to solve the problem that some instances in large-scale data sets may have inaccurate or missing labels. These data quality defects will directly affect the training effect of the model and its performance in practical applications.
[0016] (II) Technical solution
[0017] In order to solve the above technical problems, the present invention proposes a method for detecting labeling errors based on data decomposition, which comprises the following steps:
[0018] S1. Data decomposition: The original object detection dataset is decomposed into a foreground dataset and a background dataset. The foreground dataset only contains the foreground objects in the annotation box, while the background dataset blurs the foreground areas in all the annotation boxes and retains the background areas.
[0019] S2. Training classifier: Use the foreground dataset and the background dataset to train two classification models respectively. Considering that the quality of the dataset may be poor, a robust loss function is used to enhance the model's ability to resist interference from noisy data.
[0020] S3. Generate suspicious scores: Calculate the suspicious scores of the foreground annotation box and the background area based on the prediction results of the classification model; the higher the score, the greater the probability that there is data failure in the area;
[0021] S4. Manual inspection and correction: Based on the suspicion scores from high to low, priority is given to checking and correcting the suspicious areas with the highest ranking; corrections include: modifying incorrect annotations, adjusting the position of the positioning box, and adding missing annotations.
[0022] (III) Beneficial effects
[0023] The present invention proposes a method for detecting labeling errors based on data decomposition. The method of the present invention effectively detects the labeling accuracy of the foreground and background areas in the data set through the steps of data set decomposition, robust classifier training and suspicion score calculation. First, the labeling information of each image is decomposed, and the foreground target and background area are taken as independent instance units; then a robust classifier is used for training to reduce the impact of noise data on the model, and then the priority of each instance is determined by calculating the suspicion score. The instance with a higher suspicion score indicates that it is more likely to be labeled incorrectly, and thus it is checked and corrected first. The method of the present invention helps developers quickly locate and repair problematic labels in a data set without relying on model prediction results. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 This is a technical roadmap for the labeling error defect detection method based on data disassembly of the present invention. DETAILED DESCRIPTION
[0025] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.
[0026] The purpose of the present invention is to propose a method for detecting labeling errors based on data disassembly, which is a method for locating target detection data faults based on data disassembly, and is used to automatically identify and locate possible labeling errors and missing labeling instances in a data set, thereby improving the quality of the target detection data set and optimizing model performance. In the target detection task, the model extracts the target object from the input image and outputs its category, location, and related labeling information. Since some instances may have inaccurate or missing labels in large-scale data sets, these data quality defects will directly affect the training effect of the model and its performance in practical applications.
[0027] Currently, labeling errors for target recognition can be divided into the following four situations.
[0028] 1. Classification Error: This means that the category of the target object is incorrectly labeled, such as mislabeling a "dog" as a "cat".
[0029] 2. Localization Error: This refers to the inaccurate position of the bounding box, which may not correctly cover the entire area of the target object, resulting in the offset or size error of the bounding box.
[0030] 3. Redundant Annotation: It means that a target object is repeatedly annotated with multiple bounding boxes, resulting in redundant annotation information.
[0031] 4. Missing Annotation: refers to the target object in the image not being labeled, resulting in the lack of corresponding annotation boxes in the data set.
[0032] The present invention detects classification errors through foreground and background classifiers, identifies positioning errors by combining foreground-background decomposition strategies, and uses background classifiers and background subsets to detect missing labels, thereby effectively addressing four common data failures in target detection data sets. Through these targeted designs, the present invention can more efficiently locate and correct labeling errors in data sets, and improve the overall quality of target detection data.
[0033] In order to detect the mislabeled data in the target detection and target tracking data set, the present invention proposes a labeling error defect detection technology based on data disassembly for target detection and target tracking, as follows Figure 1 The implementation process of this method is as follows:
[0034] S1. Data decomposition: The original object detection dataset is decomposed into a foreground dataset and a background dataset. The foreground dataset only contains the foreground objects in the annotation box, while the background dataset blurs the foreground areas in all the annotation boxes and retains the background areas.
[0035] S2. Training classifiers: Use the foreground dataset and the background dataset to train two classification models respectively. Considering that the quality of the dataset may be poor, a robust loss function is used to enhance the model's ability to resist interference from noisy data.
[0036] S3. Generate suspicious scores: Calculate the suspicious scores of the foreground annotation box and the background area based on the prediction results of the classification model. The higher the score, the greater the probability that there is data failure in the area.
[0037] S4. Manual inspection and correction: Based on the suspicion scores from high to low, priority is given to checking and correcting the suspicious areas with the highest ranking. Correction includes modifying incorrect annotations, adjusting the position of the positioning box, adding missing annotations, etc.
[0038] Step 1: Data decomposition
[0039] For a given dataset, each image in the dataset contains several target instances, some of which may be mislabeled, so the image-level mislabeling detection can be converted to the target instance-level mislabeling detection. The target detection dataset is converted into a simpler classification dataset, and the original dataset is decomposed into two datasets through the data decomposition method: a dataset containing only the foreground and a dataset containing the background. The data decomposition strategy decomposes each image into separate target instances and treats the foreground objects and background areas separately. All labeled foreground objects are extracted from the image through foreground extraction, and each foreground object is regarded as an independent sample to form a foreground dataset. By treating the unlabeled areas in the image as the background area, a classification dataset containing the background is formed. Through this decomposition strategy, it is possible to analyze the labeling errors and missing labels in the data in a more fine-grained manner.
[0040] Given an image I, the set of object instances annotated in I is denoted by F(I) = {(Loc(I) k ,Cls(I) k )|k=1,…,a i}, where Loc(I) k is the position of the kth labeled object instance, Cls(I) k is the category of the kth labeled object instance, a i is the number of labeled objects. For any two objects α, β∈F(I), there are three positional relationships, namely overlap, inclusion and separation. These relationships can be formalized as follows:
[0041] 1. Overlap: The bounding boxes of two objects at least partially overlap.
[0042] Area(α∩β)>0∧Area(α∩β) <minArea(α,β)
[0043] 2. Containment: The bounding box of one object completely contains another object.
[0044] Area(α∩β)>0∧Area(α∩β)=minArea(α,β)
[0045] 3. Separation: The bounding boxes of the two objects do not overlap.
[0046] Area(α∩β)=0
[0047] For each annotated bounding box α∈F(I), the present invention proposes the following masking strategy to blur irrelevant foreground annotations of image I:
[0048] 1. If β∈F(I) overlaps with α, blur the region of β\α, that is, blur only the region that does not contain object α;
[0049] 2. If β∈F(I) contains α, we assume that α is a smaller object in front of β, and blur the region of β\α;
[0050] 3. If β∈F(I) is contained by α, we assume that β is a smaller object in front of α, and blur the β region;
[0051] 4. If β∈F(I) does not intersect α, then blur the β region.
[0052] In addition to the foreground image, a background image of input I is also generated, where the union of all bounding boxes is blurred to keep only the background area.
[0053] In the dataset containing only foreground, consider cropping each annotated bounding box from the image, blurring the unrelated areas, and assigning the same annotation as the corresponding category. Therefore, the disassembled pure foreground dataset only contains annotated information, which is consistent with the categories defined in the object detection dataset. The representation of this dataset is as follows:
[0054] D F ={(x f ,y f )|x f ∈R W×H ,y f ∈R n}
[0055] Among them, x f Represents the cropped input data, which maintains the same width and height ratio after scaling, y f Represents x f The corresponding category information, n is the number of categories to be detected in the original target detection dataset.
[0056] In datasets containing background, an additional background class is added to compensate for the missing objects to be detected. This class corresponds to processed images where all annotated foreground regions are blurred. Therefore, an input can be decomposed into k+1 independent instance units, where k is the number of bounding boxes in the input. The representation of this dataset is as follows:
[0057] D B ={(x b ,y b )|x b ∈R W×H ,y b ∈R n+1}
[0058] With D F In comparison, D B Contains an additional background class.
[0059] Step 2: Train the classifier
[0060] After the original data set is disassembled, the disassembled data set D is used F and D B Train the corresponding classifier f F and f B , design the suspicion score according to the reasoning result of the classifier. The goal of classifier training is to minimize the value of the loss function by adjusting and modifying the internal parameters. Both the foreground classifier and the background classifier are sampled from the corresponding data set and robustness training is performed. The accuracy of the two is compared to generate an optimal classifier for subsequent prediction.
[0061] Considering that the disassembled dataset may inherit the data errors of the original dataset, it may not only contain samples of the wrong category, but also various types of noise. Therefore, when training the classifier, a robust training loss, namely the generalized cross entropy loss, is chosen to effectively combat the influence of label noise. The generalized cross entropy loss is denoted by L q , and its calculation formula is as follows:
[0062]
[0063] Where x and y represent the input and one-hot encoded class labels, respectively, f(x) is the classification function, q is the order, and e j is the label of the jth sample, j is the sample number, and f j (x) is the classification of the jth sample; when limq→0, L q (f(x), y) is equivalent to the classification cross entropy loss CCE; when q = 1, L q(f(x), y) is equivalent to the mean absolute error loss MAE. The categorical cross entropy loss function is a commonly used loss function with a fast convergence speed. However, it is significantly affected by mislabeled samples, resulting in poor noise tolerance. In contrast, the mean absolute error loss is more robust to noisy labels, but its training is time-consuming and prone to underfitting. Therefore, L q It effectively balances the advantages of the classification cross loss (CCE) and the mean absolute error loss (MAE), and well addresses the challenges of different error types in our disassembled dataset.
[0064] Step 3: Generate a suspiciousness score for annotation errors
[0065] This step predicts the foreground and background areas in the original image based on the trained classifier and calculates the suspicion score of each area.
[0066] For a given dataset D and its annotation A, the components in dataset D that need to be sorted are S = ∪ I∈D {F(I),F(I bg )}, where F(I bg ) represents the input where all foreground regions are blurred. For each component s∈S, the suspiciousness score W(s) is expressed as:
[0067] 1. If s∈F(I), the model with better prediction performance is recorded as f∈{f F , f B}, then W(s)=L CCE (f(x s ),y s ). That is, for the foreground object, select the classifier f with better prediction performance F or B , calculate the cross entropy loss between the predicted result and the true category as the suspiciousness score of the foreground object;
[0068] 2. If s∈F(I bg ), then W(s)=L CCE (f B (x s ),y s ). That is, for each input background region, use the background model f B , calculate the cross entropy loss between its prediction result and the background category as the suspiciousness score of the background area.
[0069] Finally, each input foreground annotation and background region is assigned a suspicion score, with higher scores indicating greater suspicion of its annotation error.
[0070] Step 4: Manual inspection and correction
[0071] After the suspiciousness score is calculated, the human inspector can quickly locate and correct the annotation errors and missing annotations based on the suspiciousness score prompts. The specific process includes:
[0072] S41. Prioritize the suspicious areas with the highest rankings according to the suspicion scores. High suspicion scores indicate that there is a high probability of errors or omissions in these areas;
[0073] S42, for each suspicious area, observe its annotation information and background area, and check whether there is any wrong annotation, inaccurate positioning or missing annotation;
[0074] S43, correcting the suspicious area according to the observation results. The correction method includes: correcting the wrong annotation to a correct annotation, adjusting the positioning frame to more accurately surround the target object, adding a missing annotation frame, etc.;
[0075] S44. Check and correct each suspicious area one by one in descending order of suspicion score until all areas have been checked and corrected. During the correction process, pay attention to maintaining the consistency of annotation to ensure the quality of annotation.
[0076] The method of the present invention effectively detects the accuracy of annotation of foreground and background areas in a data set through steps such as data set decomposition, robust classifier training, and suspicion score calculation. First, the annotation information of each image is decomposed, and the foreground target and background area are taken as independent instance units; then a robust classifier is used for training to reduce the impact of noise data on the model, and then the priority of each instance is determined by calculating the suspicion score. The instance with a higher suspicion score indicates that it is more likely to be incorrectly labeled, and thus it is checked and corrected first. The method of the present invention helps developers quickly locate and repair problematic annotations in a data set without relying on model prediction results.
[0077] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for detecting labeling errors based on data decomposition, characterized in that: The method comprises the following steps: S1. Data decomposition: The original object detection dataset is decomposed into a foreground dataset and a background dataset. The foreground dataset only contains the foreground objects in the annotation box, while the background dataset blurs the foreground areas in all the annotation boxes and retains the background areas. S2. Training classifier: Use the foreground dataset and the background dataset to train two classification models respectively. Considering that the quality of the dataset may be poor, a robust loss function is used to enhance the model's ability to resist interference from noisy data. S3. Generate suspicious scores: Calculate the suspicious scores of the foreground annotation box and the background area based on the prediction results of the classification model; the higher the score, the greater the probability that there is data failure in the area; S4. Manual inspection and correction: Based on the suspicion scores from high to low, the top suspicious areas are inspected and corrected first; Corrections include: modifying incorrect annotations, adjusting the positioning box position, and adding missing annotations.
2. The method for detecting labeling errors based on data decomposition according to claim 1, characterized in that: In S1, for a given data set, each image in the data set contains several target instances, some of which may be incorrectly labeled. Therefore, the image-level labeling error detection is converted to the target instance-level labeling error detection, and the target detection data set is converted into a simpler classification data set. The original data set is decomposed into two data sets through a data decomposition method: a data set containing only the foreground and a data set containing the background.
3. The method for detecting labeling errors based on data decomposition according to claim 2, characterized in that: The data decomposition strategy decomposes each image into separate target instances and processes the foreground objects and background areas separately; all labeled foreground objects are extracted from the image through foreground extraction, and each foreground object is regarded as an independent sample to form a foreground dataset; and the unlabeled areas in the image are regarded as background areas to form a classified dataset containing the background.
4. The method for detecting labeling errors based on data decomposition according to claim 1, characterized in that: In S1, given an image I, the set of object instances annotated in I is denoted as F(I)={(Loc(I) k ,Cls(I) k )|k=1,…,a i }, where Loc(I) k is the position of the kth labeled object instance, Cls(I) k is the category of the kth labeled object instance, a i is the number of labeled objects; for any two objects α, β∈F(I), there are three positional relationships, namely overlap, inclusion and separation; these relationships are formalized as follows: Overlap: The bounding boxes of the two objects at least partially overlap Area(α∩β)>0∧Area(α∩β) <minArea(α,β) Contains: The bounding box of one object completely contains another object Area(α∩β)>0∧Area(α∩β)=minArea(α,β) Separation: The bounding boxes of the two objects do not overlap Area(α∩β)=0.
5. The method for detecting labeling errors based on data decomposition according to claim 4, characterized in that: For each annotated bounding box α∈F(I), the following masking strategy is used to blur irrelevant foreground annotations of image I: If β∈F(I) overlaps with α, blur the region of β\α, that is, blur only the region that does not contain object α; If β∈F(I) contains α, assuming that α is a smaller object in front of β, the region of β\α is blurred; If β∈F(I) is contained by α, assume that β is a smaller object in front of α, then blur the β region; If β∈F(I) does not intersect α, then the β region is blurred; In addition to the foreground image, a background image of input I is also generated, where the union of all bounding boxes is blurred to keep only the background area.
6. The method for detecting labeling errors based on data decomposition according to claim 5, characterized in that: In the dataset containing only the foreground, consider cropping each annotated bounding box from the image, blurring the unrelated areas, and assigning the same annotation as the corresponding category; therefore, the disassembled pure foreground dataset only contains the annotated information, which is consistent with the categories defined in the target detection dataset; the representation of this dataset is as follows: D F ={(x f ,y f )|x f ∈R W×H ,y f ∈R n } Among them, x f Represents the cropped input data, which maintains the same width and height ratio after scaling, y f Represents x f The corresponding category information, n is the number of categories to be detected in the original target detection dataset.
7. The method for detecting labeling errors based on data decomposition according to claim 6, characterized in that: In the dataset containing background, an additional background class is added to compensate for the missing objects to be detected; this class corresponds to the processed image in which all annotated foreground areas are blurred; therefore, an input is decomposed into k+1 independent instance units, where k is the number of bounding boxes in the input; the representation of this dataset is as follows: D B ={(x b ,y b )|x b ∈R W×H ,y b ∈R n+1 } With D F In comparison, D B Contains an additional background class.
8. The method for detecting labeling errors based on data decomposition according to claim 7, characterized in that: The S2 comprises: after disassembling the original data set, using the disassembled data set D F and D B Train the corresponding classifier f F and f B , design a suspiciousness score based on the inference results of the classifier; the goal of classifier training is to minimize the value of the loss function by adjusting and modifying internal parameters; both the foreground classifier and the background classifier are sampled from the corresponding data set and robustness training is performed, and the accuracy of the two is compared to generate an optimal classifier for subsequent prediction; Considering that the disassembled dataset may inherit the data errors of the original dataset, it may not only contain samples of the wrong category, but also various types of noise. Therefore, when training the classifier, a robust training loss, namely the generalized cross entropy loss, is chosen to effectively combat the influence of label noise. The generalized cross entropy loss is denoted by L q , and its calculation formula is as follows: Where x and y represent the input and one-hot encoded class labels, respectively, f(x) is the classification function, q is the order, and e j is the label of the jth sample, j is the sample number, and f j (x) is the classification of the jth sample; when limq→0, L q (f(x), y) is equivalent to the classification cross entropy loss CCE; when q = 1, L q (f(x), y) is equivalent to the mean absolute error loss MAE.
9. The method for detecting labeling errors based on data decomposition according to claim 8, characterized in that: S3 includes: for a given data set D and its annotation A, the components in the data set D that need to be sorted are S=∪ I∈D {F(I),F(I bg )}, where F(I bg ) represents the input with all foreground regions blurred; for each component s∈S, the suspiciousness score W(s) is expressed as: If s∈F(I), the model with better prediction performance is denoted as f∈{f F , f B }, then W(s)=L CCE (f(x s ),y s ); that is, for the foreground object, select the classifier f with better prediction performance F or B , calculate the cross entropy loss between the predicted result and the true category as the suspiciousness score of the foreground object; If s∈F(I bg ), then W(s)=L CCE (f B (x s ),y s ); That is, for each input background area, use the background model f B , calculate the cross entropy loss between its prediction result and the background category as the suspiciousness score of the background area; Finally, each input foreground annotation and background region is assigned a suspicion score, with higher scores indicating greater suspicion of its annotation error.
10. The method for detecting labeling errors based on data decomposition according to claim 9, characterized in that: S4 includes: after the suspiciousness score is calculated, the manual inspector quickly locates and corrects the labeling errors and missing labels according to the suspiciousness score prompt; the specific process includes: S41. Prioritize the suspicious areas with the highest rankings according to the suspicion scores; high suspicion scores indicate that there is a high probability of errors or omissions in these areas; S42, for each suspicious area, observe its annotation information and background area, and check whether there is any wrong annotation, inaccurate positioning or missing annotation; S43, correcting the suspicious area according to the observation result; the correction method includes: correcting the wrong annotation to a correct annotation, adjusting the positioning frame to more accurately surround the target object, and adding a missing annotation frame; S44. Check and correct each suspicious area one by one in descending order of suspicion score until all areas have been checked and corrected. During the correction process, pay attention to maintaining consistency in annotation to ensure annotation quality.