Control unit for improving annotation system in smart model
By optimizing sample selection through error analysis and sampling strategy modules in the control unit, the problem of poor performance of existing annotation systems in rare categories is solved, achieving more efficient training and lower-cost label coverage, thus improving the accuracy and efficiency of the annotation system.
Patent Information
- Application Number
- CN202511503492.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-10-22
- Filing Date
- 2025-10-21
- Publication Date
- 2026-04-24
AI Technical Summary
Existing annotation systems perform poorly when processing multi-granularity image data, especially rare categories. Furthermore, existing systems assume that human labels are correct and fail to effectively handle human label errors, resulting in low training efficiency and high costs.
The control unit uses an error analysis module to identify missing and mislabeled labels. Combined with data statistics and network uncertainty factors, a sampling strategy module optimizes sample selection and iteratively fine-tunes the intelligent model until the desired performance is achieved.
It improves the performance of intelligent models on rare categories, reduces the need for labeled data, lowers costs, and can more accurately cover labels on the dataset, thus improving the efficiency of the annotation system.
Smart Images
Figure CN121921585A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a control unit for improving the annotation system in intelligent models. Background Technology
[0002] An annotation system is a system for labeling any data sample in a human-understandable format. In the case of image modal data, the label of a single data point (image or video) can exist at multiple granularity levels, such as pixel-level semantic class, object detection, object localization, class labeling, etc. Annotation systems can be implemented through manual or automated machine labeling.
[0003] A technical paper titled "Deep Active Learning for Computer Vision: Past and Future" explains the concept of generating samples with the highest uncertainty rather than selecting from available samples. Attached Figure Description
[0004] Figure 1 The illustration shows a control unit for improving an annotation system in an intelligent model according to an embodiment of the present invention; and Figure 2 The diagram illustrates a flowchart of a method for improving an annotation system in an intelligent model according to the present invention. Detailed Implementation
[0005] Figure 1 The illustration depicts a control unit for improving an annotation system in an intelligent model according to an embodiment of the present invention. The control unit 10 receives data from multiple data sources 15 to form a dataset 11 and fine-tunes an intelligent model 14 using the received data. The control unit 10 then determines a prediction / label for each data point of the received data and identifies at least one error in the determined prediction / label for each data point of the received data via an error analysis module 16. The control unit 10 transmits samples with the identified errors back to the dataset via a sampling strategy module 18 for fine-tuning the intelligent model 14, thereby improving the annotation system 12 in the intelligent model 14.
[0006] The construction of the control unit and the operation of its components are explained in further detail. Control unit 10 is an operational device for processing and storing information received from at least one source. Control unit 10 is selected from a group of control units, including microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, one or more microchips or integrated circuits interconnected using a motherboard, hard-wired logic, software stored in a memory device and executed by a microprocessor, firmware, application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs), and / or any component that operates on signals based on operating instructions. It should be noted that the control unit can be any other kind, and is not limited to the devices described above known to those skilled in the art. The identified errors are selected from any of the error types, including missing labels and mislabeled errors. Missing labels refer to objects that were not labeled by the labeler during the human annotation process, while mislabeled errors refer to objects that were partially or completely mislabeled. The error analysis module includes two parameters related to the identification of at least one error: a label inconsistency factor and a network uncertainty factor. The label inconsistency factor is derived from a comparison between human labels and predictions, and the network uncertainty factor is derived from the confidence measure of Model 14 regarding its predictions. Furthermore, the network uncertainty factor is derived by utilizing the predicted distribution values and calculating the entropy score for each pixel in each data point.
[0007] The sampling strategy module utilizes metadata from the error analysis module, combined with data statistics from the network uncertainty module, to fine-tune the intelligent module. During iterative training of the intelligent module, the sampling strategy module uses any of the following techniques to reduce the heuristic function: rank ranking of samples derived from the error analysis module, frequency of samples derived from data statistics, distribution of samples derived from error analysis and / or data statistics, and patches from samples derived from the error analysis module. The control unit repeats the sampling-based retraining / fine-tuning of the intelligent module until the desired performance is achieved.
[0008] Figure 2A flowchart illustrating a method for improving an annotation system in an intelligent model according to the present invention is shown. In step S1, data from multiple data sources 15 are received to form a dataset 11, and the intelligent model 14 is fine-tuned using the received data. In step S2, a prediction / label for each data point of the received data is determined. In step S3, an error analysis module 16 identifies at least one error in the determined prediction / label for each data point of the received data. In step S4, samples with identified errors are transmitted back to the dataset via a sampling strategy module 18 for fine-tuning the intelligent model 14, thereby improving the annotation system in the intelligent model 14.
[0009] The method is explained in detail. The performance of the annotated model 12 on dataset 11 is improved by utilizing an iterative training strategy. Dataset 11 in the fields of machine learning and artificial intelligence is severely skewed in its distribution. In this case, labels with a higher percentage of presence are called "frequent" classes, while labels with a lower percentage are called "rare" classes. For example, for a specific task of image segmentation (in the context of autonomous driving), one will observe that the percentage of pixels in the "sky" class will be much higher than that in the "animal" class.
[0010] When training an intelligent model 14 on such a dataset, performance is lower on such rare classes compared to frequent classes. The goal of this invention is to improve performance on such “rare” classes. Furthermore, typical iterative training systems consider network-based uncertainty (in prediction) and select (permutate) samples solely for retraining purposes. A problem arises here because such systems assume human labels are “oracles” and therefore there are no errors in human labels. However, there is a high probability of errors occurring in these types of scenarios. To better understand this invention, two types of errors are explained here. One type of error observable in such systems is the omission labeling error, where the labeler fails to label the object during the human annotation process, and the other type is the mislabeling error, where the object is partially or completely mislabeled.
[0011] Data set 11 receives data from various sources 15 and stores it in the memory 20 of control unit 10. Using the received data, control unit 10 fine-tunes intelligent model 14 and then determines a prediction / label associated with each data point in the dataset. According to one embodiment of the invention, dataset 11 includes multiple images and labels / predictions identified on each image in dataset 11. Each image is referred to as a data point.
[0012] Control unit 10 captures human annotation errors through error analysis module 16 (which is of a general type). Error analysis module 16 utilizes a pair of modalities when identifying errors: one is a label inconsistency factor (a comparison between human labels and predictions), and the other is a network-based uncertainty factor (errors based on the intelligent model). Based on these modalities / factors, error analysis module 16 of control unit 10 identifies the types of errors that have occurred. Once an error is identified in the prediction / label, sample strategy module 18 combines data statistics with metadata from error analysis module 16.
[0013] The sampling strategy module 18 is then simplified to a heuristic function that provides batches during iterative training. Some of the heuristic functions used by the sampling strategy module are listed below: (1) providing a rank order for samples / predictions derived from the error analysis module 16, (2) determining the frequency of samples / predictions derived from data statistics / network uncertainty factors, (3) the distribution of samples derived from the error analysis module 16 and / or data statistics factors, (4) tiles / labels from samples derived from the error analysis module 16, etc. It should be understood that the heuristic functions are based on combinations of these functions, but are not limited to them. The sampling strategy module 18 then transmits samples with identified errors back to the dataset for fine-tuning the intelligent model 14, thereby improving the annotation system 12 in the intelligent model 14.
[0014] The above process is repeated / iterated multiple times by using sampling-based retraining / fine-tuning of the intelligent model 14 until the desired performance is achieved.
[0015] Based on the aforementioned methodological techniques, the intelligent model 14 will exhibit better performance on rare classes compared to ordinary models. This will lead to a reduction in the need for samples. Utilizing this methodological technique, costs are reduced because it eliminates the need for a costly process of acquiring large amounts of labeled data. The aforementioned method allows for improved performance on rare classes without requiring additional data and also allows the intelligent model 14 to capture human labeling errors (including missed and mislabeled labels). This allows for better and more accurate label coverage on the dataset.
[0016] It should be understood that the embodiments described above are merely illustrative and do not limit the scope of the invention. Many such embodiments and other modifications and variations of the embodiments explained in the specification are contemplated. The scope of the invention is defined only by the scope of the claims.
Claims
1. A control unit (10) for improving the annotation system (12) in an intelligent model (14), said control unit (10) being adapted to: - Receive data from multiple data sources (15) to form a dataset (11), and use the received data to fine-tune the intelligent model (14). - Determine the prediction / label for each data point in the received data; - The error analysis module (16) identifies at least one error in the determined prediction / label for each data point of the received data; - Samples with identified errors are transmitted back to the dataset (11) via the sampling strategy module (18) for fine-tuning the intelligent model (14), thereby improving the annotation system (12) in the intelligent model (14).
2. The control unit (10) according to claim 1, wherein, The identified errors are selected from any of the error types, including missing labels and mislabeled labels.
3. The control unit (10) according to claim 2, wherein, Missed labeling errors are errors in which the labeler fails to label objects during the human annotation process, while mislabeling errors are errors in which objects are labeled wholly or partially incorrectly.
4. The control unit (10) according to claim 1, wherein, The error analysis module (16) includes two parameters related to the identification of the at least one error: one is a label inconsistency factor, and the other is a network uncertainty factor.
5. The control unit (10) according to claim 4, wherein, The label inconsistency factor is derived from the comparison between human labels and predictions, and the network uncertainty factor is derived from the confidence measure of the model with respect to its predictions.
6. The control unit (10) according to claim 5, wherein, The network uncertainty factor is derived by utilizing the predicted distribution values and calculating the entropy score of each pixel in each data point.
7. The control unit (10) according to claim 1, wherein, The sampling strategy module (18) is adapted to use metadata from the error analysis module (16) combined with data statistics from the network uncertainty module for fine-tuning the intelligent model (14).
8. The control unit (10) according to claim 1, wherein, The sampling strategy module (18) uses any of the following techniques to reduce the heuristic function during the iterative training of the intelligent model (14): the rank order of samples derived from the error analysis module (16), the frequency of samples derived from data statistics, the distribution of samples derived from error analysis and / or data statistics, and the tiles from samples derived from the error analysis module (16).
9. The control unit (10) according to claim 1, wherein, The control unit (10) is adapted to repeatedly retrain / fine-tune the intelligent model (14) based on sampling until the desired performance is achieved.
10. A method for improving the annotation system (12) in an intelligent model (14), the method comprising the steps of: - Receive data from multiple data sources (15) to form a dataset (11), and use the received data to fine-tune the intelligent model (14). - Determine the prediction / label for each data point in the received data; - The error analysis module (16) identifies at least one error in the determined prediction / label for each data point of the received data; - Samples with identified errors are transmitted back to the dataset via the sampling strategy module (18) for fine-tuning the smart model, thereby improving the annotation system (12) in the smart model (14).