A sample labeling method, device, equipment and medium
By using multi-model fusion and time series tracking algorithms, the problems of slow speed, low efficiency and low annotation quality in existing technologies have been solved, achieving efficient and stable automated sample annotation and reducing mislabeling and omissions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN JIMU INTELLIGENT TECH CO LTD
- Filing Date
- 2022-11-21
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies suffer from slow speed, low efficiency, and low annotation quality in manual annotation, while semi-automatic annotation methods based on deep learning models suffer from false detections, missed detections, and unstable accuracy.
Multiple detection models are used to perform forward inference on each frame of image, the prediction results are fused, and target tracking and verification are performed using information from previous frames in the time series to generate ground truth labels. The labeling accuracy and stability are improved through multi-model voting and time series tracking algorithms.
It improves annotation quality and stability, reduces the workload of manual review, increases annotation efficiency, reduces mis-annotation, and achieves automated annotation around the clock.
Smart Images
Figure CN115937580B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image annotation technology, and in particular to a sample annotation method, apparatus, device and medium. Background Technology
[0002] With the rapid development of the automotive industry, the demand for road element perception in ADAS application scenarios is increasing, and the need for data based on ADAS scenarios is becoming more urgent, with higher requirements for sample annotation quality and annotation standards. Existing sample annotation methods mainly include: 1. Manual annotation methods: These involve establishing unified annotation specifications, training annotators on the corresponding annotation rules, and manually annotating the elements that need annotation according to the specifications. 2. Semi-automatic annotation methods based on deep learning models: This involves pre-annotating a subset of samples, using these samples to train the model, and then using the model to infer the samples that need annotation, thereby annotating the elements. After the model inference and annotation are completed, the annotated data needs to be manually reviewed, and any problems modified or adjusted.
[0003] Manual annotation requires each perceptual element to be labeled manually, which is a huge workload and prone to omissions and errors. In addition, different annotators have different understandings of the annotation rules, resulting in significant differences in the annotation quality among different annotators. Furthermore, manual annotation is slow and inefficient.
[0004] Semi-automatic annotation methods based on deep learning models have significantly improved annotation efficiency, but they rely on annotation of single frames, leading to issues such as false positives and false negatives. Manual review still requires identifying and correcting any potential annotation problems. Furthermore, the performance of the chosen deep learning model determines the upper limit of the annotation data's accuracy; using an unsuitable or incorrect deep learning model can result in unstable accuracy. Summary of the Invention
[0005] In view of this, embodiments of the present invention provide a sample annotation method to solve the problems of slow and inefficient manual annotation and low annotation quality of single-model annotation methods in the prior art. The method includes:
[0006] For each frame of the sample to be labeled, multiple detection models are used to perform forward inference on the current frame image to obtain multiple prediction results for the current frame image;
[0007] Multiple prediction results for the current frame image are fused to obtain the initial label for the current frame image;
[0008] Obtain the posterior information of the previous frame image of the current frame image in the time series, and obtain the optimal annotation result of the previous frame image;
[0009] The tracking label for the current frame image is calculated using the optimal annotation result of the previous frame image.
[0010] The ground truth label of the current frame image is obtained by verifying the initial label and the tracking label of the current frame image.
[0011] This invention also provides a device for time series multi-model annotation, which solves the technical problems of slow annotation speed, low efficiency, and low annotation quality in the prior art. The device includes:
[0012] The detection model training module takes unlabeled sample images from the training sample set and inputs them into the various models included in the detection model for prediction processing, and obtains the prediction results output by each detection model.
[0013] The image preprocessing module normalizes the image to make it a standard image and reduces image noise.
[0014] The single-model single-frame tracking module outputs the results of one detection model for each frame to obtain the optimal annotation result for a single frame.
[0015] The time series tracking module tracks the posterior information of the previous frame of the current frame in the time series and obtains the tracking label of the current frame image;
[0016] The initial label and tracking label fusion verification module performs target fusion verification on the initial label and tracking label to generate a truth label.
[0017] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned sample annotation methods, thereby solving the technical problems of slow annotation speed, low efficiency, and low annotation quality in the prior art.
[0018] This invention also provides a computer-readable storage medium storing a computer program that executes any of the above-described sample annotation methods, thereby solving the technical problems of slow annotation speed, low efficiency, and low annotation quality in the prior art.
[0019] Compared with the prior art, the beneficial effects that at least one technical solution adopted in the embodiments of this specification can achieve include at least:
[0020] This invention utilizes multiple detection models to fuse targets and outputs the results for the current frame. Simultaneously, it leverages targets from the time series for verification, improving the quality and stability of ground truth generation and reducing manual review and correction. Compared to semi-automatic annotation methods based on deep learning models, this invention fully utilizes historical information from the time series and employs multiple detection model samples for annotation, resulting in higher annotation accuracy, fewer mislabeling instances, and a smaller workload for manual review. Attached Figure Description
[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a flowchart of a sample annotation method provided in an embodiment of the present invention;
[0023] Figure 2 This is a schematic diagram illustrating the principle of a sample annotation method provided in an embodiment of the present invention;
[0024] Figure 3 This is a flowchart of the model forward inference provided in the embodiments of the present invention;
[0025] Figure 4 This is a flowchart of a single detection model and single-frame target output tracking provided by an embodiment of the present invention;
[0026] Figure 5 This is a structural block diagram of a computer device provided in an embodiment of the present invention;
[0027] Figure 6 This is a structural block diagram of a sample labeling device provided in an embodiment of the present invention;
[0028] Figure 7 This is a schematic diagram of the weighted calculation of the previous frame in a time series provided in an embodiment of the present invention. Detailed Implementation
[0029] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0030] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0031] In this embodiment of the invention, a sample annotation method is provided, which mainly includes:
[0032] For each frame of the sample to be labeled, multiple detection models are used to perform forward inference on the current frame image to obtain multiple prediction results for the current frame image;
[0033] Multiple prediction results for the current frame image are fused to obtain the initial label for the current frame image;
[0034] Obtain the posterior information of the previous frame image of the current frame image in the time series, and obtain the optimal annotation result of the previous frame image;
[0035] The tracking label for the current frame image is calculated using the optimal annotation result of the previous frame image.
[0036] The ground truth label of the current frame image is obtained by verifying the initial label and the tracking label of the current frame image.
[0037] Specifically, the embodiments of the present invention include the following steps:
[0038] ① Training of multiple detection models.
[0039] A certain number of samples are labeled, and the selected detection model is trained to complete the preparation of the detection model.
[0040] When training multiple detection models, the selection of detection models and samples should differ to ensure that the prediction results do not converge due to similarity between the samples and the detection model. The specific process is as follows:
[0041] (1) For detection model 1, the training sample set is divided into k parts. For each part, the model is trained with the remaining training samples, and then the result of this part is predicted.
[0042] (2) Repeat the above steps until each detection model predicts the result;
[0043] (3) Repeat steps (1) and (2) for detection model 1, detection model 2 and detection model n to obtain n correct models.
[0044] The number of detection models is odd, which provides the number of samples available for voting in the following voting mechanism steps.
[0045] ② Image preprocessing.
[0046] To reduce image noise, the image will be normalized.
[0047] ③ Forward reasoning of the model.
[0048] Depend on Figure 3 As shown in the process, in this embodiment of the invention, multiple target detection algorithms of multiple trained detection models are used to infer the same frame image, the outputs of these n correct models are fused to obtain three-dimensional data, and this three-dimensional data is used as input to the meta-learner for output.
[0049] The meta-learner is a deep learning model that abstracts n voting mechanisms. After passing through the meta-learner, the results of inference from multiple detection models are output as the initial label of the current frame model through a majority voting method.
[0050] ④ Target tracking.
[0051] The Kalman filter and Hungarian matching algorithms are used to track the posterior information of the previous frame of the current frame. Time-series tracking is performed on the current frame, and the tracking results continuously utilize the best annotations from the previous five frames. The best annotations from the previous five frames all contribute to the output of the current frame; the closer the time to the current frame, the greater the contribution, and the longer the time, the smaller the contribution. The tracking label for the current frame is obtained using the best annotations from the previous five frames.
[0052] Tracking in time series takes advantage of the fact that the target does not change abruptly between frames in the same time series. The optimal annotation results of the current frame image and the previous frame image are checked. If they match, the tracking label of the current frame image is output as the result.
[0053] The verification method is to use time frames of the same target and determine whether they are predicted to be the same target based on the degree of overlap between the time frames.
[0054] In one implementation, such as Figure 7 As shown, assuming there are five frames in the time series from the fourth-to-last frame (t-4) to the current frame (t), the confidence scores (which represent the quality of the time frames) corresponding to the three time frames are as follows: Figure 7 The upper part is shown. Assuming the contribution factors of the five frames (the sum of the contribution factors of all frames is 1) are 0.05, 0.05, 0.1, 0.2, and 0.6 respectively, using the calculation formula (the confidence score of each frame multiplied by the contribution factor, and the results of the last five frames added together), the confidence scores of time frames 1, 2, and 3 of the current frame, calculated by weighting the previous frames in the time series, are: Figure 7 The lower half. Gray indicates a confidence level below the threshold of 0.5. That is, timeframes 1 and 2 will be retained, but timeframe 3 will not be retained.
[0055] The confidence level of time frame 1 in the current frame t is greater than the threshold of 0.5. After weighted calculation of the previous frames in the time series, the confidence level is still greater than 0.5. Therefore, time frame 1 is retained.
[0056] In the current frame t, the confidence score of time frame 2 is 0.48, slightly lower than the threshold of 0.5. Normally, time frame 2 would not be retained. However, after weighted calculation using previous frames in the time series, the confidence score is greater than 0.5, and time frame 2 is ultimately retained. In this case, the weighted calculation using previous frames in the time series compensates for the instability of single-frame prediction and reduces missed detections.
[0057] In the current frame t, the confidence score of time frame 3 is 0.6, slightly higher than the threshold. Under normal circumstances, time frame 3 would be retained. However, after weighted calculation of the previous frames in the time series, the confidence score is less than 0.5, and time frame 3 will not be retained. It should be noted that in practical applications, if the confidence score is very high, time frame 3 will still be retained, which can greatly suppress false detections.
[0058] Depend on Figure 4 As shown in the flowchart, in this embodiment of the invention, the single-frame target tracking process for a single model is as follows:
[0059] Take a frame from the video and feed it into a detection model;
[0060] The test results of this test model are obtained;
[0061] Predict the location of the tracking target in the current frame image based on the tracking trajectory;
[0062] The IOU (Intersection over Union) of the tracking and detection results is calculated as the cost matrix, and the Hungarian matching algorithm is used for matching.
[0063] Combining the detection and prediction results, the Kalman filter algorithm is used to update and obtain the optimal annotation for a single frame image;
[0064] Output the optimal annotation result.
[0065] ⑤ Target fusion test.
[0066] The initial label and the tracking label of the current frame image are subjected to target fusion verification. When the IOU (Intersection over Union) of the tracking label and the initial label of the current frame image is greater than a certain threshold, the target fusion is successful and the result is written to the file as a ground truth label. When the IOU (Intersection over Union) threshold is not met, the target fusion fails. The targets that fail to fuse are not written as ground truth labels, while other targets that successfully fuse are written as ground truth labels, thus completing the automatic data labeling.
[0067] In this embodiment, a computer device is provided, such as... Figure 5 As shown, it includes a memory 501, a processor 502, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements any of the above-mentioned sample annotation methods.
[0068] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.
[0069] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that performs any of the sample annotation methods described above.
[0070] Specifically, computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.
[0071] Based on the same inventive concept, this invention also provides a sample labeling device, as shown in the following embodiments. Since the principle by which the sample labeling device solves the problem is similar to that of the labeling method, the implementation of the sample labeling device can refer to the implementation of the sample labeling method, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0072] Figure 3 This is a structural block diagram of a sample labeling device according to an embodiment of the present invention, such as... Figure 3 As shown, it includes:
[0073] The detection model training module 01 inputs unlabeled sample images from the training sample set into each model included in the detection model for prediction processing, and obtains the prediction results output by each detection model.
[0074] Image preprocessing module 02 normalizes the image to make it a standard image and reduces image noise;
[0075] The single-model single-frame tracking module 03 outputs the results of one detection model for each frame to obtain the optimal annotation result for a single frame.
[0076] The time series tracking module 04 tracks the posterior information of the previous frame of the current frame in the time series and obtains the tracking label of the current frame image;
[0077] The initial label and tracking label fusion verification module 05 performs target fusion verification on the initial label and tracking label to generate a truth label.
[0078] In one embodiment, the detection model training module 01 includes:
[0079] The detection model selection unit selects several currently used detection models with better performance as the detection models used in this embodiment;
[0080] The detection model training unit extracts feature information from labeled sample images and predicts the training prediction results of labeled images based on the feature information of credible candidate regions.
[0081] In one embodiment, the image preprocessing module 02 includes:
[0082] The image digitization unit, under the condition of a certain storage capacity, needs to select appropriate M, N and b values according to the different properties of the image in order to obtain the best processing effect.
[0083] The image geometric transformation unit is used to perform transformations to correct systematic errors in the image acquisition system and random errors in the instrument position.
[0084] An image normalization unit is a standard form of an image that makes certain features of the image invariant under a given transformation.
[0085] Image smoothing unit, which eliminates random noise in the image;
[0086] The image restoration unit corrects image degradation caused by various reasons, so that the reconstructed or estimated image is as close as possible to the ideal, degradation-free image field;
[0087] The image enhancement unit selectively enhances or suppresses information in an image to improve its visual effect or transform it into a form more suitable for machine processing, so as to facilitate data extraction or recognition.
[0088] In one embodiment, the single-model single-frame tracking module 03 includes:
[0089] A single-model detection unit uses one detection model to predict a single frame.
[0090] The system calculates the optimal annotation result unit for a single frame, obtains the target position tracking unit, predicts the target position tracking in the current frame based on the tracking trajectory, calculates the IOU between the tracking and detection results as the cost matrix, performs matching using the Hungarian matching algorithm, combines the prediction results from the detection model, and updates the data using the Kalman filter algorithm to obtain the optimal annotation for a single frame image.
[0091] In one embodiment, the time series tracking module 04 includes:
[0092] Obtain the previous frame image unit in the time series, and select 5 previous frames in the same time series as the previous frame images;
[0093] The previous frame image influence factor setting unit sets the influence factor according to the distance of each frame from the current frame. The shorter the time from the current frame, the greater the contribution of the influence factor.
[0094] In another embodiment, software is also provided for executing the technical solutions described in the above embodiments and preferred embodiments.
[0095] In another embodiment, a storage medium is also provided, which stores the above-mentioned software. The storage medium includes, but is not limited to, optical discs, floppy disks, hard disks, and rewritable memory.
[0096] The embodiments of the present invention achieve the following technical effects:
[0097] Compared with manual annotation methods, the embodiments of the present invention have higher and more stable annotation quality, higher annotation efficiency, and the advantage of running 24 hours a day. Compared with semi-automatic annotation methods based on deep learning models, the present invention makes full use of historical information in time series and adopts a multi-model voting method, resulting in higher annotation accuracy, fewer mis-annotations, and less manual review workload.
[0098] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.
[0099] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A sample annotation method, characterized in that, include: For each frame of the sample to be labeled, multiple detection models are used to perform forward inference on the current frame image to obtain multiple prediction results for the current frame image; Multiple prediction results of the current frame image are fused to obtain the initial label of the current frame image; Obtaining the posterior information of the previous frame image of the current frame image in the time series, and obtaining the optimal annotation result of the previous frame image, includes: The preceding frame image of the current frame image is specified as the previous 5 frames of the current frame image that have been continuously recorded; The first 5 frames were processed by multiple detection models, outputting multiple optimal annotation results for each single frame, including: Take a single frame from the video as the current frame and feed it into a detection model; Obtain the detection results of this detection model; Predict the position of the tracking target in the current frame based on the tracking trajectory; The IOU of the tracking and detection results is calculated as the cost matrix, and the Hungarian matching algorithm is used for matching. Combining the detection and prediction results, the optimal annotation of the single frame image is obtained by updating the Kalman filter algorithm. Output the optimal annotation result for a single frame; The tracking label of the current frame image is calculated based on the optimal annotation result of the previous frame image; The ground truth label of the current frame image is obtained by verifying the initial label and the tracking label of the current frame image.
2. The sample labeling method as described in claim 1, characterized in that, The initial label of the current frame image is obtained by fusing multiple prediction results of the current frame image, including: The prediction results of the current frame image are used as input, and a voting mechanism is used to fuse the prediction results to obtain the initial label of the current frame image.
3. The sample labeling method as described in claim 1, characterized in that, The tracking label for the current frame image is calculated based on the optimal annotation results of the previous frame image, including: Each of the previous frame images is set with an influence factor, and the influence factor decreases the further the previous frame image is from the current frame image. The tracking label for the current frame image is calculated using the influence factor and the optimal annotation result of the previous frame image.
4. The sample labeling method as described in claim 1, characterized in that, Also includes: When the IOU threshold between the initial label of the current frame image and the tracking label of the current frame image is greater than a certain threshold, the target fusion is successful, and the initial label of the current frame image is used as the ground truth label. When the IOU threshold does not meet a certain threshold, the target fusion fails. The target that fails to fuse is not written as a ground truth label. The initial labels of other successfully fused images in the current frame are used as ground truth labels to complete the sample labeling.
5. The sample labeling method as described in claim 1, characterized in that, Also includes: Before fusing multiple prediction results of the current frame image, image preprocessing is performed on the current frame image; Each of the multiple detection models is trained through the following steps: Divide the training sample set into k parts; For each of the k datasets, a detection model is trained using the remaining datasets excluding that dataset, and the trained detection model is used to predict the prediction result for that dataset, until the prediction results for all k datasets are obtained.
6. A sample labeling device, characterized in that, include: The detection model training module inputs unlabeled sample images from the training sample set into each model included in the detection model for prediction processing, and obtains the prediction results output by each detection model. The image preprocessing module normalizes the image to make it a standard image and reduces image noise. The single-model single-frame tracking module outputs results for each of the detection models in each frame to obtain the optimal annotation result for a single frame. The single-model single-frame tracking module includes: Specify the preceding frame image of the current frame image as the 5 frames preceding the current frame image that have been continuously recorded; The first 5 frames were processed by multiple detection models, outputting multiple optimal annotation results for each single frame, including: Take a single frame from the video as the current frame and feed it into a detection model; Obtain the detection results of this detection model; Predict the position of the tracking target in the current frame based on the tracking trajectory; The IOU of the tracking and detection results is calculated as the cost matrix, and the Hungarian matching algorithm is used for matching. Combining the detection and prediction results, the optimal annotation of the single frame image is obtained by updating the Kalman filter algorithm. Output the optimal annotation result for a single frame; The time series tracking module tracks the posterior information of the previous frame of the current frame in the time series to obtain the tracking label of the current frame image; The initial label and tracking label fusion verification module performs target fusion verification on the initial label and the tracking label to generate a truth label.
7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the sample labeling method according to any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the sample labeling method according to any one of claims 1 to 5.