Image data labeling method fusing pseudo 3D artificial labeling and 2D target tracking

Abstract

The application relates to an image data labeling method fusing pseudo 3D artificial labeling and 2D target tracking, comprising the following steps: acquiring labeling data video; frame extraction is performed on the labeling data video, the extracted image frames are set as associated frames, and other image frames are set as derived frames; pseudo 3D artificial labeling is performed on a plurality of target objects in the associated frames, and artificial labeling boxes are obtained; a target detection model is used to sequentially identify target objects in all image frames of the labeling data video, and a plurality of target detection boxes and confidence degrees are obtained; based on the target detection boxes and the confidence degrees, a multi-target tracking algorithm is used to determine a tracking prediction box and a unique index of each target object; according to the offset of the centroid of the tracking prediction box relative to the centroid of the artificial labeling box in the associated frames, the associated artificial labeling box is determined, and pseudo 3D labeling boxes of the derived frames are derived. The application can effectively improve the pseudo 3D labeling efficiency and reduce the labeling cost.

Description

Technical Field

[0001] This invention relates to the field of image annotation technology, and in particular to an image data annotation method that integrates pseudo-3D manual annotation and 2D target tracking. Background Technology

[0002] In recent years, AI algorithms based on the fusion of radar and visual sensors have played a crucial role in the field of autonomous driving due to their superior environmental perception capabilities. The fusion of radar point cloud data and pseudo-3D image data is key to this success. However, training AI models requires massive amounts of labeled data, and manual labeling of pseudo-3D images is inefficient and costly. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method that can effectively improve the efficiency of pseudo-3D annotation and reduce the annotation cost.

[0004] The technical solution adopted by this invention to solve its technical problem is: to provide an image data annotation method that integrates pseudo-3D manual annotation and 2D target tracking, comprising the following steps:

[0005] Obtain labeled video data;

[0006] Frames are extracted from the labeled video data. The extracted image frames are designated as associated frames, and the other image frames are designated as derived frames.

[0007] Pseudo-3D manual annotation is performed on several target objects in the associated frames to obtain manual annotation boxes;

[0008] The target objects in all image frames of the labeled video data are identified sequentially using the target detection model, and multiple target detection boxes and their confidence scores are obtained.

[0009] Based on the target detection box and its confidence level, a multi-target tracking algorithm is used to determine the tracking prediction box and unique index for each target object.

[0010] The associated artificial annotation boxes are determined by the offset of the centroid of each tracking prediction box in the associated frame relative to the centroid of the artificial annotation box, and then the pseudo 3D annotation boxes of each derivation frame are derived.

[0011] Furthermore, determining the associated manually labeled box based on the offset of the centroid of the tracking prediction box relative to the centroid of the manually labeled box in the associated frame includes:

[0012] Construct an association deriver, which includes a first association frame, a second association frame, and all derivation frames between the two association frames;

[0013] Calculate the first centroid offset of each tracking prediction box in the first associated frame and the second associated frame respectively, and determine the artificial annotation box associated with the current tracking prediction box based on the magnitude of the first centroid offset. The first centroid offset is the pixel interval between the centroids of the tracking prediction box and the artificial annotation box in the same associated frame.

[0014] Furthermore, determining the manually labeled bounding box associated with the current tracking prediction box based on the magnitude of the first centroid offset includes:

[0015] Compare the first centroid offset with a set threshold. If the offset is less than the set threshold, the manually labeled box corresponding to the first centroid offset is considered to be associated with the current tracking prediction box.

[0016] Furthermore, the derivation of the pseudo-3D bounding boxes for each derivation frame includes:

[0017] Based on the unique index, manually labeled boxes associated with each tracking prediction box in the derivation frame are identified as associated labeled boxes.

[0018] The second centroid offset of each tracking prediction box in the derivation frame is calculated, and the pseudo 3D annotation box of the current tracking prediction box is derived using the second centroid offset. The second centroid offset is the pixel interval between the centroids of the tracking prediction box and the associated annotation box in the derivation frame.

[0019] Furthermore, the step of determining the manually labeled bounding boxes associated with each tracking prediction box in the derivation frame as associated labeled boxes based on a unique index includes:

[0020] Set a set number of derivation frames in the associated deriver that are close to the first associated frame as the first derivation frame, and set the other derivation frames as the second derivation frames.

[0021] The associated annotation box of each tracking prediction box in the first derivation frame is set as the manually labeled box associated with the tracking prediction box of the first associated frame with the same unique index.

[0022] The associated annotation box of each tracking prediction box in the second derivation frame is set as the manually labeled box associated with the tracking prediction box of the second associated frame with the same unique index.

[0023] Furthermore, the set quantity is the total number of derivation frames in the associated deriver divided by 2 and then rounded down.

[0024] Furthermore, it also includes the step of binding the derived pseudo-3D bounding boxes with the attribute data of the corresponding target objects in the radar point cloud.

[0025] Furthermore, the first associated frame and the second associated frame are sequentially adjacent.

[0026] Furthermore, the step of determining the tracking prediction box for each target object based on the target detection box and its confidence level using a multi-target tracking algorithm is achieved by using the ByteTrack target tracking algorithm to perform trajectory matching on the tracking prediction boxes in a set order.

[0027] Furthermore, the step of using the ByteTrack target tracking algorithm to perform trajectory matching on the tracking prediction box in a set order includes:

[0028] Based on the frequency of the corresponding target object appearing in the scene of the current video frame, the tracking prediction boxes in each video frame are sorted.

[0029] The ByteTrack target tracking algorithm is used to match and track the sorted sequence of tracking prediction boxes sequentially.

[0030] Furthermore, the target detection model is built based on the YOLO algorithm.

[0031] Beneficial effects

[0032] By adopting the above-mentioned technical solution, this invention has the following advantages and positive effects compared with the prior art: This invention reduces the amount of manual annotation by extracting frames from massive data, and tracks all detected targets in each frame based on the YOLOv5 multi-target detection algorithm and the ByteTrack multi-target tracking algorithm. The tracking results are associated with manually annotated pseudo-3D boxes, and the pseudo-3D boxes are derived. In addition, attribute data from other radar point clouds are bound, which effectively improves the efficiency of image annotation and reduces the annotation cost. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the process modules of an embodiment of the present invention;

[0034] Figure 2 This is a flowchart of an embodiment of the present invention;

[0035] Figure 3 This is a flowchart of the multi-target tracking algorithm according to an embodiment of the present invention. Detailed Implementation

[0036] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0037] The embodiments of the present invention relate to an image data annotation method that integrates pseudo-3D manual annotation and 2D target tracking, such as... Figure 1As shown, it includes the following process modules:

[0038] (1) 3D data processing module. The main function is to first extract frames from massive labeled data to reduce the amount of manual annotation. Each image corresponds to a labeling file, including the pseudo 3D information box of the target to be labeled in the current image, as well as other attribute data of the target in the radar point cloud. Secondly, the data video before frame extraction is serialized, and the manually labeled image frames are defined as associated frames. The associated frames are bound to the content of the manually labeled file, and other frames are derivation frames.

[0039] (2) 2D target detection and tracking module, whose main function is to track all detected targets in each frame based on YOLOv5 multi-target detection algorithm and ByteTrack multi-target tracking algorithm, and each tracked target has a unique index.

[0040] (3) The association derivation module is mainly used to calculate the centroid of the manually labeled pseudo 3D box and the centroid of the 2D target tracking box, determine whether they are associated based on the centroid pixel distance, calculate the centroid offset between the derivation frame and the associated frame, thereby deriving the pseudo 3D box and binding the target's other attribute data in the radar point cloud.

[0041] (4) Output annotation file module, whose main function is to automatically generate new annotation files from the calculation results of the derivation frame.

[0042] The following description, in conjunction with the accompanying drawings, further illustrates this embodiment. Figure 2 As shown, the specific steps include:

[0043] Step 1: Serialize the video data before frame extraction and bind manually labeled files when relevant frames are found.

[0044] Step 2: Initialize the target detector, load the YOLOv5 multi-target detection model, and return the detection boxes of all targets in the image.

[0045] Step 3: Obtain the target detection box through the detector, convert the detection box into the format required by the ByteTrack target tracker, and divide the detection box into high-confidence detection boxes and low-confidence detection boxes according to the confidence of the target detection box.

[0046] Step 4: An improved ByteTrack target tracking algorithm is used to create a tracking trajectory for each target. This algorithm uses Kalman filtering to predict the next bounding box for each trajectory. Since the original ByteTrack algorithm does not return target category information, the motion trajectories of targets of the same category are correlated. Each Kalman filter prediction traverses the trajectory in the order of discovery, lacking category correlation, which slows down the algorithm. Therefore, a target category attribute is added to each trajectory, and the detection boxes are pre-sorted according to the frequency of the target object's appearance in the corresponding scene. ByteTrack is then used to track the trajectory according to the category order, improving the algorithm speed. The specific process is as follows: Figure 3 As shown.

[0047] Step 5: For high-confidence detection boxes, calculate their Intersection over Union (IoU) with the predicted boxes, and use the Hungarian algorithm to match the IoU, obtaining three results: matched trajectory, unmatched trajectory, and unmatched high-confidence detection boxes. After matching the trajectory, update the boxes in the trajectory with the detection boxes.

[0048] Step 6: For low-confidence detection boxes, calculate their Intersection over Union (IoU) with the unmatched trajectory prediction boxes from the previous step. Use the Hungarian algorithm to match the IoU, obtaining three results: matched trajectory, unmatched trajectory, and unmatched low-confidence detection boxes. After matching the trajectory, update the boxes in the trajectory to detection boxes.

[0049] Step 7: For unmatched high-scoring detection boxes, match them with inactive trajectories (new trajectories appearing in trajectory prediction), obtaining three results: matched, unmatched trajectory, and unmatched detection box. For matched boxes, update the status; for unmatched trajectories, mark them for deletion; for unmatched detection boxes, create a new tracking trajectory if the confidence score is greater than a threshold, otherwise delete it.

[0050] Step 8: Return the tracking prediction box for each target and the target's unique index.

[0051] Step 9: Construct two associated frames and all intermediate derived frames into an associated deducer. If the frame extraction interval is n, then the length of an associated deducer is n+2. Each element includes the current frame, the pseudo-3D manually annotated information of the current frame, the radar point cloud information, and the target tracking information of the current frame. The pseudo-3D information of the current frame refers to 16 pseudo-3D coordinate values, each coordinate being a normalized value of the 2D image pixel coordinates. The radar point cloud information includes timestamp information, lane information, direction information, etc.

[0052] Step 10: Once an association inferrer is built, calculate the centroid of the tracking prediction box and the centroid of the manually labeled box for each target in the association frame, calculate their pixel interval, and if it is less than the threshold, then they are associated, meaning they are the same target.

[0053] Step 11: Derive pseudo-3D coordinate values ​​based on the offset of the tracking box centroid in the associated frame and the derived frame. If the frame extraction interval is 10, the first associated frame of the associated deriver derives the next 5 frames, and the last associated frame of the associated deriver derives the first 5 frames.

[0054] Step 12: When an association inferencer is completed, output the inference annotation file. Each target is a line of data, including pseudo 3D coordinates and other attribute data in the radar point cloud, such as timestamp information, lane information, direction information, etc.

[0055] Step 13: Pop up the last associated frame information as the first frame information of the next associated deducer, continue to add new deduced frames, and repeat this process to update the associated deducer.

Claims

1. An image data annotation method that integrates pseudo-3D manual annotation and 2D target tracking, characterized in that, Includes the following steps: Obtain labeled video data; Frames are extracted from the labeled video data. The extracted image frames are designated as associated frames, and the other image frames are designated as derived frames. Pseudo-3D manual annotation is performed on several target objects in the associated frames to obtain manual annotation boxes; The target objects in all image frames of the labeled video data are identified sequentially using the target detection model, and multiple target detection boxes and their confidence scores are obtained. Based on the target detection box and its confidence level, a multi-target tracking algorithm is used to determine the tracking prediction box and unique index for each target object. Based on the offset of the centroid of the tracking prediction box in the associated frame relative to the centroid of the manual annotation box, the associated manual annotation box is determined, and then the pseudo 3D annotation box of each derivation frame is derived. The step of determining the associated manual annotation box based on the offset of the centroid of the tracking prediction box in the associated frame relative to the centroid of the manual annotation box includes: Construct an association deriver, which includes a first association frame, a second association frame, and all derivation frames between the two association frames; Calculate the first centroid offset of each tracking prediction box in the first associated frame and the second associated frame respectively, and determine the artificial annotation box associated with the current tracking prediction box based on the magnitude of the first centroid offset. The first centroid offset is the pixel interval between the centroids of the tracking prediction box and the artificial annotation box in the same associated frame. The derivation of the pseudo-3D bounding boxes for each derivation frame includes: Based on the unique index, manually labeled boxes associated with each tracking prediction box in the derivation frame are identified as associated labeled boxes. The second centroid offset of each tracking prediction box in the derivation frame is calculated, and the pseudo 3D annotation box of the current tracking prediction box is derived using the second centroid offset. The second centroid offset is the pixel interval between the centroids of the tracking prediction box and the associated annotation box in the derivation frame.

2. The method according to claim 1, characterized in that, The step of determining the manually labeled bounding box associated with the current tracking prediction box based on the magnitude of the first centroid offset includes: Compare the first centroid offset with a set threshold. If the offset is less than the set threshold, the manually labeled box corresponding to the first centroid offset is considered to be associated with the current tracking prediction box.

3. The method according to claim 1, characterized in that, The step of determining the manually labeled bounding boxes associated with each tracking prediction box in the derivation frame based on a unique index includes: Set a set number of derivation frames that are close to the first associated frame in the associated deriver as the first derivation frame, and set the other derivation frames as the second derivation frames. The associated annotation box of each tracking prediction box in the first derivation frame is set as the manually labeled box associated with the tracking prediction box of the first associated frame with the same unique index. The associated annotation box of each tracking prediction box in the second derivation frame is set as the manually labeled box associated with the tracking prediction box of the second associated frame with the same unique index.

4. The method according to claim 3, characterized in that, The set quantity is the total number of derivation frames in the associated deriver divided by 2 and then rounded down.

5. The method according to claim 1, characterized in that, It also includes the step of binding the derived pseudo-3D bounding boxes with the attribute data of the corresponding target objects in the radar point cloud.

6. The method according to claim 1, characterized in that, The first associated frame and the second associated frame are sequentially adjacent.

7. The method according to claim 1, characterized in that, The step of determining the tracking prediction box for each target object based on the target detection box and its confidence level using a multi-target tracking algorithm is achieved by using the ByteTrack target tracking algorithm to perform trajectory matching on the tracking prediction boxes in a set order.

8. The method according to claim 7, characterized in that, The step of using the ByteTrack target tracking algorithm to perform trajectory matching on the tracking prediction box in a set order includes: Based on the frequency of the corresponding target object appearing in the scene of the current video frame, the tracking prediction boxes in each video frame are sorted. The ByteTrack target tracking algorithm is used to match and track the sorted sequence of tracking prediction boxes sequentially.

Images