Four-dimensional automatic labeling method and system based on image and point cloud

Abstract

The invention relates to a four-dimensional automatic labeling method and system based on an image and point cloud, electronic equipment and a storage medium. The method comprises the following steps: taking a time sequence point cloud and a time sequence image as input, detecting the time sequence point cloud based on a point cloud detection model and a visual detection model to generate a point cloud three-dimensional detection frame and an image three-dimensional detection frame, and performing cross validation in a point cloud coordinate system to generate a target detection frame; performing association processing on the target detection frames greater than a preset confidence coefficient to generate a forward trajectory, reversely executing a tracking algorithm to generate a reverse trajectory, fusing the reverse trajectory, and extracting point cloud data of each frame after trajectory fusion; and respectively carrying out optimization processing on the fused track and each frame of three-dimensional detection frame in the fused track, and then carrying out consistency coordinate transformation to generate a four-dimensional labeling result. The accuracy of four-dimensional automatic labeling is improved, the high cost caused by manual labeling is reduced, and high-quality true value data is provided for the three-dimensional perception model of automatic driving.

Description

Background Art

[0002] With the rapid development of autonomous driving, the accuracy requirements for 3D target detection are getting higher and higher, the models are becoming more and more complex, and the amount of data required for model training is also increasing. However, the labor cost of 3D annotation on the market is currently high. How to achieve low-cost automatic annotation of massive 3D data is a problem that many autonomous driving companies must solve. At present, the mainstream 3D automatic annotation methods are mainly based on four-dimensional annotation on time-series 3D data, that is, adding one-dimensional time series information to 3D data, mainly including automatic annotation methods based on real-time detection and automatic annotation methods based on offline time series data. Among them, the automatic annotation method based on real-time detection uses an online 3D target detection model to detect the input data frame by frame, and then uses the overlap degree and position coordinates of the detection frames between consecutive frames to associate and track the data of the instance object, realizing four-dimensional annotation on the time series. This method uses an online 3D detection model as the input for tracking. Since the online detection model does not use the point cloud features of the future time series, it is usually easy to cause target loss and incorrect association of tracking, and cannot guarantee the generation of a continuous and complete object tracking sequence.

[0003] Automatic labeling methods based on offline time series data are generally divided into three modules: detection, tracking, and optimization. First, an offline detection model is used to perform target detection on historical time series images or point cloud data under multi-frame combinations. Then the tracking module converts the detected target into each frame, performs data association, and forms an object sequence. Then, based on the object trajectory characteristics, the object's motion state (stationary or moving) is determined. Finally, the time series point cloud features of stationary and dynamic objects are extracted respectively, and the predicted box is optimized in position and geometry to obtain accurate three-dimensional labeling.

[0004] In the prior art, CN202311121001.7-A method for automatically annotating millimeter-wave radar point clouds based on video recognition: This invention proposes a method for automatically annotating millimeter-wave radar point clouds based on video recognition. The specific implementation scheme is: collect radar point clouds and corresponding video data, synchronize the start time of radar and video; select two frames of radar data, and use the time-series DBSCAN clustering algorithm based on the elliptical neighborhood improvement to segment radar targets; input the video frames synchronized with the radar data frames into the YOLOv7 network to obtain the semantic labels of the video targets, the target area of ​​interest, and the target center pixel coordinates; the pixel coordinates of the radar target density core points and the pixel coordinates of the video target center are associated with the target area of ​​interest, and the target association and radar point cloud annotation are completed. The invention fuses image and radar point cloud information in the initial detection stage with good accuracy and robustness. However, this method lacks tracking of timing information, and the annotation efficiency is not high. Secondly, the annotation noise is not optimized at the back end, and the overall annotation accuracy is poor.

[0005] CN202410281528.4-A method, terminal and storage medium for automatic annotation of point cloud data: A method, terminal and storage medium for automatic annotation of point cloud data are disclosed. The main steps of the method include: obtaining the initial target detection results obtained by performing target detection on each point cloud frame in the continuous point cloud frames by at least two different target detection models; determining the target trajectory sequence corresponding to each point cloud target in the continuous point cloud frames based on the initial target detection results corresponding to each point cloud frame; determining the motion state category corresponding to the point cloud target based on the detection frame position in the target trajectory sequence; based on the motion state category, averaging and adjusting the initial detection frame position and initial detection frame size of the point cloud target in each point cloud frame to obtain the adjusted target detection result; and automatically annotating each point cloud frame using the adjusted target detection result. The above method can effectively improve the accuracy of automatic annotation. However, this method mainly uses radar point cloud for target detection. The number of point clouds of the target object decreases with the increase of detection distance, and it is easy to lose the detection of distant targets. Secondly, this method optimizes static and dynamic targets separately, which is easy to cause missing information features. In addition, when optimizing the prediction box size and position, only the prediction is averaged, which will bring random noise.

[0006] Therefore, one or more methods are needed to solve the above problems.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0008] The purpose of the present disclosure is to provide a four-dimensional automatic labeling method, system, electronic device and computer-readable storage medium based on images and point clouds, thereby overcoming one or more problems caused by the limitations and defects of related technologies at least to a certain extent.

[0009] According to one aspect of the present disclosure, a four-dimensional automatic annotation method based on an image and a point cloud is provided, comprising:

[0010] The offline 3D detection step takes the time-series point cloud and the time-series image as input, detects the time-series point cloud based on the 3D point cloud detection model to generate a point cloud 3D detection frame, detects the time-series image based on the 3D vision detection model to generate an image 3D detection frame, cross-validates the point cloud 3D detection frame and the image 3D detection frame in the point cloud coordinate system, and generates a target detection frame;

[0011] In the target tracking step, the target detection frame greater than the preset confidence is associated with each other to generate a forward trajectory, the tracking algorithm is executed in reverse to generate a reverse trajectory, the forward trajectory and the reverse trajectory are fused based on the weighted box fusion (WBF) strategy, and the point cloud data of each frame after the trajectory fusion is extracted;

[0012] The optimization step performs geometric size, position trajectory, and confidence optimization processing on the fused trajectory and each frame of the 3D detection frame in the fused trajectory, and then performs consistent coordinate transformation to generate a 4D annotation result.

[0013] In an exemplary embodiment of the present disclosure, the method of detecting the time-series point cloud based on the three-dimensional point cloud detection model to generate a point cloud three-dimensional detection frame further includes:

[0014] The 3D point cloud detection model is based on the dynamic sparse voxel transformer DSVT algorithm, uses a combination of five consecutive frames of point cloud data as input, and generates a point cloud 3D detection frame based on test enhancement TTA and multi-model integration.

[0015] In an exemplary embodiment of the present disclosure, the method of detecting the time-series images based on the three-dimensional visual detection model to generate the three-dimensional detection frame of the image further includes:

[0016] The three-dimensional visual detection model is based on multi-view Figure 2 The D target guided 3D target detector MV2D algorithm is used to detect and generate a 3D detection frame for the image.

[0017] In an exemplary embodiment of the present disclosure, the target tracking step of the method further includes:

[0018] The target detection frames are divided into high confidence groups and low confidence groups based on the preset confidence, and the target detection frames in the high confidence group are associated to generate forward trajectories;

[0019] After the preset tracking algorithm is executed in reverse to generate a reverse trajectory, the forward trajectory and the backward trajectory are associated with each other through position similarity.

[0020] In an exemplary embodiment of the present disclosure, optimizing the geometric dimensions of the object in the optimization step of the method further comprises:

[0021] The same tracking target in different frames is converted to the frame center coordinate system, and the point clouds in the trajectory of the tracking target are merged, and a group of point clouds are randomly selected from the merged point clouds;

[0022] Calculate the distance between each point cloud in the randomly selected set of point clouds and the six surfaces of the detection frame;

[0023] Select t-frame samples from the complete target trajectory, extract features of the t-frame samples based on the pointnet algorithm, and generate an initial geometric query Q_geo;

[0024] Extracting the randomly selected set of point clouds based on the pointnet algorithm to generate dense features corresponding to the geometric key Key_geo and the geometric value V_geo respectively;

[0025] The geometric query Q_geo is first sent to the multi-head self-attention mechanism for encoding to obtain the contextual information between samples. Then the geometric query Q_geo, geometric key Key_geo, and geometric value V_geo undergo a cross-attention mechanism to supplement the features of the perspective and then predict the geometric size.

[0026] In an exemplary embodiment of the present disclosure, optimizing the position trajectory of the object in the optimization step of the method further includes:

[0027] A bounding box position is randomly selected from the bounding box sequence of each target object tracked as the local coordinate system, and then other boxes in the trajectory of the target object are converted to the coordinate system. A fixed point is selected in each frame of the target object, and the distance from the fixed point to the center of the box and the eight corner points is calculated as the global position feature, that is, the position key K_pos and the position value V_pos;

[0028] Based on the pointnet algorithm, features are extracted from all points in the target trajectory, and corresponding confidence scores are added to enrich the feature dimension as the position query Q_pos;

[0029] The position query Q_pos is first sent to the self-attention mechanism to capture the distance between the position query Q_pos and other boxes, and then the position query Q_pos, the position key K_pos, and the position value V_pos are sent to the cross-attention mechanism to encode the local to global position, predict the offset and angle offset of the center point of each box in the local coordinate system, and adjust the position trajectory of the target object's movement.

[0030] In an exemplary embodiment of the present disclosure, optimizing the confidence of the object in the optimization step of the method further includes:

[0031] Based on the classification branch, detection is performed by updating the score. If the IOU between the prediction and the true value is lower than the preset threshold, it is a negative sample. If it is higher than the preset threshold, it is a positive sample.

[0032] Based on the prediction branch: detect the target object and optimize the geometry and position trajectory of the detection frame and the ground truth frame.

[0033] The results of the classification branch and the prediction branch are fused, and the square root of the results of the classification branch and the prediction branch is calculated to generate an optimized confidence.

[0034] In one aspect of the present disclosure, a four-dimensional automatic annotation system based on images and point clouds is provided, comprising:

[0035] An offline 3D detection module is used to take a time-series point cloud and a time-series image as input, detect the time-series point cloud based on a 3D point cloud detection model to generate a point cloud 3D detection frame, detect the time-series image based on a 3D visual detection model to generate an image 3D detection frame, cross-validate the point cloud 3D detection frame and the image 3D detection frame in a point cloud coordinate system, and generate a target detection frame;

[0036] The target tracking module is used to associate the target detection frame greater than the preset confidence level to generate a forward trajectory, reversely execute the preset tracking algorithm to generate a reverse trajectory, fuse the forward trajectory and the reverse trajectory based on the weighted box fusion WBF strategy, and extract the point cloud data of each frame in the fused trajectory;

[0037] The optimization module is used to optimize the geometric size, position trajectory, and confidence of the fused trajectory and the three-dimensional detection frame of each frame in the fused trajectory, and then perform consistent coordinate transformation to generate a four-dimensional labeling result.

[0038] In one aspect of the present disclosure, there is provided an electronic device, comprising:

[0039] Processor; and

[0040] A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of the above items.

[0041] In one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to any one of the above items is implemented.

[0042] In an exemplary embodiment of the present disclosure, a four-dimensional automatic annotation method based on images and point clouds is disclosed, wherein the method includes: taking a time-series point cloud and a time-series image as input, detecting the time-series point cloud based on a three-dimensional point cloud detection model to generate a point cloud three-dimensional detection frame, detecting the time-series image based on a three-dimensional visual detection model to generate an image three-dimensional detection frame, cross-validating the point cloud three-dimensional detection frame and the image three-dimensional detection frame in the point cloud coordinate system to generate a target detection frame; associating the target detection frame greater than a preset confidence to generate a forward trajectory, reversely executing the tracking algorithm to generate a reverse trajectory, fusing the forward trajectory and the reverse trajectory based on the weighted frame fusion WBF strategy, and extracting each frame of point cloud data after the trajectory fusion; respectively performing geometric size, position trajectory, and confidence optimization processing on the fused trajectory and each frame of the fused trajectory three-dimensional detection frame, and then performing a consistent coordinate transformation to generate a four-dimensional annotation result. The present disclosure improves the accuracy of four-dimensional automatic annotation, reduces the high cost of manual annotation, and provides high-quality true value data for the three-dimensional perception model of autonomous driving.

[0043] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other features and advantages of the present disclosure will become more apparent by describing in detail example embodiments thereof with reference to the attached drawings.

[0045] Figure 1 A flowchart of a four-dimensional automatic annotation method based on an image and a point cloud according to an exemplary embodiment of the present disclosure is shown;

[0046] Figure 2 The overall block diagram of a technical solution of a four-dimensional automatic annotation method based on an image and a point cloud according to an exemplary embodiment of the present disclosure is shown;

[0047] Figure 3 A geometric optimization effect diagram of a target object (person, vehicle) according to a four-dimensional automatic annotation method based on an image and point cloud according to an exemplary embodiment of the present disclosure is shown;

[0048] Figure 4 A target object (person, vehicle) trajectory optimization effect diagram of a four-dimensional automatic labeling method based on an image and point cloud according to an exemplary embodiment of the present disclosure is shown;

[0049] Figure 5 A schematic block diagram of a four-dimensional automatic annotation system based on images and point clouds according to an exemplary embodiment of the present disclosure is shown;

[0050] Figure 6 A block diagram schematically shows an electronic device according to an exemplary embodiment of the present disclosure;

[0051] Figure 7 A schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0052] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0053] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, materials, systems, steps, etc. may be adopted. In other cases, known structures, methods, systems, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0054] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of these functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor systems and / or microcontroller systems.

[0055] In this example embodiment, a four-dimensional automatic annotation method based on image and point cloud is first provided; Figure 1 As shown in , the four-dimensional automatic annotation method based on image and point cloud may include the following steps:

[0056] The offline 3D detection step S110 takes the time-series point cloud and the time-series image as input, detects the time-series point cloud based on the 3D point cloud detection model to generate a point cloud 3D detection frame, detects the time-series image based on the 3D vision detection model to generate an image 3D detection frame, cross-validates the point cloud 3D detection frame and the image 3D detection frame in the point cloud coordinate system, and generates a target detection frame;

[0057] In the target tracking step S120, the target detection frame with a confidence level greater than a preset value is associated to generate a forward trajectory, the tracking algorithm is reversely executed to generate a reverse trajectory, the forward trajectory and the reverse trajectory are fused based on the weighted box fusion (WBF) strategy, and the point cloud data of each frame after the trajectory fusion is extracted;

[0058] In the optimization step S130, the geometric size, position trajectory, and confidence level of the fused trajectory and each frame of the 3D detection frame in the fused trajectory are optimized respectively, and then a consistent coordinate transformation is performed to generate a 4D labeling result.

[0059] In an exemplary embodiment of the present disclosure, a four-dimensional automatic annotation method based on images and point clouds is disclosed, wherein the method includes: taking a time-series point cloud and a time-series image as input, detecting the time-series point cloud based on a three-dimensional point cloud detection model to generate a point cloud three-dimensional detection frame, detecting the time-series image based on a three-dimensional visual detection model to generate an image three-dimensional detection frame, cross-validating the point cloud three-dimensional detection frame and the image three-dimensional detection frame in the point cloud coordinate system to generate a target detection frame; associating the target detection frame greater than a preset confidence to generate a forward trajectory, reversely executing the tracking algorithm to generate a reverse trajectory, fusing the forward trajectory and the reverse trajectory based on the weighted frame fusion WBF strategy, and extracting each frame of point cloud data after the trajectory fusion; respectively performing geometric size, position trajectory, and confidence optimization processing on the fused trajectory and each frame of the fused trajectory three-dimensional detection frame, and then performing a consistent coordinate transformation to generate a four-dimensional annotation result. The present disclosure improves the accuracy of four-dimensional automatic annotation, reduces the high cost of manual annotation, and provides high-quality true value data for the three-dimensional perception model of autonomous driving.

[0060] Next, a four-dimensional automatic labeling method based on images and point clouds in this example embodiment will be further described.

[0061] Embodiment 1:

[0062] In the offline three-dimensional detection step S110, the time-series point cloud and the time-series image are taken as input, the time-series point cloud is detected based on the three-dimensional point cloud detection model to generate a point cloud three-dimensional detection frame, the time-series image is detected based on the three-dimensional visual detection model to generate an image three-dimensional detection frame, the point cloud three-dimensional detection frame and the image three-dimensional detection frame are cross-validated in the point cloud coordinate system to generate a target detection frame.

[0063] In the embodiment of this example, the method further includes: detecting the time-series point cloud based on the three-dimensional point cloud detection model to generate a point cloud three-dimensional detection frame:

[0064] The 3D point cloud detection model is based on the dynamic sparse voxel transformer DSVT algorithm, uses a combination of five consecutive frames of point cloud data as input, and generates a point cloud 3D detection frame based on test enhancement TTA and multi-model integration.

[0065] In the embodiment of this example, the method of detecting the time series image based on the three-dimensional visual detection model to generate the image three-dimensional detection frame also includes:

[0066] The three-dimensional visual detection model is based on multi-view Figure 2 The D target guided 3D target detector MV2D algorithm is used to detect and generate a 3D detection frame for the image.

[0067] In the target tracking step S120, the target detection box with a confidence level greater than a preset value is associated to generate a forward trajectory, the tracking algorithm is reversely executed to generate a reverse trajectory, the forward trajectory and the reverse trajectory are fused based on the weighted box fusion (WBF) strategy, and the point cloud data of each frame after the trajectory fusion is extracted.

[0068] In the embodiment of this example, the target tracking step of the method further includes:

[0069] The target detection frames are divided into high confidence groups and low confidence groups based on the preset confidence, and the target detection frames in the high confidence group are associated to generate forward trajectories;

[0070] After the preset tracking algorithm is executed in reverse to generate a reverse trajectory, the forward trajectory and the backward trajectory are associated with each other through position similarity.

[0071] In the optimization step S130, the geometric size, position trajectory, and confidence optimization processing are performed on the fused trajectory and each frame three-dimensional detection frame in the fused trajectory, and then a consistent coordinate transformation is performed to generate a four-dimensional labeling result.

[0072] In the embodiment of this example, optimizing the geometric dimensions of the object in the optimization step of the method further includes:

[0073] The same tracking target in different frames is converted to the frame center coordinate system, and the point clouds in the trajectory of the tracking target are merged, and a group of point clouds are randomly selected from the merged point clouds;

[0074] Calculate the distance between each point cloud in the randomly selected set of point clouds and the six surfaces of the detection frame;

[0075] Select t-frame samples from the complete target trajectory, extract features of the t-frame samples based on the pointnet algorithm, and generate an initial geometric query Q_geo;

[0076] Extracting the randomly selected set of point clouds based on the pointnet algorithm to generate dense features corresponding to the geometric key Key_geo and the geometric value V_geo respectively;

[0077] The geometric query Q_geo is first sent to the multi-head self-attention mechanism for encoding to obtain the contextual information between samples. Then the geometric query Q_geo, geometric key Key_geo, and geometric value V_geo undergo a cross-attention mechanism to supplement the features of the perspective and then predict the geometric size.

[0078] In the embodiment of this example, optimizing the position trajectory of the object in the optimization step of the method further includes:

[0079] A bounding box position is randomly selected from the bounding box sequence of each target object tracked as the local coordinate system, and then other boxes in the trajectory of the target object are converted to the coordinate system. A fixed point is selected in each frame of the target object, and the distance from the fixed point to the center of the box and the eight corner points is calculated as the global position feature, that is, the position key K_pos and the position value V_pos;

[0080] Based on the pointnet algorithm, features are extracted from all points in the target trajectory, and corresponding confidence scores are added to enrich the feature dimension as the position query Q_pos;

[0081] The position query Q_pos is first sent to the self-attention mechanism to capture the distance between the position query Q_pos and other boxes, and then the position query Q_pos, the position key K_pos, and the position value V_pos are sent to the cross-attention mechanism to encode the local to global position, predict the offset and angle offset of the center point of each box in the local coordinate system, and adjust the position trajectory of the target object's movement.

[0082] In the embodiment of this example, the confidence of the optimized object in the optimization step of the method further includes:

[0083] Based on the classification branch, detection is performed by updating the score. If the IOU between the prediction and the true value is lower than the preset threshold, it is a negative sample. If it is higher than the preset threshold, it is a positive sample.

[0084] Based on the prediction branch: detect the target object and optimize the geometry and position trajectory of the detection frame and the ground truth frame.

[0085] The results of the classification branch and the prediction branch are fused, and the square root of the results of the classification branch and the prediction branch is calculated to generate an optimized confidence.

[0086] Embodiment 2:

[0087] In the embodiment of this example, the present invention uses time-series point cloud and image data as input, and includes detection, tracking, and optimization modules. Compared with the disclosed four-dimensional automatic labeling invention, the detection module integrates three-dimensional visual detection information and cross-validates the results of the point cloud detection model, making up for the shortcomings of point cloud sparseness at long distances and difficult detection in radar blind spots; the tracking module performs forward and reverse tracking on the detection target in turn to ensure sequence integrity; the optimization module combines dynamic and static targets to optimize the geometry and position of the predicted bounding box, and the optimization effect is more in line with the motion properties of objects in the autonomous driving scene; in addition, the optimized point cloud annotation can also be fed back to the image, adding a four-dimensional annotation function for the time-series image. The present invention further improves the accuracy of four-dimensional automatic labeling, which not only reduces the high cost of manual labeling, but also provides high-quality true value data for three-dimensional perception models of autonomous driving, such as BEV models, Occupancy network models, etc.

[0088] In the embodiment of this example, the present invention realizes the four-dimensional automatic labeling function based on the offline three-dimensional detection module, the target tracking module, and the optimization module. The overall framework of the technical solution is as follows: Figure 2 shown.

[0089] The offline 3D detection module mainly includes a 3D point cloud detection model and a 3D visual detection model. The purpose is to generate enough detection candidate boxes and improve the detection recall rate. The point cloud 3D detection uses the DSVT algorithm and makes two improvements: 1. Use a combination of five consecutive frames of point clouds as input to improve prediction performance. 2. In order to improve the ability to adapt to complex environments, test augmentation (TTA) and multi-model integration (different resolutions, network structures and capacities) are used. The 3D visual detection model uses the MV2D algorithm, which can upgrade any two-dimensional target detector to a multi-view Figure 3 3D target detection, and use 2D detectors to generate target queries based on rich image semantics. These dynamically generated queries can help MV2D recall targets in the field of view and have a strong ability to locate 3D targets. Input point cloud and image data into the 3D detection model of the corresponding modality to obtain the 3D detection frames of the point cloud and image, and project the image 3D detection frame into the point cloud coordinate system. Cross-validate the visual 3D detection frame and the point cloud 3D detection frame, that is, the position deviation of the 3D center of the point cloud frame and the image frame and the IOU value of the frame are weighted summed. If the sum is within the threshold, the point cloud detection frame is taken. Otherwise, the image detection frame is taken considering the case of object occlusion and truncation or long-distance sparse point cloud. The offline 3D detection module aims to improve the recall rate of the front-end detection and provide a complete target sequence for the tracking module to the greatest extent.

[0090] The target tracking module adopts a two-stage association strategy to reduce false matches. First, the target detection frames are divided into high-confidence groups and low-confidence groups according to the confidence level. The high-confidence groups are associated with updated trajectories, and the unupdated trajectories are associated with the low-confidence groups. At the same time, the length of the object trajectory can last until the sequence ends to avoid tracking switching or truncation. Then the tracking algorithm is executed in reverse to generate another set of trajectories, which are associated by position similarity. Finally, the WBF strategy is used to fuse all successfully matched target trajectories to further improve the integrity of the target sequence. For the well-distinguished object sequence, the corresponding point cloud of each frame is extracted and saved; the redundant frames that have not been updated and some shorter sequences are directly merged into the final output without the need for subsequent optimization.

[0091] The optimization module decomposes the traditional bounding box regression task into three independent sub-modules, optimizing the geometric size, position trajectory and confidence of the object respectively:

[0092] The geometric optimization module is based on the transformer network, and the specific steps are as follows: 1. Convert the same tracked target in different frames to the coordinate system of the frame center, merge the point cloud in this trajectory, and randomly select a group of n points from it. 2. For each point in n, calculate its distance to the six faces of the detection frame, and add the confidence of the detection frame to enrich the feature dimension of the point. 3. Good Q initialization is conducive to transformer convergence, so first select t frames from the complete target trajectory, and then use the pointnet algorithm to extract features for the selected t samples to generate the initial geometric query Q_geo. 4. Pointnet is also used to extract features for the n points generated in step 2 to obtain Key_geo and V_geo. 5. Q_geo is first sent to the multi-head self-attention mechanism for encoding to obtain rich contextual information between samples, and then Q_geo, Key_geo, and V_geo are cross-attention mechanisms to supplement the features of the required perspectives, and then predict the accurate geometric dimensions. The geometric optimization effects on people and cars are as follows: Figure 3 shown.

[0093] The position trajectory optimization module is also based on the transformer network, which is mainly divided into three steps: 1. For each target object, randomly select the position of a bounding box from the bounding box sequence it tracks as a new local coordinate system, and then convert other boxes in the trajectory to this coordinate system. Fixed points are selected in each frame, and the distance of each point to the center of the box and the eight corner points is calculated as the global position features, namely K_pos and V_pos. 2. Use the pointnet algorithm to extract features from all points in the target trajectory, and add the corresponding confidence scores to enrich the feature dimensions as Q_pos. 3. Q_pos is first sent to the self-attention mechanism to capture the distance between itself and other boxes, and then Q_pos, K_pos, and V_pos are sent to the cross-attention mechanism to encode the local to global position, and finally predict the offset and angle offset of the center point of each box in the local coordinate system, and then adjust the position trajectory of the entire target movement. The trajectory optimization effect for people and cars is as follows: Figure 4 shown.

[0094] Confidence optimization module: Because the upstream detection module and tracking module will generate a lot of detection boxes, many of which may be false detections or low IOU, a confidence optimization module is introduced. It is mainly divided into two branches: the first is the classification branch, which determines the correctness of the detection by updating the score. If the IOU between the prediction and the true value is lower than a certain threshold, it is a negative sample. If it is higher than a certain value, it is a positive sample. The second is the IOU prediction branch: predict the IOU size between an object and the true value box after passing through the geometry optimization module and the position trajectory optimization module. Finally, the results of the two branches are fused, that is, the square root of the two results is taken to get the final confidence.

[0095] After using the above-mentioned optimization modules to adjust the geometric size, position trajectory, and confidence of the predicted object and then performing a consistent coordinate transformation, the final accurate four-dimensional automatic labeling result can be obtained.

[0096] In the embodiment of this example, the present invention inputs time-series point cloud data and image data to achieve four-dimensional automatic labeling of point clouds and images, reducing the cost of manual labeling and improving the labeling quality; the system architecture of the present invention is decoupled, wherein the detection, tracking, and optimization models can be replaced accordingly according to actual applications, and mature models can be reused without the need for complete training of all modules, and the operability is strong.

[0097] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0098] In addition, in this exemplary embodiment, a four-dimensional automatic annotation system based on images and point clouds is also provided. Figure 5 As shown, the four-dimensional automatic annotation system 200 based on images and point clouds may include: an offline three-dimensional detection module 210, a target tracking module 220, and an optimization module 230. Among them:

[0099] The offline 3D detection module 210 is used to take the time-series point cloud and the time-series image as input, detect the time-series point cloud based on the 3D point cloud detection model to generate a point cloud 3D detection frame, detect the time-series image based on the 3D vision detection model to generate an image 3D detection frame, cross-validate the point cloud 3D detection frame and the image 3D detection frame in the point cloud coordinate system, and generate a target detection frame;

[0100] The target tracking module 220 is used to associate the target detection frame greater than the preset confidence level to generate a forward trajectory, reversely execute the preset tracking algorithm to generate a reverse trajectory, fuse the forward trajectory and the reverse trajectory based on the weighted box fusion WBF strategy, and extract the point cloud data of each frame in the fused trajectory;

[0101] The optimization module 230 is used to optimize the geometric size, position trajectory, and confidence of the fused trajectory and the three-dimensional detection frame of each frame in the fused trajectory, and then perform consistent coordinate transformation to generate a four-dimensional labeling result.

[0102] The specific details of each of the above-mentioned four-dimensional automatic labeling system modules based on image and point cloud have been described in detail in the corresponding four-dimensional automatic labeling method based on image and point cloud, so they will not be repeated here.

[0103] It should be noted that, although several modules or units of a four-dimensional automatic annotation system 200 based on images and point clouds are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0104] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0105] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0106] Refer to the following Figure 6 An electronic device 500 according to such an embodiment of the present invention is described. Figure 6 The electronic device 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0107] like Figure 6 As shown, the electronic device 500 is in the form of a general computing device. The components of the electronic device 500 may include but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including the storage unit 520 and the processing unit 510), and a display unit 540.

[0108] The storage unit stores program codes, which can be executed by the processing unit 510, so that the processing unit 510 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 510 can perform the following steps: Figure 1 Steps S110 to S130 shown in FIG.

[0109] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 5201 and / or a cache storage unit 5202 , and may further include a read-only storage unit (ROM) 5203 .

[0110] The storage unit 520 may also include a program / utility 5204 having a set (at least one) of program modules 5205, such program modules 5205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0111] Bus 530 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0112] The electronic device 500 may also communicate with one or more external devices 570 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 500, and / or may communicate with any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 550. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 560. As shown, the network adapter 560 communicates with other modules of the electronic device 500 via a bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0113] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal system, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0114] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0115] refer to Figure 7 As shown, a program product 600 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, system or device.

[0116] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0117] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, system, or device.

[0118] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0119] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0120] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0121] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0122] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A four-dimensional automatic annotation method based on image and point cloud, characterized in that: The method comprises: The offline 3D detection step takes the time-series point cloud and the time-series image as input, detects the time-series point cloud based on the 3D point cloud detection model to generate a point cloud 3D detection frame, detects the time-series image based on the 3D vision detection model to generate an image 3D detection frame, cross-validates the point cloud 3D detection frame and the image 3D detection frame in the point cloud coordinate system, and generates a target detection frame; In the target tracking step, the target detection frame greater than the preset confidence is associated with each other to generate a forward trajectory, the tracking algorithm is executed in reverse to generate a reverse trajectory, the forward trajectory and the reverse trajectory are fused based on the weighted box fusion (WBF) strategy, and the point cloud data of each frame after the trajectory fusion is extracted; The optimization step performs geometric size, position trajectory, and confidence optimization processing on the fused trajectory and each frame of the 3D detection frame in the fused trajectory, and then performs consistent coordinate transformation to generate a 4D annotation result.

2. The method according to claim 1, characterized in that The method further comprises: detecting the time-series point cloud based on the three-dimensional point cloud detection model to generate a point cloud three-dimensional detection frame; The 3D point cloud detection model is based on the dynamic sparse voxel transformer DSVT algorithm, uses a combination of five consecutive frames of point cloud data as input, and generates a point cloud 3D detection frame based on test enhancement TTA and multi-model integration.

3. The method according to claim 1, characterized in that In the method, detecting the time series images based on the three-dimensional visual detection model to generate the three-dimensional detection frame of the image also includes: The three-dimensional visual detection model performs detection based on a multi-view 2D target guided 3D target detector MV2D algorithm to generate a three-dimensional detection frame of the image.

4. The method according to claim 1, characterized in that The target tracking step of the method further comprises: The target detection frames are divided into high confidence groups and low confidence groups based on the preset confidence, and the target detection frames in the high confidence group are associated to generate forward trajectories; After the preset tracking algorithm is executed in reverse to generate a reverse trajectory, the forward trajectory and the backward trajectory are associated with each other through position similarity.

5. The method according to claim 1, characterized in that The step of optimizing the geometric dimensions of the object in the method further comprises: The same tracking target in different frames is converted to the frame center coordinate system, and the point clouds in the trajectory of the tracking target are merged, and a group of point clouds are randomly selected from the merged point clouds; Calculate the distance between each point cloud in the randomly selected set of point clouds and the six surfaces of the detection frame; Select t-frame samples from the complete target trajectory, extract features of the t-frame samples based on the pointnet algorithm, and generate an initial geometric query Q_geo; Extracting the randomly selected set of point clouds based on the pointnet algorithm to generate dense features corresponding to the geometric key Key_geo and the geometric value V_geo respectively; The geometric query Q_geo is first sent to the multi-head self-attention mechanism for encoding to obtain the contextual information between samples. Then the geometric query Q_geo, geometric key Key_geo, and geometric value V_geo undergo a cross-attention mechanism to supplement the features of the perspective and then predict the geometric size.

6. The method according to claim 1, characterized in that The optimization step of the method further includes optimizing the position trajectory of the object: A bounding box position is randomly selected from the bounding box sequence of each target object tracked as the local coordinate system, and then other boxes in the trajectory of the target object are converted to the coordinate system. A fixed point is selected in each frame of the target object, and the distance from the fixed point to the center of the box and the eight corner points is calculated as the global position feature, that is, the position key K_pos and the position value V_pos; Based on the pointnet algorithm, features are extracted from all points in the target trajectory, and corresponding confidence scores are added to enrich the feature dimension as the position query Q_pos; The position query Q_pos is first sent to the self-attention mechanism to capture the distance between the position query Q_pos and other boxes, and then the position query Q_pos, the position key K_pos, and the position value V_pos are sent to the cross-attention mechanism to encode the local to global position, predict the offset and angle offset of the center point of each box in the local coordinate system, and adjust the position trajectory of the target object's movement.

7. The method according to claim 1, characterized in that The optimization step of optimizing the confidence of the object in the method further includes: Based on the classification branch, detection is performed by updating the score. If the IOU between the prediction and the true value is lower than the preset threshold, it is a negative sample. If it is higher than the preset threshold, it is a positive sample. Based on the prediction branch: detect the target object and optimize the geometry and position trajectory of the detection frame and the ground truth frame. The results of the classification branch and the prediction branch are fused, and the square root of the results of the classification branch and the prediction branch is calculated to generate an optimized confidence.

8. A four-dimensional automatic annotation system based on images and point clouds, characterized in that: The system comprises: An offline 3D detection module is used to take a time-series point cloud and a time-series image as input, detect the time-series point cloud based on a 3D point cloud detection model to generate a point cloud 3D detection frame, detect the time-series image based on a 3D visual detection model to generate an image 3D detection frame, cross-validate the point cloud 3D detection frame and the image 3D detection frame in a point cloud coordinate system, and generate a target detection frame; The target tracking module is used to associate the target detection frame greater than the preset confidence level to generate a forward trajectory, reversely execute the preset tracking algorithm to generate a reverse trajectory, fuse the forward trajectory and the reverse trajectory based on the weighted box fusion WBF strategy, and extract the point cloud data of each frame in the fused trajectory; The optimization module is used to optimize the geometric size, position trajectory, and confidence of the fused trajectory and the three-dimensional detection frame of each frame in the fused trajectory, and then perform consistent coordinate transformation to generate a four-dimensional labeling result.

9. An electronic device, characterized in that: include Processor; and A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images