Point cloud labeling method and system and storage medium

The method automates point cloud annotation by generating initial boxes with neural networks and adjusting them in batches based on identity markers and confidence information, enhancing efficiency and accuracy.

CN119068019APending Publication Date: 2024-12-03HESAI TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202310642791.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing point cloud annotation methods require manual intervention, leading to low efficiency and accuracy due to the need for frame-by-frame adjustments of annotation boxes.

Method used

A method that generates initial annotation boxes using neural networks, determines standard sizes based on identity markers and confidence information, and adjusts these boxes in batches to improve efficiency and accuracy.

Benefits of technology

The method automates the annotation process, reducing manual adjustments and improving efficiency and accuracy by using neural networks and multi-target tracking to determine standard sizes for annotation boxes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068019A_ABST
    Figure CN119068019A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a point cloud labeling method and system and a storage medium, and the method comprises the steps: obtaining a plurality of to-be-labeled point cloud frames; performing labeling processing on the plurality of point cloud frames, and respectively generating an initial labeling box corresponding to each obstacle in each point cloud frame; performing marking processing on each initial marking frame to obtain a pre-marking frame containing an identity label, the identity label being suitable for distinguishing an obstacle corresponding to the pre-marking frame; determining the standard size of each pre-labeling frame corresponding to the same obstacle based on each pre-labeling frame and the identity label of each pre-labeling frame; and adjusting the corresponding pre-labeling boxes in batches based on the standard sizes, and outputting the labeling boxes corresponding to the obstacles in the point cloud frames. By adopting the technical scheme, the marking efficiency of the point cloud can be improved, errors in a manual marking process can be avoided, and the marking accuracy of the point cloud is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of laser radar technology, and in particular to a point cloud annotation method, system and storage medium. Background Art

[0002] Based on the point cloud generated by LiDAR, specific objects in the point cloud can be identified through machine learning, artificial intelligence, image recognition and other methods. For example, cars, pedestrians, bicycles and other types of obstacles can be identified from the point cloud. To obtain the ability to automatically identify specific objects in the point cloud, a series of steps such as modeling, labeling, training, testing, evaluation and correction are required in the machine learning process.

[0003] An important step in machine learning is to label point clouds. Taking a car as an example, the point cloud corresponding to the car can be labeled first; then, the labeled point cloud is "fed" to the neural network, and the neural network can perform supervised learning and automatically find the internal rules of the point cloud corresponding to the car, so as to identify the specific object of the car from the new point cloud.

[0004] The point cloud map of the laser radar is displayed in three dimensions. Accordingly, a three-dimensional annotation box can be used to annotate the point clouds in the point cloud map, that is, the three-dimensional annotation box can be used to select the specific object corresponding to the point cloud. Compared with the two-dimensional annotation box, the three-dimensional annotation box can observe the specific object from multiple angles and dimensions, thereby obtaining clearer observation results.

[0005] Existing point cloud annotation methods usually manually annotate the point clouds in the acquired multi-frame point cloud frames in the form of annotation boxes; or use clustering algorithms to target cluster the point clouds in the acquired multi-frame point cloud frames, automatically generate clustering boxes that are close to the point clouds, and then manually adjust the clustering boxes to obtain the final annotation boxes.

[0006] Both of the above methods require manual participation, and the efficiency of point cloud annotation is low. Therefore, how to improve the efficiency of point cloud annotation is a technical problem that needs to be solved urgently.

[0007] The contents of the background technology section are merely the technologies known to the public and do not necessarily represent the existing technologies in the field. Summary of the invention

[0008] In view of this, an embodiment of the present invention provides a point cloud annotation method, system and storage medium, which can improve the annotation efficiency of point clouds.

[0009] First, an embodiment of the present invention provides a point cloud annotation method, including:

[0010] Get multiple point cloud frames to be annotated;

[0011] Performing annotation processing on the multiple point cloud frames, and generating initial annotation boxes corresponding to each obstacle in each point cloud frame;

[0012] Marking each initial marking frame to obtain a pre-marking frame including an identity mark, wherein the identity mark is suitable for distinguishing obstacles corresponding to the pre-marking frame;

[0013] Determine the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and the identity identifier of each pre-marked frame;

[0014] Based on the batch adjustment of the corresponding pre-annotated boxes for each standard size, the annotation boxes corresponding to each obstacle in each point cloud frame are output.

[0015] Optionally, the step of performing labeling processing on the multiple point cloud frames to generate initial labeling boxes corresponding to each obstacle in each point cloud frame includes:

[0016] The multiple point cloud frames are annotated using a preset neural network model, and an initial annotation frame corresponding to each obstacle is generated in each point cloud frame, wherein the initial annotation frame contains semantic information.

[0017] Optionally, the marking process of each initial marked frame to obtain a pre-marked frame containing an identity identifier includes:

[0018] Based on the multi-target tracking method, each initial annotation box is marked to obtain a pre-annotated box containing the identity.

[0019] Optionally, the marking process for each initial annotation frame includes:

[0020] Acquire the semantic information contained in the initial annotation frame;

[0021] Based on the semantic information, a corresponding tracking model is selected to perform labeling processing on each initial annotation box.

[0022] Optionally, the semantic information includes: attribute information and / or state information of the obstacle;

[0023] Based on the attribute information and / or state information of the obstacle, a corresponding tracking model is selected to perform marking processing on each initial annotation box.

[0024] Optionally, the determining, based on the pre-marked frames and the identities of the pre-marked frames, a standard size of each pre-marked frame corresponding to the same obstacle includes:

[0025] Obtaining first confidence information of each initial annotation box;

[0026] Obtain tracking confidence weight information corresponding to each tracking model;

[0027] Determine the comprehensive confidence of the pre-annotated frame corresponding to each initial annotated frame according to the first confidence information of each initial annotated frame and the tracking confidence weight information corresponding to the tracking model selected when marking the corresponding initial annotated frame;

[0028] Select the pre-labeled frames with the same identity to obtain the pre-labeled frames corresponding to the same obstacle;

[0029] According to the comprehensive confidences corresponding to the pre-marked frames with the same identity identifier, the standard sizes of the pre-marked frames corresponding to the same obstacle are determined.

[0030] Optionally, the determining, based on the pre-marked frames and the identities of the pre-marked frames, a standard size of each pre-marked frame corresponding to the same obstacle includes:

[0031] Obtaining first confidence information of each initial annotation box;

[0032] Select the pre-labeled frames with the same identity to obtain the pre-labeled frames corresponding to the same obstacle;

[0033] According to the first confidence information of the initial marked frames corresponding to the pre-marked frames with the same identity identifier, the standard size of each pre-marked frame corresponding to the same obstacle is determined.

[0034] Optionally, the determining, based on the pre-marked frames and the identities of the pre-marked frames, a standard size of each pre-marked frame corresponding to the same obstacle includes:

[0035] Select the pre-labeled frames with the same identity to obtain the pre-labeled frames corresponding to the same obstacle;

[0036] Integrate the point clouds in each pre-annotated frame corresponding to the same obstacle in each point cloud frame to generate a completed point cloud corresponding to the same obstacle;

[0037] The standard size of each pre-marked box corresponding to the same obstacle is determined according to the completed point cloud.

[0038] Optionally, the determining, based on the pre-marked frames and the identities of the pre-marked frames, a standard size of each pre-marked frame corresponding to the same obstacle includes:

[0039] Select the pre-labeled frames with the same identity to obtain the pre-labeled frames corresponding to the same obstacle;

[0040] The point clouds within the preset range of each pre-annotated frame corresponding to the same obstacle in the multi-frame point cloud frames are extracted respectively, and multi-frame single target detection is performed. The standard size of each pre-annotated frame corresponding to the same obstacle is determined according to the target detection result.

[0041] Optionally, the obtaining of multi-frame point cloud frames to be labeled includes: obtaining global multi-frame point cloud frames to be labeled.

[0042] Optionally, obtaining the global multi-frame point cloud frame to be annotated includes:

[0043] Obtain all point cloud frames to be annotated distributed within a preset time period;

[0044] and / or,

[0045] Get the point cloud frames to be annotated with a preset number of frames distributed continuously.

[0046] Optionally, before determining the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and the identity identifier of each pre-marked frame, the method further includes:

[0047] Pre-marked frames with the same identity are selected to obtain pre-marked frames corresponding to the same obstacle, and the pre-marked frames corresponding to the same obstacle are adjusted.

[0048] Optionally, adjusting the pre-marked frames corresponding to the same obstacle includes:

[0049] According to the global tracking method, the pre-annotated boxes that need to be adjusted in the pre-annotated boxes corresponding to the same obstacle are identified;

[0050] Correct the pre-annotated boxes that need adjustment.

[0051] Optionally, the step of identifying, according to the global tracking method, the pre-annotated frame that needs to be adjusted among the pre-annotated frames corresponding to the same obstacle includes:

[0052] For any pre-annotated frame corresponding to the same obstacle, a pre-annotated frame that needs to be adjusted among the pre-annotated frames corresponding to the same obstacle is identified according to pre-annotated frames in at least one point cloud frame before and after the point cloud frame to which it belongs.

[0053] Optionally, before determining the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and the identity identifier of each pre-marked frame, the method further includes:

[0054] Detecting whether the pre-annotated boxes corresponding to each obstacle are lost in the global multi-frame point cloud frame;

[0055] If it is detected that a pre-annotated box corresponding to any obstacle is lost in any point cloud frame in the global multi-frame point cloud frame, the lost pre-annotated box is completed.

[0056] Optionally, the completing the lost pre-annotated box includes:

[0057] For the lost pre-annotated box corresponding to the same obstacle, the lost pre-annotated box is completed in the forward time sequence according to the non-lost pre-annotated box corresponding to the same obstacle in at least one point cloud frame before the point cloud frame to which it belongs, and the lost pre-annotated box is completed in the reverse time sequence according to the non-lost pre-annotated box corresponding to the same obstacle in at least one point cloud frame after the point cloud frame to which it belongs.

[0058] Optionally, the step of batch-adjusting the corresponding pre-annotated frames based on the standard sizes and outputting the annotation frames corresponding to the obstacles in the point cloud frames includes:

[0059] Based on each standard size, batch adjust the size of the corresponding pre-marked frame to the standard size;

[0060] Batch adjust the positions of the corresponding pre-annotated boxes;

[0061] Output the annotation box corresponding to each obstacle in each point cloud frame.

[0062] Optionally, the batch adjusting the positions of the corresponding pre-marked boxes includes:

[0063] Performing clustering processing on each point cloud frame in the multiple point cloud frames, and generating a corresponding clustering frame at a corresponding position of each obstacle in each point cloud frame;

[0064] The position of the corresponding pre-labeled frame is adjusted according to the position of the clustering frame.

[0065] Optionally, before clustering the multiple point cloud frames, the method further includes: removing other point clouds in the multiple point cloud frames except for the point clouds corresponding to obstacles.

[0066] Optionally, the batch adjusting the positions of the corresponding pre-marked boxes includes:

[0067] Obtaining second confidence information of each point in each pre-annotated frame in the multi-frame point cloud frame;

[0068] Based on the standard sizes and the second confidence information distribution of each point in each pre-annotated box, the positions of the corresponding pre-annotated boxes are adjusted in batches.

[0069] Optionally, obtaining multiple point cloud frames to be annotated includes:

[0070] Acquire multiple point cloud frames to be annotated from a point cloud detection device, wherein the point cloud detection device is disposed on a movable target;

[0071] Before marking each initial annotation frame, the following steps are also included:

[0072] Acquire the position and posture information of the movable target; and adjust the position and posture information in each initial annotation frame according to the position and posture information of the movable target.

[0073] The embodiment of the present invention further provides a point cloud annotation system, including:

[0074] A perception unit, adapted to obtain multiple point cloud frames to be annotated, and perform annotation processing on the multiple point cloud frames, and generate initial annotation boxes corresponding to each obstacle in each point cloud frame;

[0075] The tracking unit is suitable for marking each initial annotation frame to obtain a pre-annotated frame including an identity identifier, determining a standard size of each pre-annotated frame corresponding to the same obstacle based on each pre-annotated frame and the identity identifier of each pre-annotated frame, and batch adjusting the corresponding pre-annotated frames based on each standard size, and outputting the annotation frame corresponding to each obstacle in each point cloud frame, wherein the identity identifier is suitable for distinguishing the obstacle corresponding to the pre-annotated frame.

[0076] Optionally, the perception unit is suitable for annotating the multiple point cloud frames through a preset neural network model, and generating an initial annotation box corresponding to each obstacle in each point cloud frame, wherein the initial annotation box contains semantic information.

[0077] Optionally, the tracking unit is adapted to perform labeling processing on each initial labeling frame based on a multi-target tracking method, and output a pre-labeling frame containing an identity identifier.

[0078] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is run, the steps of the point cloud annotation method described in any of the aforementioned embodiments are executed.

[0079] The point cloud annotation method in the embodiment of the present invention is adopted, by acquiring and annotating multiple point cloud frames to be annotated, and generating initial annotation frames corresponding to each obstacle in each point cloud frame. Then, by marking each initial annotation frame, a pre-annotated frame containing an identification mark suitable for distinguishing the obstacle corresponding to the pre-annotated frame is obtained, and the standard size of each pre-annotated frame corresponding to the same obstacle can be determined based on each pre-annotated frame and the identification mark of each pre-annotated frame, so the corresponding pre-annotated frame can be adjusted in batches based on each standard size, and the annotation frame corresponding to each obstacle in each point cloud frame is output. Since the above-mentioned whole process can realize batch adjustment of the size of each pre-annotated frame using the standard size corresponding to the same obstacle, there is no need to manually adjust each pre-annotated frame frame by frame and one by one, so the annotation efficiency of the point cloud can be improved, and the errors in the manual annotation process can also be avoided, thereby improving the accuracy of point cloud annotation.

[0080] Furthermore, the multiple point cloud frames are annotated using a preset neural network model, and an initial annotation box containing semantic information corresponding to each obstacle is generated in each point cloud frame, which can further improve the annotation efficiency.

[0081] Furthermore, the multi-frame point cloud frames are first annotated using a preset neural network model, and then each initial annotation frame is marked based on a multi-target tracking method, so that a pre-annotated frame containing an identity identifier can be automatically generated, which can improve the specificity of the annotation of obstacles with different characteristics, and is conducive to determining the standard size of each pre-annotated frame corresponding to the same obstacle, thereby further improving the accuracy of point cloud annotation.

[0082] Furthermore, by obtaining the semantic information contained in the initial annotation box, and then selecting the corresponding tracking model to mark each initial annotation box based on the semantic information, the accuracy of the multi-target tracking method can be improved, thereby further improving the accuracy of point cloud annotation.

[0083] Furthermore, by selecting pre-labeled frames with the same identity identifier to obtain pre-labeled frames corresponding to the same obstacle, and determining the standard sizes of the pre-labeled frames corresponding to the same obstacle according to the first confidence information of the initial labeling frames corresponding to the pre-labeled frames with the same identity identifier, the pre-labeled frame that can best reflect the standard size information of the obstacle can be selected from the pre-labeled frames corresponding to the same obstacle according to the first confidence information, and the standard sizes of the pre-labeled frames corresponding to the same obstacle can be determined based on the size information of the pre-labeled frame that can best reflect the standard size information of the obstacle, thereby effectively ensuring the accuracy of the standard sizes of the pre-labeled frames for the same obstacle obtained.

[0084] Furthermore, the comprehensive confidence of the pre-annotated frames corresponding to each initial annotation frame is determined based on the first confidence information of each initial annotation frame and the tracking confidence weight information corresponding to the tracking model selected when marking the corresponding initial annotation frame, and then the pre-annotated frames with the same identity are selected to obtain the pre-annotated frames corresponding to the same obstacle, and the standard size of each pre-annotated frame corresponding to the same obstacle is determined based on the comprehensive confidence corresponding to the pre-annotated frames with the same identity. The above scheme further considers the tracking confidence weight information corresponding to the adopted tracking model on the basis of considering the first confidence information of the initial annotation frame, and through the evaluation of the tracking accuracy, the accuracy of the standard size of each pre-annotated frame obtained for the same obstacle can be further improved.

[0085] Furthermore, by selecting the pre-annotated frames with the same identity, the pre-annotated frames corresponding to the same obstacle can be obtained, and then the point clouds in the pre-annotated frames corresponding to the same obstacle in each point cloud frame can be integrated to generate a completed point cloud corresponding to the same obstacle, so that the standard size of the pre-annotated frames corresponding to the same obstacle can be determined according to the completed point cloud. Compared with the point cloud in any pre-annotated frame corresponding to the same obstacle, the completed point cloud obtained by integrating the point clouds in the pre-annotated frames corresponding to the same obstacle in each point cloud frame is closer to the actual point cloud corresponding to the obstacle, so the standard size determined according to the completed point cloud is closer to the actual size corresponding to the obstacle, so that the accuracy of the standard size of the pre-annotated frames obtained for the same obstacle can be effectively guaranteed.

[0086] Furthermore, by selecting the pre-annotated frames with the same identity to obtain the pre-annotated frames corresponding to the same obstacle, point clouds within the preset range of the pre-annotated frames corresponding to the same obstacle in the multi-frame point cloud frames are respectively extracted, and multi-frame single target detection is performed, and the standard size of the pre-annotated frames corresponding to the same obstacle is determined according to the target detection results. In the above manner, since only a single target, that is, a single detection object, is targeted each time, the network structure can be effectively simplified and a more accurate target detection result can be provided, thereby further improving the accuracy of the standard size of the pre-annotated frames of the same obstacle obtained.

[0087] Furthermore, by obtaining the global multi-frame point cloud frames to be annotated, and then according to the global tracking method, identifying the pre-annotated frames that need to be adjusted in the pre-annotated frames corresponding to the same obstacle, and correcting the pre-annotated frames that need to be adjusted, the errors in the pre-annotated frames can be effectively identified and corrected based on the global perspective, thereby further improving the accuracy of point cloud annotation.

[0088] Furthermore, before determining the standard size of each pre-labeled frame corresponding to the same obstacle based on each pre-labeled frame and the identity identifier of each pre-labeled frame, by detecting whether the pre-labeled frame corresponding to each obstacle is lost in the global multi-frame point cloud frame, if it is detected that the pre-labeled frame corresponding to any obstacle is lost in any point cloud frame in the global point cloud frame, for the lost pre-labeled frame corresponding to the same obstacle, the lost pre-labeled frame is complemented in the forward time sequence according to the non-lost pre-labeled frame corresponding to the same obstacle in at least one point cloud frame before the point cloud frame to which it belongs, and the lost pre-labeled frame is complemented in the reverse time sequence according to the non-lost pre-labeled frame corresponding to the same obstacle in at least one point cloud frame after the point cloud frame to which it belongs. Compared with the frame complement algorithm in the traditional tracking module, the lost pre-labeled frame can be complemented in the forward and reverse time sequences respectively, which can improve the number of complemented pre-labeled frames and the accuracy of the complement result.

[0089] Furthermore, in addition to batch adjusting the sizes of the corresponding pre-annotated boxes to the standard sizes through various standard sizes, the steps of manually fine-tuning the positions can be omitted by batch adjusting the positions of the pre-annotated boxes, thereby further improving the point cloud annotation efficiency and the accuracy of the point cloud annotation results.

[0090] Furthermore, the second confidence information of each point in each pre-annotated frame in the multi-frame point cloud frame can be obtained, and then based on each standard size and the distribution of the second confidence information of each point in each pre-annotated frame, the position of the corresponding pre-annotated frame can be adjusted in batches. The above scheme uses the standard size and the distribution information of the points in the point cloud to realize the automatic adjustment of the position of the pre-annotated frame, which can omit the step of manually fine-tuning the position, and further improve the efficiency of point cloud annotation and the accuracy of the annotation results.

[0091] Furthermore, each point cloud frame in the multi-frame point cloud frame can be clustered separately, by generating a corresponding clustering frame at the corresponding position of each obstacle in each point cloud frame, and then adjusting the position of the corresponding pre-annotated frame in batches according to the position of the clustering frame. The above scheme realizes automatic adjustment of the position of the pre-annotated frame based on the position of the clustering frame, which can omit the step of manually fine-tuning the position, and further improves the efficiency of point cloud annotation and the accuracy of the annotation results.

[0092] Furthermore, for multiple point cloud frames to be annotated obtained from a point cloud detection device set on a movable target, before marking each initial annotation frame, the accuracy of target matching in the multi-target tracking process can be improved by obtaining and adjusting the posture information in each initial annotation frame based on the posture information of the movable target. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the embodiments of this specification or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0094] Figure 1 A schematic diagram of a process of a point cloud annotation method in an embodiment of the present invention is shown;

[0095] FIG. 2A to FIG. 2D A schematic diagram of a process of determining the standard sizes of various pre-marked boxes corresponding to the same obstacle in an embodiment of the present invention is shown;

[0096] FIG. 3A to FIG. 3B A schematic diagram of a process of adjusting corresponding pre-marked frames in batches based on various standard sizes in an embodiment of the present invention is shown;

[0097] Figure 4A and Figure 4B Schematic diagrams of pre-marked frames before and after adjustment in an embodiment of the present invention are respectively shown;

[0098] Figure 5 A schematic diagram showing the principle of correcting an erroneous point cloud frame in the existing point cloud annotation process is shown;

[0099] Figure 6 A schematic diagram showing the principle of correction of a point cloud frame with an erroneous pre-annotated frame by the point cloud annotation method in an embodiment of the present invention is shown;

[0100] Figure 7 A schematic diagram showing the principle of completing a point cloud frame with missing annotation frames by a point cloud annotation method in an embodiment of the present invention is shown;

[0101] Figure 8 A schematic diagram of the process of a point cloud annotation method in a specific application in an embodiment of the present invention is shown;

[0102] Fig. 9 A schematic diagram of a process flow of a point cloud annotation method in another specific application in an embodiment of the present invention is shown;

[0103] Fig.10 A schematic diagram of the structure of a point cloud annotation system according to an embodiment of the present invention is shown;

[0104] FIG. 11A to FIG. 11D A schematic diagram showing the structures of various tracking units in the embodiments of the present invention is shown;

[0105] Fig.12 A schematic structural diagram of another point cloud annotation system in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0106] As described in the background technology, the existing point cloud annotation methods usually manually annotate the point clouds in the acquired multi-frame point cloud frames in the form of annotated frames, or use clustering algorithms to target cluster the point clouds in the acquired multi-frame point cloud frames, automatically generate clustering frames that are close to the point clouds, and then manually adjust the clustering frames based on the annotation results of the data association between the previous and next point cloud frames to obtain the final annotation frame. This method requires manual intervention to adjust the pre-annotated frames frame by frame and one by one, and there are problems with low efficiency and low accuracy of point cloud annotation.

[0107] In response to the above problems, the inventors considered that when the size of the obstacle does not change, the sizes of the annotation boxes corresponding to the same obstacle in multiple point cloud frames should be consistent. Therefore, multiple annotation boxes corresponding to the same obstacle in multiple point cloud frames can be determined, and a standard size can be used as a reference to batch adjust the annotation boxes corresponding to the same obstacle in multiple point cloud frames, thereby improving the efficiency and accuracy of point cloud annotation.

[0108] In order to solve the above problem, in an embodiment of the present invention, the standard size of each pre-annotated frame corresponding to each obstacle is determined by respectively generating the pre-annotated frame corresponding to each obstacle in each point cloud frame and the identity identifier corresponding to the obstacle contained in each pre-annotated frame, and then the corresponding pre-annotated frame is adjusted in batches according to the determined standard sizes. Compared with manually intervening to adjust the pre-annotated frame frame by frame and one by one, adjusting the pre-annotated frame in batches according to the standard size can improve the efficiency of point cloud annotation and the accuracy of the annotation frame size.

[0109] In order to enable those skilled in the art to better understand and implement the embodiments of the present invention, the concepts, schemes, principles and advantages of the embodiments of the present invention are described in detail below with reference to the accompanying drawings and through specific application examples.

[0110] First, the embodiment of the present invention provides a point cloud annotation method, see Figure 1 The flowchart of the point cloud annotation method shown in FIG. 1 shows that the point cloud can be annotated using the following steps:

[0111] Step A: obtain multiple point cloud frames to be labeled.

[0112] In a specific implementation, multiple point cloud frames to be annotated can be obtained from a point cloud detection device, and the point cloud detection device is set on a movable target. According to different application scenarios, multiple point cloud frames to be annotated can be obtained by different point cloud detection devices. For example, the point cloud detection device can be a laser radar, a millimeter wave radar, a stereo camera, etc., and the movable target can be a car, an aircraft, a pedestrian, a mobile car, etc. As a specific example, for an autonomous driving scene, multiple point cloud frames to be annotated can be obtained by a laser radar set on a car; for capturing dynamic scenes, users can obtain multiple point cloud frames to be annotated from a point cloud data set obtained by shooting obstacles with a handheld red-green-blue depth camera (RGB-D camera).

[0113] Step B: annotating the multiple point cloud frames, and generating initial annotation boxes corresponding to each obstacle in each point cloud frame.

[0114] As an optional example, the multi-frame point cloud frames may be annotated using a preset neural network model, and an initial annotation box corresponding to each obstacle may be generated in each point cloud frame, and the initial annotation box may include semantic information. By annotating the multi-frame point cloud frames using a preset neural network model, point cloud data corresponding to obstacles may be found in the multi-frame point cloud frames to be annotated, and initial annotation boxes corresponding to each obstacle and including semantic information may be generated at corresponding positions, wherein the semantic information may include information of the corresponding obstacle. As an optional example, the semantic information may include attribute information and / or status information of the obstacle.

[0115] Specifically, the preset neural network model can be obtained by training a large number of point cloud data sets that have been labeled with corresponding obstacles through a deep learning method. Through deep learning and training, it can derive the correspondence rules between each obstacle and the point cloud data in each point cloud frame in the point cloud data set, and has the ability to automatically find the point cloud data corresponding to the obstacle in the new point cloud frame and perform labeling processing.

[0116] Step C: marking each initial marking frame to obtain a pre-marked frame including an identity mark, wherein the identity mark is suitable for distinguishing obstacles corresponding to the pre-marked frame.

[0117] In a specific implementation, various methods may be used to mark each initial annotation box, for example, a multi-target tracking method may be used.

[0118] As an optional example, in the case of multiple obstacles, each initial annotation box can be marked based on a multi-target tracking method to obtain a pre-annotated box containing an identity. Specifically, the target point clouds in each frame of point cloud data can be associated to establish the trajectory information of each target, thereby marking each initial annotation box to obtain a pre-annotated box containing an identity.

[0119] As a specific example, for multiple obstacles, based on the multi-target tracking method, the initial annotation boxes corresponding to the same obstacle in multiple point cloud frames can be marked to generate pre-annotated boxes with the same identity. The initial annotation boxes corresponding to different obstacles in the same point cloud frame can be marked to generate pre-annotated boxes with different identity.

[0120] Considering that there may be false detection and / or missed detection in the process of tracking multiple targets by the multi-target tracking method, a variety of tracking models that are adapted to the relevant information of the obstacles can be created according to the information of the obstacles, thereby improving the accuracy of target tracking.

[0121] In a specific implementation, the tracking model adapted to the relevant information of the obstacle can be obtained by training the preset neural network model, or the developer can select an adapted tracking model according to the characteristics of the obstacle. Accordingly, in this step, the semantic information contained in the initial annotation frame can be obtained, and based on the semantic information, the corresponding tracking model can be selected to mark each initial annotation frame. When the semantic information includes the attribute information and / or state information of the obstacle, the corresponding tracking model can be selected to mark each initial annotation frame based on the attribute information and / or state information of the obstacle.

[0122] The attribute information of obstacles can reflect the characteristics of obstacles. When applied to autonomous driving scenarios, different tracking models can be applied in the multi-target tracking process according to the attribute information of the vehicle. For example, a single vehicle model that considers vehicle steering modeling can be selected, such as an adaptive four-wheel vehicle front wheel steering model, a two-wheel vehicle front wheel steering model, etc.

[0123] The state information of obstacles can reflect the motion state of obstacles. When applied to autonomous driving scenarios, different tracking models can be applied in the multi-target tracking process according to the state information of the vehicle. For example, the Constant Velocity (CV) model, the Constant Acceleration (CA) model, the Constant Turning Rate and Velocity (CTRV) model, the Constant Turning Rate and Acceleration (CTRA) model, the Constant Steering Angle and Velocity (CSAV) model, the Constant Curvature and Acceleration (CCA) model, etc. In these tracking models, the obstacle objects are simplified to particle processing.

[0124] Since the initial annotation box output by deep learning itself has semantic information, when performing subsequent multi-target tracking, the appropriate tracking model can be selected based on the semantic information, thereby improving the accuracy of multi-target tracking.

[0125] Step D: determining the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and the identity of each pre-marked frame.

[0126] In a specific implementation, the pre-marked frames corresponding to the same obstacle may be determined based on the identity identifiers of the pre-marked frames, and then the standard sizes of the pre-marked frames corresponding to the same obstacle may be determined based on the pre-marked frames corresponding to the same obstacle.

[0127] The preset neural network model can be obtained through deep learning and training. In the specific implementation, through the fusion of multi-target tracking algorithm and deep learning, the pre-labeled box with identity identification can be automatically output, which is conducive to the determination of standard size.

[0128] Step E: Based on the standard sizes, the corresponding pre-annotated frames are adjusted in batches, and the annotation frames corresponding to the obstacles in each point cloud frame are output.

[0129] Based on each standard size, the size of the corresponding pre-annotated box is adjusted to the standard size in batches, and the annotation box corresponding to each obstacle in each point cloud frame is output.

[0130] The above-mentioned point cloud annotation method is adopted, and by acquiring and annotating multiple point cloud frames to be annotated, an initial annotation frame corresponding to each obstacle is generated in each point cloud frame. By marking each initial annotation frame, a pre-annotated frame containing an identity mark suitable for distinguishing the obstacle corresponding to the pre-annotated frame is obtained. Based on each pre-annotated frame and the identity mark of each pre-annotated frame, the standard size of each pre-annotated frame corresponding to the same obstacle is determined, so that the corresponding pre-annotated frame can be adjusted in batches based on each standard size, and the annotation frame corresponding to each obstacle in each point cloud frame is output. The whole process uses each standard size to realize batch adjustment of each pre-annotated frame, and there is no need to manually adjust each pre-annotated frame frame by frame and one by one, so the annotation efficiency and annotation accuracy of the point cloud can be improved.

[0131] In order to enable those skilled in the art to better understand and implement, the present invention is described in detail below through specific examples and in combination with specific application scenarios.

[0132] In a specific implementation, the standard size of each pre-marked box corresponding to the same obstacle can be determined in a variety of ways. Figure 2A The flowchart of determining the standard size of each pre-marked frame corresponding to the same obstacle shown may specifically include the following steps.

[0133] Step D11, obtaining first confidence information of each initial annotation box.

[0134] In a specific implementation, the first confidence information can be obtained based on information about one or more obstacles, such as attribute information of the obstacle, status information of the obstacle, whether the obstacle is blocked by other objects, the number of points in the initial annotation box corresponding to the obstacle, the distance between the obstacle and the point cloud detection device, the position of the obstacle in the detection angle of the point cloud detection device, etc.

[0135] For example, the farther the obstacle is from the point cloud detection device, the lower the obstacle detection accuracy, and the first confidence information of the initial annotation box corresponding to the obstacle will be reduced accordingly; the closer the obstacle is to the edge in the detection angle of the point cloud detection device, the lower the obstacle detection accuracy, and the first confidence information of the initial annotation box corresponding to the obstacle will be reduced accordingly; the fewer points included in the initial annotation box, the first confidence information of the initial annotation box will also be reduced accordingly. The first confidence information can be obtained based on one or more obstacle information by weighted accumulation, and the weighted weights corresponding to each obstacle information can be pre-set.

[0136] It can be understood that the method for obtaining the first confidence information is not limited to the above embodiment.

[0137] Since the semantic information may include the information of the obstacle, the first confidence information of each initial annotation box may be acquired based on the semantic information, or the initial annotation box with the first confidence information may be directly generated.

[0138] Step D12: Select the pre-labeled frames with the same identity to obtain the pre-labeled frames corresponding to the same obstacle.

[0139] In a specific implementation, different obstacles can be distinguished by identity identifiers. For example, obstacles corresponding to pre-labeled boxes with the same identity identifier are the same obstacle, and obstacles corresponding to pre-labeled boxes with different identity identifiers are different obstacles.

[0140] Step D13, determining the standard size of each pre-marked frame corresponding to the same obstacle according to the first confidence information of the initial marked frame corresponding to each pre-marked frame with the same identity identifier.

[0141] As an optional example, the first confidence information in each initial annotation box corresponding to the same obstacle can be compared, and the initial annotation box with the highest confidence can be selected, and the size of the corresponding pre-annotation box can be determined as the standard size of each pre-annotation box corresponding to the same obstacle.

[0142] In a specific implementation, step D11 is not necessarily executed before step D12, but may be executed after D12, or simultaneously.

[0143] In other embodiments of the present invention, see Figure 2B Another flow chart of determining the standard size of each pre-marked frame corresponding to the same obstacle may specifically include the following steps.

[0144] Step D21, obtaining first confidence information of each initial annotation box.

[0145] In a specific implementation, the method for obtaining the first confidence information of each initial annotation box refers to the relevant description of step D11 in the above embodiment, which will not be described in detail here.

[0146] Step D22, obtaining tracking confidence weight information corresponding to each tracking model.

[0147] In a specific implementation, the tracking confidence weight information corresponding to each tracking model is obtained by determining the type of the tracking model. Specifically, when creating multiple tracking models that are adapted to the relevant information of the obstacle according to the information of the obstacle, the corresponding tracking confidence weight information can be set for each tracking model, so that the tracking confidence weight information corresponding to each tracking model can be obtained by determining the type of the tracking model.

[0148] Step D23, determining the comprehensive confidence of the pre-annotated frame corresponding to each initial annotated frame according to the first confidence information of each initial annotated frame and the tracking confidence weight information corresponding to the tracking model selected when marking the corresponding initial annotated frame.

[0149] Step D24 , selecting pre-labeled frames with the same identity to obtain pre-labeled frames corresponding to the same obstacle.

[0150] Step D25, determining the standard size of each pre-labeled frame corresponding to the same obstacle according to the comprehensive confidence corresponding to each pre-labeled frame with the same identity identifier.

[0151] In a specific implementation, the comprehensive confidences corresponding to the pre-annotated boxes corresponding to the same obstacle can be compared, and the pre-annotated box corresponding to the highest comprehensive confidence can be selected, and its size can be determined as the standard size of the pre-annotated boxes corresponding to the same obstacle.

[0152] In other embodiments of the present invention, see Figure 2C Another flow chart of determining the standard size of each pre-marked frame corresponding to the same obstacle may specifically include the following steps.

[0153] Step D31 , selecting pre-labeled frames with the same identity to obtain pre-labeled frames corresponding to the same obstacle.

[0154] Step D32, integrating the point clouds in each pre-annotated frame corresponding to the same obstacle in each point cloud frame to generate a completed point cloud corresponding to the same obstacle.

[0155] Since the point clouds in each pre-annotated frame corresponding to the same obstacle in each point cloud frame may be incomplete, but the incomplete parts of the point clouds in different point cloud frames may be different, for example, the missing part in the previous frame may not be missing in the next frame. Therefore, a more complete completed point cloud can be generated by integrating the point clouds in each pre-annotated frame corresponding to the same obstacle in each point cloud frame.

[0156] Step D33, determining the standard size of each pre-labeled box corresponding to the same obstacle according to the completed point cloud.

[0157] Since the completed point cloud is more complete than the point clouds in the pre-annotated frames corresponding to the same obstacle in each point cloud frame, the size of the pre-annotated frame corresponding to the completed point cloud is used as the standard size of the pre-annotated frames corresponding to the same obstacle. A standard size that is closer to the actual size of the object can be obtained, thereby improving the accuracy of the standard size.

[0158] In other embodiments of the present invention, see Figure 2D The flowchart of determining the standard size of each pre-marked frame corresponding to the same obstacle shown may specifically include the following steps.

[0159] Step D41 , selecting pre-labeled frames with the same identity to obtain pre-labeled frames corresponding to the same obstacle.

[0160] Step D42, respectively extracting point clouds within preset ranges of pre-annotated frames corresponding to the same obstacle in the multi-frame point cloud frames, performing multi-frame single target detection, and determining standard sizes of pre-annotated frames corresponding to the same obstacle according to the target detection results.

[0161] In a specific implementation, multi-frame single target detection is performed on the point cloud within a preset range of each pre-annotated box corresponding to a single obstacle in the multi-frame point cloud frame, and the standard size of each pre-annotated box corresponding to the same obstacle is determined according to the target detection result.

[0162] As an optional example, multi-frame single target detection can be implemented by a variety of methods, such as a transformer model (Transformer), a convolutional long short-term memory network (convolutional Long Short-Term Memory, convLSTM), a convolutional recurrent neural network (convolutional Recurrent Neural Network, convRNN), etc. Among them, the preset range can be set by custom settings.

[0163] Multi-frame single target detection only targets one obstacle at a time, which can reduce the mutual interference and / or false detection that may occur between multiple obstacles. Compared with labeling or marking multiple obstacles at the same time, multi-frame single target detection can improve the accuracy and robustness of the target detection results. Therefore, the standard size of each pre-labeled box corresponding to the same obstacle determined by the target detection results of multi-frame single target detection is closer to the actual size of the object and has higher accuracy.

[0164] In a specific implementation, the corresponding pre-marked frames can be adjusted in batches based on each standard size in a variety of ways. Figure 3A The schematic diagram of a process of batch adjusting corresponding pre-marked frames based on various standard sizes shown may specifically include the following steps.

[0165] Step E11 , batch-adjusting the sizes of corresponding pre-marked frames to the standard sizes based on the standard sizes.

[0166] Step E12: batch adjust the positions of the corresponding pre-marked boxes.

[0167] In a specific implementation, each point cloud frame in the multi-frame point cloud frame can be clustered separately, a corresponding clustering frame is generated at the corresponding position of each obstacle in each point cloud frame, and then the position of the corresponding pre-labeled frame is adjusted according to the position of the clustering frame.

[0168] As an optional example, each point cloud frame in the multi-frame point cloud frame can be clustered to generate a corresponding clustering frame at the corresponding position of each obstacle in each point cloud frame; then the position of the corresponding pre-annotated frame can be adjusted based on the clustering frame so that the edge of the pre-annotated frame is close to the clustering frame, so that the annotation frame corresponding to each obstacle in the output point cloud frame better matches the point cloud corresponding to each obstacle. By batch adjusting the corresponding pre-annotated frames according to the standard sizes corresponding to each obstacle and the clustering frames, the whole process does not need to manually adjust the pre-annotated frames frame by frame and one by one, so the annotation efficiency and accuracy of the point cloud can be improved.

[0169] In a specific implementation, before clustering the multi-frame point cloud frames, other point clouds except the point clouds corresponding to obstacles in the multi-frame point cloud frames may be removed. By removing other point clouds except the point clouds corresponding to obstacles in the multi-frame point cloud frames, noise in the point cloud frames may be reduced, thereby reducing mutual interference between the point clouds corresponding to obstacles and other point clouds.

[0170] As an optional example, point clouds related to the ground in the multi-frame point cloud frames may be removed to obtain point cloud frames that only include point clouds related to obstacles.

[0171] Step E13: output the annotation box corresponding to each obstacle in each point cloud frame.

[0172] By adjusting the sizes of the corresponding pre-annotated boxes and the positions of the corresponding pre-annotated boxes in batches, the matching degree between the annotation boxes corresponding to each obstacle and the corresponding point cloud can be improved.

[0173] In a specific implementation, step E11 is not necessarily executed before step E12, but may be executed after step E12, or simultaneously.

[0174] In other embodiments of the present invention, see Figure 3B Another flow chart of batch adjusting corresponding pre-marked frames based on various standard sizes shown may specifically include the following steps.

[0175] Step E21, obtaining second confidence information of each point in each pre-labeled frame in the multi-frame point cloud frame.

[0176] In a specific implementation, the second confidence information of each point in each pre-marked frame may depend on the point cloud density of the area where each point is located. As an optional example, the higher the point cloud density of the area where each point is located, the higher the second confidence information corresponding to each point.

[0177] Step E22: based on the standard sizes and the second confidence information distribution of each point in each pre-marked frame, batch adjust the sizes and positions of the corresponding pre-marked frames.

[0178] In a specific implementation, when the pre-labeled boxes are adjusted in batches, the border of the pre-labeled box on the side where the second confidence information of the points in the box is distributed higher can be used as a reference to adjust the position of the border of the pre-labeled box on the side where the second confidence information of the points in the box is distributed lower, until the size of the pre-labeled box is adjusted to the corresponding standard size.

[0179] Step E23, outputting the annotation box corresponding to each obstacle in each point cloud frame.

[0180] Specifically, based on the distribution of the second confidence information of each point in each pre-annotated frame in the multi-frame point cloud frame, the adjustment basis for adjusting each pre-annotated frame is determined, and then the pre-annotated frames are adjusted in batches based on the standard sizes, so that the annotation frames corresponding to each obstacle in the output point cloud frames and the point clouds corresponding to each obstacle can better correspond. The entire process achieves batch adjustment of each pre-annotated frame through the distribution of the second confidence information of each point in each standard size and each pre-annotated frame, without the need to manually adjust each pre-annotated frame frame by frame and one by one, thereby improving the annotation efficiency and annotation accuracy of the point cloud.

[0181] See also Figure 4A The schematic diagram of the pre-marked box before adjustment is shown and Figure 4BThe schematic diagram of the adjusted pre-marking frame is shown in FIG. Figure 4A As shown in FIG. 1 , the point cloud density of each point g in the left area of ​​the pre-labeled frame f is greater than the point cloud density of each point g in the right area. It can be seen that the second confidence information corresponding to each point g in the left area of ​​the pre-labeled frame f is higher than that in the right area. When adjusting the pre-labeled frame f, its left border can be used as a reference to adjust its right border until its size is adjusted to the corresponding standard size, and the obtained Figure 4B The adjusted pre-annotated box f' is shown.

[0182] Assume that multiple point cloud frames to be annotated are obtained from a point cloud detection device (such as a laser radar), and the point cloud detection device is set on a movable target (such as a car). When the multiple point cloud frames are annotated, each generated initial annotation frame may contain the pose information of the movable target, for example, the pose information of the movable target is calculated according to the moving speed of the movable target. However, during the movement of the movable target, its pose may undergo other changes other than translation. For example, when the movable target passes through a speed bump, the point cloud detection device and the movable target shake, and when the movable target goes up / down a steep slope, the pitch angle of the point cloud detection device and the movable target may change. The above-mentioned pose change causes the actual pose information of the movable target and the point cloud detection device to change, and there is a deviation between the pose information of the movable target contained in the corresponding initial annotation frame. At this time, if the pose information of the movable target contained in the initial annotation frame is continued to be used to perform subsequent steps, the accuracy of target matching will be reduced.

[0183] In order to solve the above problems, in a specific implementation, before marking each initial annotation frame, high-precision posture information of the movable target can be obtained. For example, the accuracy of the posture information of the movable target can be improved by using a laser odometer, and then the posture information in each initial annotation frame can be adjusted according to the high-precision posture information of the movable target.

[0184] Before marking each initial marking frame, by obtaining high-precision posture information of the movable target and adjusting the posture information of each initial marking frame according to the high-precision posture information of the movable target, the posture information of the movable target already in each initial marking frame can be verified and corrected, thereby improving the accuracy of the posture information in each initial marking frame, reducing the deviation in the subsequent marking processing of each initial marking frame, and further improving the accuracy of target matching in subsequent steps.

[0185] In a specific implementation, before determining the standard size of each pre-labeled frame corresponding to the same obstacle based on each pre-labeled frame and the identity identifier of each pre-labeled frame, the pre-labeled frames with the same identity identifier can be selected to obtain the pre-labeled frames corresponding to the same obstacle, and the pre-labeled frames corresponding to the same obstacle can be adjusted.

[0186] Since some of the initial annotation frames in the multi-frame point cloud frames obtained by the perception method may have errors, such as false detection, missed detection, etc., the corresponding pre-annotated frames containing identity identifiers may also have errors. Therefore, before determining the standard sizes of the pre-annotated frames corresponding to the same obstacle, the pre-annotated frames corresponding to the same obstacle are obtained by selecting the pre-annotated frames with the same identity identifier, and the pre-annotated frames corresponding to the same obstacle are adjusted to correct the errors that may exist in some of the pre-annotated frames, so that the standard sizes of the pre-annotated frames corresponding to the same obstacle can be determined based on the corrected pre-annotated frames and the identity identifiers of the pre-annotated frames, thereby increasing the accuracy of the standard sizes of the pre-annotated frames corresponding to the same obstacle.

[0187] The existing point cloud annotation process is based on the idea of ​​filtering, and the annotation errors of the current point cloud frame can be identified by referring to the previous point cloud frame. Specifically, based on the comparative analysis between the current point cloud frame and the previous point cloud frame, it is determined whether there are annotation errors in the current point cloud frame. After identifying the annotation errors, the process of correcting the current point cloud frame in the existing point cloud annotation process is also based on the idea of ​​filtering, and the annotation errors in the current point cloud frame can be smoothly corrected by referring to the previous point cloud frame.

[0188] See also Figure 5 The diagram shown is a schematic diagram of the principle of correcting erroneous point cloud frames in the existing point cloud annotation process. Figure 5 The figure shows point cloud frames i, i+1, and i+2 arranged in time sequence t. There is a label frame e corresponding to the same obstacle (e.g., a car) in each point cloud frame. By comparing frame by frame, it is found that the label frame e in the point cloud frame i+2 in the xy two-dimensional coordinate system is rotated 90° counterclockwise compared to the previous point cloud frame, i.e., the label frame e corresponding to the same car in the point cloud frame i+1. However, considering that it is usually difficult for a moving car to suddenly turn 90° within a short frame interval (e.g., 0.1 second), Based on this, it can be judged that there is an error in the annotation box e corresponding to the same car in the point cloud frame i+2. To this end, the annotation box e in the point cloud frame i+2 corresponding to the same car can be corrected in combination with the annotation box e in the current point cloud frame i+2 and the annotation box e in the previous point cloud frame i+1. For example, the erroneous annotation box angle in the point cloud frame i+2 is corrected according to the smoothing principle. The annotation box e in the corrected point cloud frame i+2 is rotated 45° counterclockwise compared to the annotation box e corresponding to the same car in the previous point cloud frame i+1.

[0189] By adopting the above-mentioned existing point cloud annotation process, the annotation frame that may have errors is smoothly corrected. For point cloud frames that actually have errors, the error margin of the corrected point cloud frame will be reduced compared to the correct point cloud frame. However, the inventors found in the process of research and practice that for point cloud frames that do not actually have errors, the error margin of the corrected point cloud frame will increase compared to the correct point cloud frame, and for point cloud frames that have errors, the error margin of the corrected point cloud frame is only reduced, but it is still impossible to complete the correction completely and effectively, because it is usually difficult for a moving car to suddenly turn 45° within a short frame interval (such as 0.1 seconds).

[0190] In order to solve the above problem, in some embodiments of the present invention, a global multi-frame point cloud frame to be annotated can be obtained. Accordingly, a global tracking method can be used to identify the pre-annotated frame that needs to be adjusted among the pre-annotated frames corresponding to the same obstacle, and then correct the pre-annotated frame that needs to be adjusted.

[0191] Specifically, a global tracking method is used to view the point cloud of the global multi-frame point cloud frame to be annotated with a global vision, and the pre-annotated frame that needs to be adjusted in each pre-annotated frame corresponding to the same obstacle is identified, so as to correct the pre-annotated frame that needs to be adjusted. Compared with the aforementioned existing point cloud annotation process to identify the pre-annotated frame that needs to be adjusted, the global tracking method can more accurately identify the pre-annotated frame that needs to be adjusted and the motion state of the corresponding obstacle, thereby achieving more effective correction.

[0192] In addition, if some of the initial annotation boxes in the acquired multi-frame point cloud frames may have orientation errors, the corresponding pre-annotated boxes containing identity identifiers may also have orientation errors. At this time, when a global tracking method is used to perform multi-target tracking based on a global field of view, the overall movement of the target can be observed, so the orientation of the target can be judged based on the overall movement of the target, thereby accurately identifying the pre-annotated boxes corresponding to the same obstacle whose orientation information needs to be adjusted. Therefore, the global field of view can also be used to effectively correct the orientation information of the erroneous pre-annotated boxes.

[0193] As an optional example, for any pre-annotated frame corresponding to the same obstacle, the pre-annotated frame that needs to be adjusted among the pre-annotated frames corresponding to the same obstacle can be identified based on the pre-annotated frames in at least one point cloud frame before and after the point cloud frame to which it belongs. Accordingly, it can be adjusted based on the pre-annotated frames in at least one point cloud frame before and / or after the point cloud frame to which it belongs.

[0194] In a specific implementation, the global multi-frame point cloud frame includes all point cloud frames within a time period. Among them, the global point cloud frame can be obtained in a variety of ways. For example, all point cloud frames to be annotated distributed within a preset time period can be obtained, or point cloud frames to be annotated with a preset number of frames continuously distributed can be obtained.

[0195] In order to enable those skilled in the art to better understand the above global tracking method, a specific application scenario is used to illustrate it below.

[0196] See also Figure 6 The diagram shown is a schematic diagram of the correction principle of the point cloud annotation method in the embodiment of the present invention for a point cloud frame with an erroneous pre-annotated frame. Figure 6 The figure shows point cloud frames i, i+1, i+2, i+3, and i+4 arranged in time sequence t, wherein the aforementioned point cloud frames are global point cloud frames to be annotated distributed within a preset time period (e.g., 0.5 seconds), and each point cloud frame has a pre-annotated frame e corresponding to the same obstacle (e.g., a car). By comparing with the previous point cloud frame and the next point cloud frame, it is found that the pre-annotated frame e in the point cloud frame i+2 in the xy two-dimensional coordinate system is larger than that in the previous point cloud frame (i.e., the point cloud frame i+1). ) and the pre-annotated frame e corresponding to the same car in the next point cloud frame (i.e., point cloud frame i+3) rotates 90° counterclockwise, but under normal conditions, a moving car cannot suddenly turn 90° within a shorter frame interval (e.g., 0.1 seconds) and then rotate 90° clockwise within a shorter frame interval (e.g., 0.1 seconds). Therefore, it can be judged that the car should move in a straight line along the x direction in the time period from point cloud frame i to point cloud frame i+4, and the pre-annotated frame e corresponding to the car in point cloud frame i+2 is wrong. At this time, the pre-annotated frame e corresponding to the same car in the current point cloud frame i+2 can be corrected in combination with the pre-annotated frame e corresponding to the same car in the point cloud frame i, point cloud frame i+1 before the current point cloud frame i+2, and the point cloud frame i+3 and point cloud frame i+4 after the current point cloud frame i+2. For example, the pre-labeled box e in the corrected point cloud frame i+2 is rotated counterclockwise by 0° compared to the pre-labeled box e corresponding to the same car in the previous point cloud frame i+1 and / or the pre-labeled box e corresponding to the same car in the next point cloud frame i+3.

[0197] In a specific implementation, before determining the standard size of each pre-labeled frame corresponding to the same obstacle based on each pre-labeled frame and the identity identifier of each pre-labeled frame, it can be detected whether the pre-labeled frame corresponding to each obstacle is lost in the global multi-frame point cloud frame. If it is detected that the pre-labeled frame corresponding to any obstacle is lost in any point cloud frame in the global multi-frame point cloud frame, the lost pre-labeled frame is completed.

[0198] Since the pre-annotated frames corresponding to each obstacle may be lost in the global multi-frame point cloud frame, before determining the standard size of each pre-annotated frame corresponding to the same obstacle based on each pre-annotated frame and the identity identifier of each pre-annotated frame, the pre-annotated frames corresponding to the same obstacle can be obtained by selecting the pre-annotated frames with the same identity identifier, and the pre-annotated frames corresponding to the same obstacle in the global multi-frame point cloud frame can be detected. When it is detected that the pre-annotated frame corresponding to any obstacle is lost in any point cloud frame in the global multi-frame point cloud frame, the lost pre-annotated frame can be supplemented, so that the standard size of each pre-annotated frame corresponding to the same obstacle can be determined based on the supplemented pre-annotated frames and the identity identifier of each pre-annotated frame, thereby increasing the accuracy of the standard size of each pre-annotated frame corresponding to the same obstacle.

[0199] In a specific implementation, the loss of the corresponding pre-annotated box in each point cloud frame can be determined by judging whether the target obstacle disappears suddenly.

[0200] The existing point cloud annotation process is based on the filtering idea. The pre-annotated frame of the point cloud frame that is currently missing the pre-annotated frame is completed by referring to the pre-annotated frame in the previous point cloud frame. Due to the completion deviation in the completion process, generally only one or two frames of point cloud frames that are missing the pre-annotated frame are completed in turn.

[0201] In a specific implementation, a global multi-frame point cloud frame to be labeled can be obtained. For a lost pre-labeled frame corresponding to the same obstacle, the lost pre-labeled frame can be completed in a forward time sequence according to a non-lost pre-labeled frame corresponding to the same obstacle in at least one point cloud frame before the point cloud frame to which it belongs, and the lost pre-labeled frame can be completed in a reverse time sequence according to a non-lost pre-labeled frame corresponding to the same obstacle in at least one point cloud frame after the point cloud frame to which it belongs.

[0202] Based on the global field of view provided by the global multi-frame point cloud frame, one or two frames of point cloud frames with missing pre-annotated boxes can be completed in forward and reverse timing respectively. Compared with the existing point cloud annotation process, the number of completed pre-annotated boxes and the accuracy of the completion results can be increased.

[0203] In order to enable those skilled in the art to better understand, a specific example is used below for description.

[0204] See also Figure 7 The diagram shown is a schematic diagram of the principle of completing a point cloud frame with a missing annotation frame by a point cloud annotation method according to an embodiment of the present invention. Figure 7Point cloud frames i, point cloud frames i+1, ..., point cloud frames j-1, and point cloud frames j are shown arranged in time sequence t, wherein the aforementioned point cloud frames are all point cloud frames to be annotated distributed within a preset time period (e.g., 0.7 seconds), and there are pre-annotated frames e corresponding to the same obstacle (e.g., a car) in point cloud frames i, point cloud frames i+1, point cloud frames j-1, and point cloud frames j, and pre-annotated frames e' corresponding to the same obstacle (e.g., a car) in point cloud frames i+2 and point cloud frames j-2 are lost. At this time, the pre-annotated frame e' lost in point cloud frame i+2 can be supplemented in the forward time sequence according to the pre-annotated frame e corresponding to the same obstacle in point cloud frame i before point cloud frame i+2 and point cloud frame i+1; the pre-annotated frame e' lost in point cloud frame j-2 can be supplemented in the reverse time sequence according to the pre-annotated frame e corresponding to the same obstacle in point cloud frame j-1 after point cloud frame j-2 and point cloud frame j.

[0205] In order to enable those skilled in the art to better understand and implement the point cloud annotation method in the embodiment of the present invention, a specific example is given below for explanation.

[0206] See also Figure 8 The flowchart of a point cloud annotation method in a specific application in an embodiment of the present invention is shown. Specifically, point cloud annotation can be performed on multiple point cloud frames to be annotated according to the following process.

[0207] S001, obtaining a global multi-frame point cloud frame to be labeled.

[0208] In a specific implementation, a global multi-frame point cloud frame to be labeled is obtained from a point cloud detection device, and the point cloud detection device is set on a movable target.

[0209] S002, performing annotation processing on the global multi-frame point cloud frames, and generating initial annotation boxes corresponding to each obstacle in each point cloud frame.

[0210] In a specific implementation, the global multi-frame point cloud frames are annotated using a preset neural network model, and an initial annotation box corresponding to each obstacle is generated in each point cloud frame.

[0211] S003, obtaining the position and posture information of the movable target, and adjusting the position and posture information of each initial annotation frame according to the position and posture information of the movable target.

[0212] In a specific implementation, by obtaining the posture information of the movable target and adjusting the posture information of each initial annotation frame according to the posture information of the movable target, the posture information of the movable target already in each initial annotation frame can be verified and corrected, thereby improving the accuracy of the posture information in each initial annotation frame.

[0213] S004: Mark each initial marked frame to obtain a pre-marked frame containing an identity identifier.

[0214] In a specific implementation, based on a multi-target tracking method, each initial annotation frame is labeled to obtain a pre-annotated frame containing an identity identifier.

[0215] As a specific example, based on the multi-target tracking method, the initial annotation boxes corresponding to the same obstacle in multiple point cloud frames are marked to generate pre-annotated boxes with the same identity. The initial annotation boxes corresponding to different obstacles in the same point cloud frame are marked to generate pre-annotated boxes with different identity.

[0216] S005: Acquire the pre-marked frames corresponding to the same obstacle based on the identity identifiers of the pre-marked frames.

[0217] S006, identifying whether there is a pre-marked box that needs to be adjusted among the pre-marked boxes corresponding to the same obstacle, if yes, executing step S007, if not, executing step S008.

[0218] In a specific implementation, it is possible to identify whether there is a pre-annotated frame that needs to be adjusted among the pre-annotated frames corresponding to the same obstacle based on a global tracking method. As an optional example, for any pre-annotated frame corresponding to the same obstacle, it is possible to identify the pre-annotated frame that needs to be adjusted among the pre-annotated frames corresponding to the same obstacle based on the pre-annotated frames in at least one point cloud frame before and after the point cloud frame to which it belongs.

[0219] S007, correcting the pre-marked frame that needs to be adjusted. After the pre-marked frame is corrected, execute step S008.

[0220] In a specific implementation, the pre-annotated frame to be adjusted may be corrected according to the global tracking method. As an optional example, the pre-annotated frame to be adjusted may be adjusted according to the pre-annotated frame in at least one point cloud frame before and after the point cloud frame to which the pre-annotated frame to be adjusted belongs.

[0221] S008: Acquire the pre-marked frames corresponding to the same obstacle based on the identity identifiers of the pre-marked frames.

[0222] S009, identifying whether the pre-annotated boxes corresponding to each obstacle are missing in the global multi-frame point cloud frame, if yes, executing step S010, if not, executing step S011.

[0223] In a specific implementation, the loss of the corresponding pre-annotated box in each point cloud frame can be determined by judging whether the target obstacle disappears suddenly.

[0224] S010, completing the lost pre-annotated box, after the pre-annotated box is completed, executing step S011.

[0225] In a specific implementation, for a lost pre-annotated box corresponding to the same obstacle, the lost pre-annotated box can be completed in a forward time sequence based on a non-lost pre-annotated box corresponding to the same obstacle in at least one point cloud frame before the point cloud frame to which it belongs, and the lost pre-annotated box can be completed in a reverse time sequence based on a non-lost pre-annotated box corresponding to the same obstacle in at least one point cloud frame after the point cloud frame to which it belongs.

[0226] S011, obtaining first confidence information of each initial annotation box.

[0227] In a specific implementation, the first confidence information can be obtained based on information about one or more obstacles, such as attribute information of the obstacle, status information of the obstacle, whether the obstacle is blocked by other objects, the number of points in the initial annotation box corresponding to the obstacle, the distance between the obstacle and the point cloud detection device, the position of the obstacle in the detection angle of the point cloud detection device, etc.

[0228] S012: Acquire pre-marked frames corresponding to the same obstacle based on the identity identifiers of the pre-marked frames.

[0229] S013: Determine a standard size of each pre-marked frame corresponding to the same obstacle according to first confidence information of the initial marked frame corresponding to each pre-marked frame corresponding to the same obstacle.

[0230] In a specific implementation, the first confidence information in each initial annotation box corresponding to the same obstacle can be compared, and the initial annotation box corresponding to the first confidence information with the highest confidence can be selected, and the size of the corresponding pre-annotation box can be determined as the standard size of each pre-annotation box corresponding to the same obstacle.

[0231] S014: adjusting the sizes of corresponding pre-marked frames to standard sizes in batches based on the standard sizes.

[0232] S015, batch adjust the positions of the corresponding pre-marked boxes.

[0233] In a specific implementation, clustering processing is performed on the multiple point cloud frames, a corresponding clustering frame is generated at a corresponding position of each obstacle in each point cloud frame, and then the positions of the corresponding pre-labeled frames are adjusted in batches according to the clustering frame.

[0234] S016, outputting the annotation box corresponding to each obstacle in each point cloud frame.

[0235] See also Fig. 9The flowchart of another specific application of the point cloud annotation method in the embodiment of the present invention is shown. Specifically, point cloud annotation can be performed on multiple point cloud frames to be annotated according to the following process.

[0236] S101, obtaining a global multi-frame point cloud frame to be labeled.

[0237] S102, performing annotation processing on the global multi-frame point cloud frames, and generating initial annotation boxes corresponding to each obstacle in each point cloud frame.

[0238] S103, obtaining the position and posture information of the movable target, and adjusting the position and posture information of each initial annotation frame according to the position and posture information of the movable target.

[0239] S104: Marking is performed on each initial marked frame to obtain a pre-marked frame containing an identity identifier.

[0240] S105: Acquire pre-marked frames corresponding to the same obstacle based on the identity identifiers of the pre-marked frames.

[0241] S106, identifying whether there is a pre-marked box that needs to be adjusted among the pre-marked boxes corresponding to the same obstacle, if yes, executing step S107, if not, executing step S108.

[0242] S107, correcting the pre-marked frame that needs to be adjusted. After the pre-marked frame is corrected, step S108 is executed.

[0243] S108: Acquire pre-marked frames corresponding to the same obstacle based on the identity identifiers of the pre-marked frames.

[0244] S109, determining whether the pre-annotated boxes corresponding to the obstacles are lost in the global multi-frame point cloud frame, if so, executing step S110, if not, executing step S111.

[0245] S110, completing the lost pre-annotated box, after the pre-annotated box is completed, executing step S111.

[0246] S111: Acquire pre-marked frames corresponding to the same obstacle based on the identity identifiers of the pre-marked frames.

[0247] S112, integrating the point clouds in the pre-annotated frames corresponding to the same obstacle in each point cloud frame to generate a completed point cloud corresponding to the same obstacle.

[0248] S113, determining a standard size of each pre-labeled box corresponding to the same obstacle according to the completed point cloud.

[0249] S114, obtaining second confidence information of each point in each pre-labeled frame in the multi-frame point cloud frame.

[0250] In a specific implementation, the second confidence information of each point in each pre-marked frame may depend on the point cloud density of the area where each point is located. Specifically, the higher the point cloud density of the area where each point is located, the higher the second confidence information corresponding to each point.

[0251] S115 , based on the standard sizes and the second confidence information distribution of each point in each pre-marked box, batch adjust the sizes and positions of the corresponding pre-marked boxes.

[0252] In a specific implementation, when the pre-labeled boxes are adjusted in batches, the border of the pre-labeled box on the side where the second confidence information of the points in the box is distributed higher can be used as a reference to adjust the position of the border of the pre-labeled box on the side where the second confidence information of the points in the box is distributed lower, until the size of the pre-labeled box is adjusted to the corresponding standard size.

[0253] S116, outputting the annotation box corresponding to each obstacle in each point cloud frame.

[0254] Specifically, steps S101 to S110 may refer to steps S001 to S010 in the aforementioned embodiments, respectively, and will not be described in detail here.

[0255] For ease of implementation, the present invention also provides a point cloud annotation system, see Fig.10 The structure diagram of a point cloud annotation system shown in FIG. 1 is a schematic diagram of a point cloud annotation system. The point cloud annotation system 10 may include a sensing unit 11 and a tracking unit 12, wherein:

[0256] The perception unit 11 is adapted to obtain multiple point cloud frames to be annotated, and perform annotation processing on the multiple point cloud frames, and generate initial annotation boxes corresponding to each obstacle in each point cloud frame;

[0257] The tracking unit 12 is suitable for marking each initial annotation frame to obtain a pre-annotated frame including an identity identifier, determining a standard size of each pre-annotated frame corresponding to the same obstacle based on each pre-annotated frame and the identity identifier of each pre-annotated frame, and batch adjusting the corresponding pre-annotated frames based on each standard size, and outputting the annotation frame corresponding to each obstacle in each point cloud frame, wherein the identity identifier is suitable for distinguishing the obstacle corresponding to the pre-annotated frame.

[0258] The above-mentioned point cloud annotation system is adopted, and multiple frames of point cloud frames to be annotated are acquired and annotated by the perception unit 11, so that an initial annotation frame corresponding to each obstacle is generated in each point cloud frame. Each initial annotation frame is marked and processed by the tracking unit 12, and a pre-annotated frame containing an identity mark suitable for distinguishing the obstacle corresponding to the pre-annotated frame is obtained. The tracking unit 12 determines the standard size of each pre-annotated frame corresponding to the same obstacle based on each pre-annotated frame and the identity mark of each pre-annotated frame, so that the corresponding pre-annotated frames can be adjusted in batches based on each standard size, and the annotation frame corresponding to each obstacle in each point cloud frame is output. The point cloud annotation system 10 realizes batch adjustment of each pre-annotated frame through each standard size, without the need to manually adjust each pre-annotated frame frame by frame and one by one, so that the annotation efficiency and annotation accuracy of the point cloud can be improved.

[0259] In a specific implementation, the perception unit 11 is suitable for annotating the multiple point cloud frames through a preset neural network model, and generating an initial annotation box corresponding to each obstacle in each point cloud frame, wherein the initial annotation box contains semantic information.

[0260] Specifically, the perception unit processes the multiple point cloud frames through a preset neural network model, finds point cloud data corresponding to obstacles in the multiple point cloud frames to be annotated, and generates an initial annotation box corresponding to each obstacle and containing semantic information at the corresponding position.

[0261] In a specific implementation, the tracking unit 12 is adapted to perform labeling processing on each initial labeling frame based on a multi-target tracking method, and output a pre-labeling frame containing an identity identifier.

[0262] As an optional example, the tracking unit is based on a multi-target tracking method. After marking each initial annotation frame, each initial annotation frame corresponding to the same obstacle in multiple point cloud frames is marked to generate pre-annotated frames with the same identity. After marking, each initial annotation frame corresponding to different obstacles in the same point cloud frame is marked to generate pre-annotated frames with different identity.

[0263] In a specific implementation, the tracking unit can be implemented by various structures. Fig.11A The schematic diagram of the structure of a tracking unit shown in FIG. 1 , the tracking unit 12 may include a marking module 1211, a first determining module 1212, a first adjusting module 1213 and an output module 1214, wherein:

[0264] The marking module 1211 is adapted to mark each initial marking frame based on a multi-target tracking method, and output a pre-marked frame containing an identity identifier;

[0265] The first determination module 1212 is adapted to obtain first confidence information of each initial annotation frame, and obtain each pre-annotated frame corresponding to the same obstacle through the identity identifier, and is further adapted to determine a standard size of each pre-annotated frame corresponding to the same obstacle according to the first confidence information of each initial annotation frame corresponding to the same obstacle;

[0266] The first adjustment module 1213 is adapted to batch adjust the sizes of the corresponding pre-marked frames to the standard sizes based on the standard sizes, and batch adjust the positions of the corresponding pre-marked frames;

[0267] The output module 1214 is adapted to output a labeling box corresponding to each obstacle in each point cloud frame.

[0268] In other embodiments of the present invention, see Fig. 11B The structural diagram of another tracking unit shown in FIG. 1 is that the tracking unit 12 may include a marking module 1221, a second determining module 1222, a second adjusting module 1223 and an output module 1224, wherein:

[0269] The marking module 1221 is adapted to mark each initial marking frame based on a multi-target tracking method, and output a pre-marked frame containing an identity identifier;

[0270] The second determination module 1222 is adapted to select the pre-annotated frames with the same identity to obtain the pre-annotated frames corresponding to the same obstacle, and to integrate the point clouds in the pre-annotated frames corresponding to the same obstacle in each point cloud frame to generate a completed point cloud corresponding to the same obstacle, and is further adapted to determine the standard size of the pre-annotated frames corresponding to the same obstacle according to the completed point cloud;

[0271] The second adjustment module 1223 is adapted to obtain second confidence information of each point in each pre-annotated frame in the multi-frame point cloud frame, and batch adjust the corresponding pre-annotated frames based on each standard size and the distribution of the second confidence information of each point in each pre-annotated frame;

[0272] The output module 1224 is adapted to output a labeling box corresponding to each obstacle in each point cloud frame.

[0273] In other embodiments of the present invention, see Fig. 11C The schematic diagram of another tracking unit structure shown in FIG. 1 may include a marking module 1231, a third determining module 1232, a second adjusting module 1233 and an output module 1234, wherein:

[0274] The marking module 1231 is adapted to mark each initial marking frame based on a multi-target tracking method, and output a pre-marked frame containing an identity identifier;

[0275] The third determination module 1232 is adapted to obtain the first confidence information of each initial annotation frame and the tracking confidence weight information corresponding to each tracking model, and determine the comprehensive confidence of the pre-annotated frame corresponding to each initial annotation frame according to the obtained first confidence information of each initial annotation frame and the tracking confidence weight information corresponding to each tracking model, and is further adapted to select each pre-annotated frame with the same identity to obtain each pre-annotated frame corresponding to the same obstacle, and determine the standard size of each pre-annotated frame corresponding to the same obstacle according to the comprehensive confidence corresponding to each pre-annotated frame with the same identity;

[0276] The second adjustment module 1233 is adapted to obtain second confidence information of each point in each pre-annotated frame in the multi-frame point cloud frame, and batch adjust the corresponding pre-annotated frames based on each standard size and the distribution of the second confidence information of each point in each pre-annotated frame;

[0277] The output module 1234 is adapted to output a labeling box corresponding to each obstacle in each point cloud frame.

[0278] In other embodiments of the present invention, see Fig.11D The schematic diagram of another tracking unit structure shown in FIG. 1 is a schematic diagram of another tracking unit structure, wherein the tracking unit 12 may include a marking module 1241, a fourth determining module 1242, a first adjusting module 1243 and an output module 1244, wherein:

[0279] The marking module 1241 is adapted to mark each initial marking frame based on a multi-target tracking method, and output a pre-marked frame containing an identity identifier;

[0280] The fourth determination module 1242 is adapted to select the pre-annotated frames with the same identity to obtain the pre-annotated frames corresponding to the same obstacle, respectively extract the point clouds within the preset range of the pre-annotated frames corresponding to the same obstacle in the multi-frame point cloud frames, perform multi-frame single target detection, and determine the standard size of the pre-annotated frames corresponding to the same obstacle according to the target detection result;

[0281] The first adjustment module 1243 is adapted to batch adjust the sizes of the corresponding pre-marked frames to the standard sizes based on the standard sizes, and batch adjust the positions of the corresponding pre-marked frames;

[0282] The output module 1244 is adapted to output a labeling box corresponding to each obstacle in each point cloud frame.

[0283] In a specific implementation, the function of the tracking unit 12 may also be implemented by a plurality of functional units. As an optional example, Fig.12The schematic diagram of the structure of the point cloud annotation system shown in FIG. 20 may include a sensing unit 21, a marking unit 22, a standard size determination unit 23, an adjustment unit 24 and an output unit 25, wherein:

[0284] The perception unit 21 is adapted to obtain multiple point cloud frames to be annotated, and perform annotation processing on the multiple point cloud frames, and generate initial annotation boxes corresponding to each obstacle in each point cloud frame;

[0285] The marking unit 22 is adapted to mark each initial marking frame to obtain a pre-marked frame containing an identity mark;

[0286] The standard size determination unit 23 is adapted to determine the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and an identity identifier of each pre-marked frame, wherein the identity identifier is adapted to distinguish the obstacle corresponding to the pre-marked frame;

[0287] The adjusting unit 24 is adapted to adjust the corresponding pre-marked frames in batches based on each standard size;

[0288] The output unit 25 outputs the annotation box corresponding to each obstacle in each point cloud frame.

[0289] It should be noted that in a specific implementation, any two or more of the above-mentioned unit modules may also be integrated or combined. The embodiments of the present invention are not limited to the point cloud annotation system of the above structure, as long as point cloud annotation can be implemented based on the principles of the embodiments of the present invention.

[0290] In a specific implementation, the point cloud annotation system in the embodiment of the present invention can be implemented by using the point cloud annotation method described in any of the aforementioned embodiments. For details, please refer to the aforementioned embodiments and will not be described in detail here.

[0291] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed, the steps of the point cloud annotation method described in any of the aforementioned embodiments are performed. The specific steps can be referred to the aforementioned embodiments and will not be repeated here.

[0292] In a specific implementation, the computer-readable storage medium may be any appropriate readable storage medium such as an optical disk, a mechanical hard disk, a solid-state hard disk, etc.

[0293] It should be noted that the terms "first", "second", "third", "fourth" and the like in the embodiments of the present invention are only used to distinguish the descriptions and are not used to limit their meanings.

[0294] Although the embodiments of the present invention are disclosed above, the present invention is not limited thereto. Any person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of this specification shall be subject to the scope defined by the claims.

Claims

1. A point cloud annotation method, characterized in that: include: Get multiple point cloud frames to be annotated; Performing annotation processing on the multiple point cloud frames, and generating initial annotation boxes corresponding to each obstacle in each point cloud frame; Marking each initial marking frame to obtain a pre-marking frame including an identity mark, wherein the identity mark is suitable for distinguishing obstacles corresponding to the pre-marking frame; Determine the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and the identity identifier of each pre-marked frame; Based on the batch adjustment of the corresponding pre-annotated frames of each standard size, the annotation frames corresponding to each obstacle in each point cloud frame are output.

2. The method according to claim 1, characterized in that The step of labeling the multiple point cloud frames and generating initial labeling boxes corresponding to the obstacles in each point cloud frame includes: The multiple point cloud frames are annotated using a preset neural network model, and an initial annotation frame corresponding to each obstacle is generated in each point cloud frame, wherein the initial annotation frame contains semantic information.

3. The method according to claim 2, characterized in that The marking process of each initial marked frame to obtain a pre-marked frame containing an identity identifier includes: Based on the multi-target tracking method, each initial annotation box is marked to obtain a pre-annotated box containing the identity.

4. The method according to claim 3, characterized in that The marking process for each initial annotation frame includes: Acquire the semantic information contained in the initial annotation frame; Based on the semantic information, a corresponding tracking model is selected to perform labeling processing on each initial annotation box.

5. The method according to claim 4, characterized in that The semantic information includes: attribute information and / or state information of the obstacle; Based on the attribute information and / or state information of the obstacle, a corresponding tracking model is selected to perform marking processing on each initial annotation box.

6. The method according to claim 4, characterized in that The determining of the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and the identity identifier of each pre-marked frame includes: Obtaining first confidence information of each initial annotation box; Obtain the tracking confidence weight information corresponding to each tracking model; Determine the comprehensive confidence of the pre-annotated frame corresponding to each initial annotated frame according to the first confidence information of each initial annotated frame and the tracking confidence weight information corresponding to the tracking model selected when marking the corresponding initial annotated frame; Select the pre-labeled frames with the same identity to obtain the pre-labeled frames corresponding to the same obstacle; According to the comprehensive confidences corresponding to the pre-marked frames with the same identity identifier, the standard sizes of the pre-marked frames corresponding to the same obstacle are determined.

7. The method according to claim 1, characterized in that The determining of the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and the identity identifier of each pre-marked frame includes: Obtaining first confidence information of each initial annotation box; Select the pre-labeled frames with the same identity to obtain the pre-labeled frames corresponding to the same obstacle; According to the first confidence information of the initial marked frames corresponding to the pre-marked frames with the same identity identifier, the standard size of each pre-marked frame corresponding to the same obstacle is determined.

8. The method according to claim 1, characterized in that The determining of the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and the identity identifier of each pre-marked frame includes: Select the pre-labeled frames with the same identity to obtain the pre-labeled frames corresponding to the same obstacle; Integrate the point clouds in each pre-annotated frame corresponding to the same obstacle in each point cloud frame to generate a completed point cloud corresponding to the same obstacle; The standard size of each pre-marked box corresponding to the same obstacle is determined according to the completed point cloud.

9. The method according to claim 1, characterized in that: The determining of the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and the identity identifier of each pre-marked frame includes: Select the pre-labeled frames with the same identity to obtain the pre-labeled frames corresponding to the same obstacle; The point clouds within the preset range of each pre-annotated frame corresponding to the same obstacle in the multi-frame point cloud frames are extracted respectively, and multi-frame single target detection is performed. The standard size of each pre-annotated frame corresponding to the same obstacle is determined according to the target detection result.

10. The method according to claim 1, characterized in that The step of obtaining multiple point cloud frames to be annotated includes: Get the global multi-frame point cloud frame to be annotated.

11. The method according to claim 10, characterized in that Acquiring the global multi-frame point cloud frame to be annotated, including: Obtain all point cloud frames to be annotated distributed within a preset time period; and / or, Get the point cloud frames to be annotated with a preset number of frames distributed continuously.

12. The method according to claim 10, characterized in that Before determining the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and the identity identifier of each pre-marked frame, the method further includes: Pre-marked frames with the same identity are selected to obtain pre-marked frames corresponding to the same obstacle, and the pre-marked frames corresponding to the same obstacle are adjusted.

13. The method according to claim 12, characterized in that The pre-marked frames corresponding to the same obstacle are adjusted, including: According to the global tracking method, the pre-annotated boxes that need to be adjusted in the pre-annotated boxes corresponding to the same obstacle are identified; Correct the pre-annotated boxes that need adjustment.

14. The method according to claim 13, characterized in that The step of identifying the pre-annotated frame that needs to be adjusted among the pre-annotated frames corresponding to the same obstacle according to the global tracking method includes: For any pre-annotated frame corresponding to the same obstacle, a pre-annotated frame that needs to be adjusted among the pre-annotated frames corresponding to the same obstacle is identified according to pre-annotated frames in at least one point cloud frame before and after the point cloud frame to which it belongs.

15. The method according to claim 10, characterized in that Before determining the standard size of each pre-marked frame corresponding to the same obstacle based on each pre-marked frame and the identity identifier of each pre-marked frame, the method further includes: Detecting whether the pre-annotated boxes corresponding to each obstacle are lost in the global multi-frame point cloud frame; If it is detected that a pre-annotated box corresponding to any obstacle is lost in any point cloud frame in the global multi-frame point cloud frame, the lost pre-annotated box is completed.

16. The method according to claim 15, characterized in that The method of completing the lost pre-annotated frame includes: For the lost pre-annotated box corresponding to the same obstacle, the lost pre-annotated box is completed in the forward time sequence according to the non-lost pre-annotated box corresponding to the same obstacle in at least one point cloud frame before the point cloud frame to which it belongs, and the lost pre-annotated box is completed in the reverse time sequence according to the non-lost pre-annotated box corresponding to the same obstacle in at least one point cloud frame after the point cloud frame to which it belongs.

17. The method according to claim 1, characterized in that The step of batch adjusting the corresponding pre-annotated frames based on the standard sizes and outputting the annotation frames corresponding to the obstacles in the point cloud frames includes: Based on each standard size, batch adjust the size of the corresponding pre-marked frame to the standard size; Batch adjust the positions of the corresponding pre-annotated boxes; Output the annotation box corresponding to each obstacle in each point cloud frame.

18. The method according to claim 17, characterized in that The batch adjustment of the positions of the corresponding pre-marked boxes includes: Performing clustering processing on each point cloud frame in the multiple point cloud frames, and generating a corresponding clustering frame at a corresponding position of each obstacle in each point cloud frame; The position of the corresponding pre-labeled frame is adjusted according to the position of the clustering frame.

19. The method according to claim 18, characterized in that Before clustering the multiple point cloud frames, the method further includes: The other point clouds except the point clouds corresponding to the obstacles are removed from the multi-frame point cloud frames.

20. The method according to claim 17, characterized in that The batch adjustment of the positions of the corresponding pre-marked boxes includes: Obtaining second confidence information of each point in each pre-annotated frame in the multi-frame point cloud frame; Based on the standard sizes and the second confidence information distribution of each point in each pre-annotated box, the positions of the corresponding pre-annotated boxes are adjusted in batches.

21. The method according to claim 1, characterized in that The step of obtaining multiple point cloud frames to be annotated includes: Acquire multiple point cloud frames to be annotated from a point cloud detection device, wherein the point cloud detection device is disposed on a movable target; Before marking each initial annotation frame, the following steps are also included: Acquire the position and posture information of the movable target; and adjust the position and posture information in each initial annotation frame according to the position and posture information of the movable target.

22. A point cloud annotation system, characterized in that: include: A perception unit, adapted to obtain multiple point cloud frames to be annotated, and perform annotation processing on the multiple point cloud frames, and generate initial annotation boxes corresponding to each obstacle in each point cloud frame; The tracking unit is suitable for marking each initial annotation frame to obtain a pre-annotated frame including an identity identifier, determining a standard size of each pre-annotated frame corresponding to the same obstacle based on each pre-annotated frame and the identity identifier of each pre-annotated frame, and batch adjusting the corresponding pre-annotated frames based on each standard size, and outputting the annotation frame corresponding to each obstacle in each point cloud frame, wherein the identity identifier is suitable for distinguishing the obstacle corresponding to the pre-annotated frame.

23. The system according to claim 22, characterized in that The perception unit is suitable for annotating the multiple point cloud frames through a preset neural network model, and generating an initial annotation box corresponding to each obstacle in each point cloud frame, wherein the initial annotation box contains semantic information.

24. The system according to claim 23, characterized in that The tracking unit is adapted to perform labeling processing on each initial labeling frame based on a multi-target tracking method, and output a pre-labeling frame containing an identity identifier.

25. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is run, the steps of the method according to any one of claims 1 to 21 are executed.

Citation Information

Patent Citations

  • Method and device for processing point cloud data

    CN107818293A

  • Target detection and tracking method and system based on multi-dimensional point cloud features

    CN114419152A

  • Method and device for generating point cloud labeling model, electronic equipment and storage medium

    CN115147474A

  • Novel compounds isolated from spinach and composition for preventing or treating inflammatory diseases comprising the same

    KR102748813B1