Point cloud annotation method, device, computer equipment and storage medium

Abstract

A point cloud annotation method, device, computer equipment and storage medium, comprising: obtaining a point cloud sequence to be annotated (S202); pre-annotating each frame of point cloud in the point cloud sequence to be annotated, obtaining pre-annotation information of each frame of point cloud, the pre-annotation information including: identification information and first pose information of an annotation frame for marking each target object in the corresponding frame of point cloud (S204); determining the annotation frame sequence corresponding to each target object according to the identification information, and determining the first point cloud in each annotation frame in the annotation frame sequence according to the first pose information of each annotation frame in the annotation frame sequence (S206); obtaining a point cloud registration result between corresponding annotation frames according to the first point cloud in each annotation frame in the annotation frame sequence, and obtaining the second pose information of each annotation frame in the annotation frame sequence based on the point cloud registration result (S028); determining the annotation result of the point cloud sequence to be annotated according to the second pose information of each annotation frame in each annotation frame sequence (S210). The method can improve the annotation efficiency and the annotation accuracy.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a point cloud annotation method, device, computer equipment and storage medium. Background Art

[0002] With the development of technologies such as sensors and deep learning, autonomous driving has made great progress. In autonomous driving, multi-sensor fusion is often used to perceive the surrounding environment. LiDAR has attracted much attention from autonomous driving researchers because it can directly provide an accurate three-dimensional scene, which makes the point cloud-based method the mainstream in three-dimensional perception. As the basis of deep learning, the quality of labeled data will directly affect the effect of the final model, so point cloud labeling has become a very critical part of autonomous driving.

[0003] Currently, point cloud annotation mostly relies on pure manual methods, which has low annotation efficiency and low annotation accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a point cloud annotation method, device, computer equipment and storage medium that can improve annotation efficiency and annotation accuracy in response to the above technical problems.

[0005] A point cloud annotation method, the method comprising:

[0006] Get the point cloud sequence to be annotated;

[0007] Pre-annotating each frame point cloud in the sequence of point clouds to be annotated to obtain pre-annotation information of each frame point cloud, wherein the pre-annotation information includes: identification information and first position information of an annotation box for marking each target object in the corresponding frame point cloud;

[0008] Determine a labeling frame sequence corresponding to each target object according to the identification information, and determine a first point cloud in each labeling frame in the labeling frame sequence according to the first position information of each labeling frame in the labeling frame sequence;

[0009] According to the first point cloud in each annotation frame in the annotation frame sequence, obtaining a point cloud registration result between corresponding annotation frames, and based on the point cloud registration result, obtaining second pose information of each annotation frame in the annotation frame sequence;

[0010] The labeling result of the to-be-labeled point cloud sequence is determined according to the second pose information of each labeling frame in each labeling frame sequence.

[0011] A point cloud annotation device, comprising:

[0012] An acquisition module is used to obtain a point cloud sequence to be annotated;

[0013] A pre-annotation module is used to pre-annotate each frame point cloud in the sequence of point clouds to be annotated, and obtain pre-annotation information of each frame point cloud, wherein the pre-annotation information includes: identification information and first pose information of the annotation box for marking each target object in the corresponding frame point cloud;

[0014] A processing module, configured to determine a sequence of annotation boxes corresponding to each target object according to the identification information, and determine a first point cloud in each annotation box in the sequence of annotation boxes according to the first position information of each annotation box in the sequence of annotation boxes;

[0015] A registration module, configured to obtain a point cloud registration result between corresponding annotation frames according to a first point cloud in each annotation frame in the annotation frame sequence, and obtain second pose information of each annotation frame in the annotation frame sequence based on the point cloud registration result;

[0016] The determination module is used to determine the labeling result of the to-be-labeled point cloud sequence according to the second pose information of each labeling box in each labeling box sequence.

[0017] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Get the point cloud sequence to be annotated;

[0019] Pre-annotating each frame point cloud in the sequence of point clouds to be annotated to obtain pre-annotation information of each frame point cloud, wherein the pre-annotation information includes: identification information and first position information of an annotation box for marking each target object in the corresponding frame point cloud;

[0020] Determine a labeling frame sequence corresponding to each target object according to the identification information, and determine a first point cloud in each labeling frame in the labeling frame sequence according to the first position information of each labeling frame in the labeling frame sequence;

[0021] According to the first point cloud in each annotation frame in the annotation frame sequence, obtaining a point cloud registration result between corresponding annotation frames, and based on the point cloud registration result, obtaining second pose information of each annotation frame in the annotation frame sequence;

[0022] The labeling result of the to-be-labeled point cloud sequence is determined according to the second pose information of each labeling frame in each labeling frame sequence.

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0024] Get the point cloud sequence to be annotated;

[0025] Pre-annotating each frame point cloud in the sequence of point clouds to be annotated to obtain pre-annotation information of each frame point cloud, wherein the pre-annotation information includes: identification information and first position information of an annotation box for marking each target object in the corresponding frame point cloud;

[0026] Determine a labeling frame sequence corresponding to each target object according to the identification information, and determine a first point cloud in each labeling frame in the labeling frame sequence according to the first position information of each labeling frame in the labeling frame sequence;

[0027] According to the first point cloud in each annotation frame in the annotation frame sequence, obtaining a point cloud registration result between corresponding annotation frames, and based on the point cloud registration result, obtaining second pose information of each annotation frame in the annotation frame sequence;

[0028] The labeling result of the to-be-labeled point cloud sequence is determined according to the second pose information of each labeling frame in each labeling frame sequence.

[0029] The above-mentioned point cloud annotation method, device, computer equipment and storage medium obtain the annotation box sequence corresponding to each target object by pre-annotating the point cloud sequence. For each annotation box sequence, the pose information of the annotation box is corrected by the point cloud alignment result between the corresponding annotation boxes. For example, humans can only participate in correcting the pose information of one annotation box in the annotation box sequence, and the pose information of all other annotation boxes in the annotation box sequence can be corrected by point cloud alignment, thereby reducing manual intervention, improving the efficiency of point cloud annotation, and having higher annotation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is an internal structure diagram of a computer device in one embodiment;

[0031] Figure 2 A schematic diagram of a process of a point cloud annotation method in one embodiment;

[0032] Figure 3 is a flow chart of step S208 in one embodiment;

[0033] Figure 4 is a flow chart of step S306 in one embodiment;

[0034] Figure 5 4 is a structural block diagram of a point cloud annotation device in one embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] The point cloud annotation method provided in this application can be applied to Figure 1 The computer device shown in FIG. 1 may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 1 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a point cloud annotation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0037] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0038] In one embodiment, Figure 2 As shown in FIG. 1 , a point cloud annotation method is provided, which is applied to Figure 1 Taking the computer device in as an example, the method includes the following steps S202 to S210.

[0039] S202, obtaining a point cloud sequence to be annotated.

[0040] The point cloud sequence to be annotated includes continuous frame point clouds, which can be point clouds continuously collected for the target scene. For example, in the vehicle intelligent driving application, the target scene is a road scene, and the detection equipment (such as a laser radar) is installed on the collection vehicle. As the collection vehicle moves to collect point clouds on the road, the computer device communicates with the detection equipment to obtain the point cloud collected by the detection equipment.

[0041] S204, pre-annotating each frame point cloud in the sequence of point clouds to be annotated, and obtaining pre-annotation information of each frame point cloud, wherein the pre-annotation information includes: identification information and first pose information of an annotation box for marking each target object in the corresponding frame point cloud.

[0042] A point cloud target detection algorithm and a multi-target tracking algorithm, or a detection and tracking joint processing algorithm, can be used to pre-label each frame of the point cloud in the point cloud sequence to be labeled, and obtain pre-labeling information of each frame of the point cloud. It should be noted that the pre-labeling can be implemented using any possible algorithm that is currently available or may appear in the future.

[0043] For any frame point cloud, its pre-annotation information includes: identification information and first pose information of the annotation box used to mark each target object in the frame point cloud. The target object represents the object to be marked. For example, in a road scene, the target object may be a vehicle. The annotation box may specifically be a regular three-dimensional bounding box, such as a rectangular box. The identification information is used to distinguish the annotation boxes corresponding to different target objects. The identification information of the annotation boxes corresponding to the same target object in different frame point clouds is the same. The first pose information is used to characterize the position and posture of the annotation box, which may specifically include the coordinates and orientation of the center point of the annotation box.

[0044] S206 , determining a labeling box sequence corresponding to each target object according to the identification information, and determining a first point cloud in each labeling box in the labeling box sequence according to the first position information of each labeling box in the labeling box sequence.

[0045] Determining a sequence of annotation boxes corresponding to each target object according to identification information may specifically include the following steps: determining each annotation box with the same identification information as an annotation box corresponding to the same target object in different frame point clouds; and combining the annotation boxes corresponding to the same target object in different frame point clouds into a sequence of annotation boxes corresponding to the same target object.

[0046] Since the identification information of the annotation boxes corresponding to the same target object in different frame point clouds is the same, after obtaining the identification information of the annotation boxes of each target object in each frame point cloud, all the annotation boxes can be classified according to the identification information, and the annotation boxes corresponding to the same identification information are classified into one category. Each category of standard boxes corresponds to the same target object, and each category of annotation boxes constitutes a annotation box sequence, including the annotation boxes corresponding to the same target object in different frame point clouds. The annotation box sequence can represent the annotation box trajectory, which can also be understood as the motion trajectory of the corresponding target object.

[0047] For any annotation box, the space range contained in the annotation box can be determined according to the first pose information and size information of the annotation box, and the point cloud within the space range can be determined as the first point cloud in the annotation box. The first point cloud can be understood as the point cloud corresponding to the pre-annotated target object.

[0048] S208, obtaining point cloud registration results between corresponding annotation frames according to the first point cloud in each annotation frame in the annotation frame sequence, and obtaining second pose information of each annotation frame in the annotation frame sequence based on the point cloud registration results.

[0049] The pre-annotated annotation box may not accurately mark the target object, and the first pose information of each annotation box can be corrected, and the second pose information represents the corrected pose information of the annotation box. It can be understood that overlapping point clouds are likely to exist in each frame of continuously collected point clouds, so for any sequence of annotation boxes, the first point clouds in each annotation box are also likely to have overlapping point clouds. Based on this, the point clouds in each annotation box can be aligned to obtain the point cloud alignment results between the corresponding annotation boxes, and the pose information of the corresponding annotation box can be corrected based on the point cloud alignment results.

[0050] S210, determining a labeling result of the to-be-labeled point cloud sequence according to the second pose information of each labeling box in each labeling box sequence.

[0051] After obtaining the second pose information of each annotation box in each annotation box sequence, it can be considered that the annotation of all frame point clouds in the point cloud sequence to be annotated is completed. The annotation result of the point cloud sequence to be annotated includes the identification information and the second pose information of all annotation boxes.

[0052] In the above point cloud annotation method, the point cloud sequence is pre-annotated to obtain the annotation box sequence corresponding to each target object. For each annotation box sequence, the pose information of the annotation box is corrected by the point cloud registration result between the corresponding annotation boxes. For example, humans can only participate in correcting the pose information of one annotation box in the annotation box sequence, and the pose information of all other annotation boxes in the annotation box sequence can be corrected through point cloud registration, thereby reducing manual intervention, improving the efficiency of point cloud annotation, and having higher annotation accuracy.

[0053] In one embodiment, the pre-annotation information also includes first size information of the annotation box, and the size information specifically includes the length, width and height of the annotation box. For each annotation box sequence, the following processing can also be performed: according to the first size information and the first pose information of each annotation box in the annotation box sequence, the number of point clouds in each annotation box is counted; according to the number of point clouds in each annotation box, the first size information is corrected to obtain the second size information of each annotation box in the annotation box sequence.

[0054] The first size information of the pre-annotated annotation box may not match the actual size of the target object. For example, if the annotation box is not close to the edge of the target object, it may contain more noise point clouds that do not belong to the target object. In addition, for a sequence of annotation boxes, although corresponding to the same target object, the first size information of each annotation box may be different. Based on this, the first size information of each annotation box can be corrected to obtain the second size information that is more consistent with the actual size of the target object. It can be understood that the more point clouds there are in the annotation box, the richer the information reflected in the box, which is convenient for the annotator to identify the actual size of the target object in the box. Therefore, the first size information of the annotation box can be corrected according to the number of point clouds in each annotation box.

[0055] In one embodiment, according to the number of point clouds in each annotation box, the first size information is corrected to obtain the second size information of each annotation box in the annotation box sequence. Specifically, the following steps may be included: determining the maximum number of point clouds from the number of point clouds in each annotation box, and obtaining the corrected size information of the annotation box corresponding to the maximum number of point clouds; using the corrected size information instead of the first size information as the second size information of each annotation box in the annotation box sequence.

[0056] For each annotation box sequence, by counting the number of point clouds in each annotation box, the annotation box with the largest number of point clouds can be found, and then it can be decided whether the first size information of the annotation box needs to be modified according to the actual situation. Specifically, the annotation personnel can decide whether to modify the size of the annotation box by observing with the naked eye whether the annotation box is close to the target object. If the first size information does not need to be modified, the revised size information is the first size information of the annotation box with the largest number of point clouds; if the first size information is modified, the revised size information is the revised size information of the annotation box with the largest number of point clouds. After obtaining the revised size information, the first size information of all the annotation boxes in the annotation box sequence is uniformly updated to the revised size information, and the second size information is the revised size information.

[0057] In one embodiment, the first point cloud in each annotation box in the annotation box sequence is determined based on the first pose information of each annotation box in the annotation box sequence. Specifically, the first point cloud in each annotation box in the annotation box sequence can be determined based on the first pose information and the second size information of each annotation box in the annotation box sequence.

[0058] When the size information of the annotation box is corrected, the point cloud information in the annotation box may also change accordingly. The first point cloud in the annotation box is determined according to the first pose information of the annotation box and the corrected size information (i.e., the second size information) for subsequent alignment, which can reduce noise interference in the point cloud alignment process and is conducive to obtaining more accurate alignment results. It can be understood that when the size information of the annotation box is corrected, the pose information of the annotation box may also change accordingly. At this time, the first point cloud in the annotation box can be determined according to the corrected pose information and the corrected size information of the annotation box.

[0059] In one embodiment, Figure 3 As shown, according to the first point cloud in each annotation box in the annotation box sequence, the point cloud alignment result between the corresponding annotation boxes is obtained, and based on the point cloud alignment result, the step of obtaining the second pose information of each annotation box in the annotation box sequence can specifically include the following steps S302 to S306.

[0060] S302, determining a pre-corrected annotation frame from the annotation frame sequence, and obtaining corrected position and pose information of the pre-corrected annotation frame as second position and pose information of the pre-corrected annotation frame.

[0061] The pre-corrected annotation frame refers to an annotation frame with relatively accurate pose information. The corrected pose information can be obtained by the annotator modifying the first pose information of the pre-corrected annotation frame according to the actual situation. It can be understood that when the first pose information of the pre-corrected annotation frame does not need to be modified, the corrected pose information of the pre-corrected annotation frame is the first pose information of the pre-corrected annotation frame.

[0062] S304: Determine a second point cloud within the pre-corrected annotation frame according to the second pose information of the pre-corrected annotation frame.

[0063] When the pose information of the annotation frame is corrected, the point cloud information in the annotation frame may also change accordingly. The point cloud in the pre-corrected annotation frame is determined according to the second pose information of the pre-corrected annotation frame as the second point cloud in the pre-corrected annotation frame.

[0064] S306, according to the second point cloud in the pre-corrected annotation frame and the first point cloud in each annotation frame to be corrected, obtain the point cloud alignment result between the corresponding annotation frames, and based on the point cloud alignment result, obtain the second pose information of each annotation frame to be corrected, the annotation frame to be corrected represents the annotation frame in the annotation frame sequence except the pre-corrected annotation frame.

[0065] For each annotation frame to be corrected, there is no need for the annotator to manually correct the pose information. The pose information of the annotation frame to be corrected can be corrected by the point cloud registration result between the annotation frame to be corrected and the corrected annotation frame. The corrected annotation frame here includes the pre-corrected annotation frame, and can also include the annotation frame to be corrected after the pose information is corrected based on the point cloud registration result.

[0066] For example, a labeling frame sequence L1 includes labeling frame 1, labeling frame 2, ..., labeling frame 10 in the corresponding point cloud acquisition order, which correspond to the same target object in the point clouds of the 1st frame to the 10th frame respectively. For this labeling frame sequence L1, you can select any labeling frame as the pre-correction labeling frame. For example, select labeling frame 1 as the pre-correction labeling frame. For any labeling frame among labeling frames 2-10, you can correct the pose information according to the point cloud alignment result with labeling frame 1, or you can correct the pose information according to the point cloud alignment result with the previous corrected labeling frame.

[0067] In one embodiment, the pre-corrected annotation frame is determined from the annotation frame sequence, and specifically, the first annotation frame in the annotation frame sequence is used as the pre-corrected annotation frame. Figure 4 As shown, according to the second point cloud in the pre-corrected annotation frame and the first point cloud in each annotation frame to be corrected, the point cloud alignment result between the corresponding annotation frames is obtained, and based on the point cloud alignment result, the step of obtaining the second pose information of each annotation frame to be corrected can specifically include the following steps S402 to S406.

[0068] S402, taking the pre-corrected annotation frame as the reference annotation frame, taking the subsequent annotation frame of the reference annotation frame as the current annotation frame to be corrected, aligning the first point cloud in the current annotation frame to be corrected with the second point cloud in the reference annotation frame, obtaining the point cloud alignment result between the current annotation frame to be corrected and the reference annotation frame, and obtaining the second pose information of the current annotation frame to be corrected based on the point cloud alignment result.

[0069] The iterative closest point (ICP) algorithm can be used to align the first point cloud in the current annotation frame to be corrected with the second point cloud in the reference annotation frame. The point cloud alignment result between the current annotation frame to be corrected and the reference annotation frame can specifically include the pose transformation parameters of the current annotation frame to be corrected relative to the reference annotation frame. Based on the point cloud alignment result, the second pose information of the current annotation frame to be corrected is obtained. Specifically, the second pose information of the reference annotation frame is transformed using the pose transformation parameters of the current annotation frame to be corrected relative to the reference annotation frame to obtain the second pose information of the current annotation frame to be corrected.

[0070] The pose information may specifically include the center point coordinates (x, y, z) and the heading of the annotation box. The heading includes the rotation angle of the annotation box on the xy plane. The pose transformation parameters may specifically include a rotation parameter (R) and a translation parameter (T). The rotation parameter (R) and the translation parameter (T) are applied to the second pose information of the reference annotation box to obtain the second pose information of the current annotation box to be corrected.

[0071] S404, obtaining a second point cloud in the current annotation box to be corrected according to the second pose information of the current annotation box to be corrected.

[0072] When the pose information of the current annotation frame to be corrected is corrected, the point cloud information in the current annotation frame to be corrected may also change accordingly. The point cloud in the current annotation frame to be corrected is determined according to the second pose information of the current annotation frame to be corrected as the second point cloud in the current annotation frame to be corrected.

[0073] S406, taking the current annotation frame to be corrected as a new reference annotation frame, returning to the next annotation frame of the reference annotation frame as the current annotation frame to be corrected, and performing a step of aligning the first point cloud in the current annotation frame to be corrected with the second point cloud in the reference annotation frame, until the second pose information of all the annotation frames to be corrected in the annotation frame sequence is obtained.

[0074] For example, a labeling frame sequence L1 includes labeling frame 1, labeling frame 2, ..., labeling frame 10 in the order of corresponding point cloud acquisition, which correspond to the same target object in the point clouds of the 1st frame to the 10th frame respectively. For this labeling frame sequence L1, labeling frame 1 is selected as the pre-corrected labeling frame. First, labeling frame 1 is used as the reference labeling frame, and labeling frame 2 is used as the current labeling frame to be corrected. The first point cloud in labeling frame 2 is registered with the second point cloud in labeling frame 1 to obtain the pose transformation parameter (R 21 , T 21 ), using the pose transformation parameters (R 21 , T 21 ) transforms the second pose information of the annotation frame 1 to obtain the second pose information of the annotation frame 2, and obtains the second point cloud in the annotation frame 2 according to the second pose information of the annotation frame 2. Then, the annotation frame 2 is used as the new reference annotation frame, and the annotation frame 3 is used as the current annotation frame to be corrected, and the first point cloud in the annotation frame 3 is registered with the second point cloud in the annotation frame 2 to obtain the pose transformation parameter (R) between the annotation frame 3 and the annotation frame 2. 32 , T 32 ), using the pose transformation parameters (R 32 , T 32 ) transform the second pose information of the annotation frame 2 to obtain the second pose information of the annotation frame 3. And so on, until the second pose information of the annotation frame 10 is obtained, so far, the second pose information of all the annotation frames in the annotation frame sequence L1 is obtained.

[0075] In this embodiment, the pose transformation parameters between adjacent annotation frames (including a corrected annotation frame and a to-be-corrected annotation frame) in the annotation frame sequence are used to transform the pose information of the corrected annotation frame to obtain the pose information of the to-be-corrected annotation frame as the corrected pose information. Since there are more overlapping point clouds in adjacent annotation frames, it is beneficial to ensure the accuracy of the point cloud registration results, thereby improving the accuracy of the corrected pose.

[0076] In one embodiment, the step of determining the annotation results of the point cloud sequence to be annotated according to the second pose information of each annotation box in each annotation box sequence may specifically include the following steps: for each annotation box sequence, a pose graph is constructed with the second pose information of each annotation box in the annotation box sequence as a node and the pose transformation parameters of adjacent annotation boxes as edges, and the second pose information of each annotation box in the annotation box sequence is optimized based on the pose graph to obtain the third pose information of each annotation box in the annotation box sequence; and the annotation results of the point cloud sequence to be annotated are obtained according to the third pose information of each annotation box in each annotation box sequence.

[0077] For each sequence of labeled frames, after obtaining the second pose information of each labeled frame, the pose information of the labeled frame can be further optimized using the pose graph, and the third pose information obtained by the pose graph optimization is used as the pose information of the final labeled frame. Through pose graph optimization, the cumulative error in the registration process can be reduced and more accurate pose information can be obtained.

[0078] In the above embodiment, the ICP algorithm and pose graph optimization are used to realize the automatic correction of the pose information of the annotation box, and only a small amount of manual annotation is required, thereby improving the annotation efficiency and having a high annotation accuracy. In addition, the annotated information carries the identification information of the corresponding target object. The data set annotated in this way can be used not only in point cloud target detection, but also in point cloud target tracking.

[0079] In one embodiment, before obtaining the point cloud registration result between corresponding annotation frames according to the first point cloud in each annotation frame in the annotation frame sequence, the following steps may be further included: segmenting the annotation frame sequence to obtain continuous annotation frame subsequences.

[0080] When the annotation box sequence is long, that is, it contains many annotation boxes, the accumulated error in the point cloud registration result will be large, affecting the accuracy of the corrected pose information. In order to reduce the accumulated error, for a long annotation box sequence, it can be divided into multiple continuous annotation box subsequences, and the pose of each annotation box subsequence in the annotation box sequence is corrected separately.

[0081] In one embodiment, according to the first point cloud in each annotation box in the annotation box sequence, the point cloud registration result between the corresponding annotation boxes is obtained, and based on the point cloud registration result, the step of obtaining the second pose information of each annotation box in the annotation box sequence can specifically include the following steps: for each annotation box subsequence in the annotation box sequence, according to the first point cloud in each annotation box in the annotation box subsequence, the point cloud registration result between the corresponding annotation boxes is obtained, and based on the point cloud registration result, the second pose information of each annotation box in the annotation box subsequence is obtained. The specific registration process can refer to the previous embodiment, which will not be repeated here.

[0082] For each annotation box subsequence in the annotation box sequence, after obtaining the second pose information of each annotation box in the annotation box subsequence, the pose information of the annotation box can be further optimized using the pose graph, and the third pose information obtained by the pose graph optimization is used as the pose information of the final annotation of the annotation box. Through pose graph optimization, the cumulative error in the alignment process can be reduced and more accurate pose information can be obtained. After obtaining the third pose information of each annotation box in all annotation box subsequences, it can be merged to obtain the third pose information of each annotation box in the annotation box sequence.

[0083] In one embodiment, the annotation frame sequence is segmented to obtain continuous annotation frame subsequences. Specifically, starting from the first annotation frame in the annotation frame sequence, the annotation frame sequence is segmented once every preset number of annotation frames to obtain continuous annotation frame subsequences.

[0084] The preset number can be set according to actual conditions, for example, it can be set to 10, that is, every 10 consecutive annotation boxes are regarded as a annotation box subsequence. For example, if the annotation box sequence L2 contains 100 annotation boxes, which are annotation box 1, annotation box 2, ..., annotation box 100, the annotation box sequence L2 can be divided into 10 annotation box subsequences, specifically, annotation boxes 1 to 10 form a first annotation box subsequence, annotation boxes 11 to 20 form a second annotation box subsequence, and so on, annotation boxes 91 to 100 form a tenth annotation box subsequence, thereby completing the segmentation of the annotation box sequence.

[0085] It should be understood that, although the steps in each flow chart involved in the above-described embodiment are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in each flow chart involved in the above-described embodiment may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0086] In one embodiment, Figure 5 As shown, a point cloud annotation device 500 is provided, comprising: an acquisition module 510, a pre-annotation module 520, a processing module 530, a registration module 540 and a determination module 550, wherein:

[0087] The acquisition module 510 is used to acquire a point cloud sequence to be labeled.

[0088] The pre-annotation module 520 is used to pre-annotate each frame point cloud in the sequence of point clouds to be annotated, and obtain pre-annotation information of each frame point cloud, wherein the pre-annotation information includes: identification information and first pose information of the annotation box used to mark each target object in the corresponding frame point cloud.

[0089] The processing module 530 is used to determine the annotation box sequence corresponding to each target object according to the identification information, and determine the first point cloud in each annotation box in the annotation box sequence according to the first position information of each annotation box in the annotation box sequence.

[0090] The registration module 540 is used to obtain the point cloud registration result between the corresponding annotation frames according to the first point cloud in each annotation frame in the annotation frame sequence, and obtain the second pose information of each annotation frame in the annotation frame sequence based on the point cloud registration result.

[0091] The determination module 550 is used to determine the labeling result of the to-be-labeled point cloud sequence according to the second pose information of each labeling box in each labeling box sequence.

[0092] In one embodiment, when the processing module 530 determines the labeling box sequence corresponding to each target object based on the identification information, it is specifically used to: determine each labeling box with the same identification information as the labeling box corresponding to the same target object in different frame point clouds; and combine the labeling boxes corresponding to the same target object in different frame point clouds into a labeling box sequence corresponding to the same target object.

[0093] In one embodiment, the pre-annotation information also includes the first size information of the annotation box; the processing module 530 is also used to: for each annotation box sequence, perform the following processing: count the number of point clouds in each annotation box according to the first size information and the first pose information of each annotation box in the annotation box sequence; correct the first size information according to the number of point clouds in each annotation box, and obtain the second size information of each annotation box in the annotation box sequence.

[0094] In one embodiment, when the processing module 530 corrects the first size information according to the number of point clouds in each annotation box and obtains the second size information of each annotation box in the annotation box sequence, it is specifically used to: determine the maximum number of point clouds from the number of point clouds in each annotation box, and obtain the corrected size information of the annotation box corresponding to the maximum number of point clouds; use the corrected size information instead of the first size information as the second size information of each annotation box in the annotation box sequence.

[0095] In one embodiment, when the processing module 530 determines the first point cloud in each annotation box in the annotation box sequence according to the first pose information of each annotation box in the annotation box sequence, it is specifically used to: determine the first point cloud in each annotation box in the annotation box sequence according to the first pose information and the second size information of each annotation box in the annotation box sequence.

[0096] In one embodiment, the registration module 540 obtains point cloud registration results between corresponding annotation frames according to the first point cloud in each annotation frame in the annotation frame sequence, and obtains second pose information of each annotation frame in the annotation frame sequence based on the point cloud registration results. The registration module 540 is specifically used to: determine a pre-corrected annotation frame from the annotation frame sequence, obtain the corrected pose information of the pre-corrected annotation frame as the second pose information of the pre-corrected annotation frame; determine the second point cloud in the pre-corrected annotation frame according to the second pose information of the pre-corrected annotation frame; obtain the point cloud registration results between corresponding annotation frames according to the second point cloud in the pre-corrected annotation frame and the first point cloud in each annotation frame to be corrected, and obtain the second pose information of each annotation frame to be corrected based on the point cloud registration results. The annotation frame to be corrected represents the annotation frame in the annotation frame sequence except the pre-corrected annotation frame.

[0097] In one embodiment, when the registration module 540 determines the pre-corrected annotation frame from the annotation frame sequence, it is specifically used to: use the first annotation frame in the annotation frame sequence as the pre-corrected annotation frame; when the registration module 540 obtains the point cloud registration results between the corresponding annotation frames according to the second point cloud in the pre-corrected annotation frame and the first point cloud in each annotation frame to be corrected, and obtains the second pose information of each annotation frame to be corrected based on the point cloud registration result, it is specifically used to: use the pre-corrected annotation frame as the reference annotation frame, and use the next annotation frame of the reference annotation frame as the current annotation frame to be corrected, compare the first point cloud in the current annotation frame to be corrected with the reference annotation frame, and compare the first point cloud in the current annotation frame to be corrected with the reference annotation frame. The second point cloud in the annotation frame is registered to obtain the point cloud registration result between the current annotation frame to be corrected and the reference annotation frame, and based on the point cloud registration result, the second pose information of the current annotation frame to be corrected is obtained; according to the second pose information of the current annotation frame to be corrected, the second point cloud in the current annotation frame to be corrected is obtained; the current annotation frame to be corrected is used as a new reference annotation frame, and the next annotation frame after the reference annotation frame is returned as the current annotation frame to be corrected, and the first point cloud in the current annotation frame to be corrected is registered with the second point cloud in the reference annotation frame, until the second pose information of all the annotation frames to be corrected in the annotation frame sequence is obtained.

[0098] In one embodiment, the point cloud registration result between the current annotation frame to be corrected and the reference annotation frame includes: the posture transformation parameters of the current annotation frame to be corrected relative to the reference annotation frame; when the registration module 540 obtains the second posture information of the current annotation frame to be corrected based on the point cloud registration result, it is specifically used to: transform the second posture information of the reference annotation frame using the posture transformation parameters to obtain the second posture information of the current annotation frame to be corrected.

[0099] In one embodiment, the posture information includes: the coordinates and orientation of the center point of the annotation box, and the posture transformation parameters include: rotation parameters and translation parameters.

[0100] In one embodiment, when determining the annotation results of the point cloud sequence to be annotated based on the second pose information of each annotation box in each annotation box sequence, the determination module 550 is specifically used to: for each annotation box sequence, construct a pose graph with the second pose information of each annotation box in the annotation box sequence as a node and the pose transformation parameters of adjacent annotation boxes as edges, optimize the second pose information of each annotation box in the annotation box sequence based on the pose graph, and obtain the third pose information of each annotation box in the annotation box sequence; obtain the annotation results of the point cloud sequence to be annotated based on the third pose information of each annotation box in each annotation box sequence.

[0101] In one embodiment, the processing module 530 is further configured to segment the annotation box sequence to obtain continuous annotation box subsequences.

[0102] In one embodiment, when the processing module 530 segments the annotation box sequence to obtain continuous annotation box subsequences, it is specifically configured to: starting from the first annotation box in the annotation box sequence, segment the annotation box sequence once every preset number of annotation boxes to obtain continuous annotation box subsequences.

[0103] In one embodiment, the preset number is ten.

[0104] In one embodiment, the registration module 540 obtains the point cloud registration result between corresponding annotation frames according to the first point cloud in each annotation frame in the annotation frame sequence, and obtains the second pose information of each annotation frame in the annotation frame sequence based on the point cloud registration result. Specifically, it is used to: for each annotation frame subsequence of the annotation frame sequence, obtain the point cloud registration result between corresponding annotation frames according to the first point cloud in each annotation frame in the annotation frame subsequence, and obtain the second pose information of each annotation frame in the annotation frame subsequence based on the point cloud registration result.

[0105] For the specific definition of the point cloud annotation device, please refer to the definition of the point cloud annotation method above, which will not be repeated here. Each module in the above-mentioned point cloud annotation device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0106] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.

[0107] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0108] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned various method embodiments.

[0109] It should be understood that the terms "first", "second", etc. in the above embodiments are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. In addition, in the description of this application, unless otherwise specified, the meaning of "multiple" refers to at least two.

[0110] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0111] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A point cloud annotation method, characterized in that: The method comprises: Get the point cloud sequence to be annotated; Pre-annotating each frame point cloud in the sequence of point clouds to be annotated to obtain pre-annotation information of each frame point cloud, wherein the pre-annotation information includes: identification information and first position information of an annotation box for marking each target object in the corresponding frame point cloud; Determine a labeling frame sequence corresponding to each target object according to the identification information, and determine a first point cloud in each labeling frame in the labeling frame sequence according to the first position information of each labeling frame in the labeling frame sequence; According to the first point cloud in each annotation frame in the annotation frame sequence, obtaining a point cloud registration result between corresponding annotation frames, and based on the point cloud registration result, obtaining second pose information of each annotation frame in the annotation frame sequence; For each marked frame sequence, a pose graph is constructed by taking the second pose information of each marked frame in the marked frame sequence as a node and the pose transformation parameters of adjacent marked frames as an edge, and the second pose information of each marked frame in the marked frame sequence is optimized based on the pose graph to obtain the third pose information of each marked frame in the marked frame sequence; According to the third pose information of each annotation box in each annotation box sequence, the annotation result of the to-be-annotated point cloud sequence is obtained.

2. The method according to claim 1, characterized in that Determining a labeling box sequence corresponding to each target object according to the identification information includes: Determine each annotation box with the same identification information as the annotation box corresponding to the same target object in different frame point clouds; The annotation boxes corresponding to the same target object in different frame point clouds are combined into a annotation box sequence corresponding to the same target object.

3. The method according to claim 1, characterized in that The pre-marking information also includes first size information of the marking box; the method also includes: For each label box sequence, the following processing is performed: According to the first size information and the first pose information of each annotation box in the annotation box sequence, counting the number of point clouds in each annotation box; The first size information is corrected according to the number of point clouds in each annotation box, and the second size information of each annotation box in the annotation box sequence is obtained.

4. The method according to claim 3, characterized in that According to the number of point clouds in each annotation box, the first size information is corrected to obtain the second size information of each annotation box in the annotation box sequence, including: Determine the maximum number of point clouds from the number of point clouds in each annotation box, and obtain the corrected size information of the annotation box corresponding to the maximum number of point clouds; The first size information is replaced by the revised size information as the second size information of each annotation box in the annotation box sequence.

5. The method according to claim 3, characterized in that: Determining a first point cloud in each of the annotation boxes in the annotation box sequence according to the first position information of each of the annotation boxes in the annotation box sequence includes: A first point cloud in each of the annotation boxes in the annotation box sequence is determined according to the first pose information and the second size information of each of the annotation boxes in the annotation box sequence.

6. The method according to claim 1, characterized in that According to the first point cloud in each annotation frame in the annotation frame sequence, obtaining a point cloud registration result between corresponding annotation frames, and based on the point cloud registration result, obtaining second pose information of each annotation frame in the annotation frame sequence, including: Determine a pre-corrected annotation frame from the annotation frame sequence, and obtain corrected position and posture information of the pre-corrected annotation frame as second position and posture information of the pre-corrected annotation frame; Determining a second point cloud within the pre-corrected annotation frame according to the second pose information of the pre-corrected annotation frame; According to the second point cloud in the pre-corrected annotation frame and the first point cloud in each annotation frame to be corrected, a point cloud alignment result between corresponding annotation frames is obtained, and based on the point cloud alignment result, second pose information of each annotation frame to be corrected is obtained, wherein the annotation frame to be corrected represents the annotation frame in the annotation frame sequence except the pre-corrected annotation frame.

7. The method according to claim 6, characterized in that Determining a pre-corrected annotation frame from the annotation frame sequence includes: taking the first annotation frame in the annotation frame sequence as the pre-corrected annotation frame; According to the second point cloud in the pre-corrected annotation frame and the first point cloud in each of the annotation frames to be corrected, a point cloud registration result between the corresponding annotation frames is obtained, and based on the point cloud registration result, second pose information of each of the annotation frames to be corrected is obtained, including: Taking the pre-corrected annotation frame as a reference annotation frame, taking the next annotation frame of the reference annotation frame as a current annotation frame to be corrected, registering a first point cloud in the current annotation frame to be corrected with a second point cloud in the reference annotation frame, obtaining a point cloud registration result between the current annotation frame to be corrected and the reference annotation frame, and obtaining second pose information of the current annotation frame to be corrected based on the point cloud registration result; Obtaining a second point cloud within the current marked frame to be corrected according to the second pose information of the current marked frame to be corrected; The step of taking the current annotation frame to be corrected as a new reference annotation frame, returning to take the next annotation frame of the reference annotation frame as the current annotation frame to be corrected, and aligning the first point cloud in the current annotation frame to be corrected with the second point cloud in the reference annotation frame until the second pose information of all the annotation frames to be corrected in the annotation frame sequence is obtained.

8. The method according to claim 7, characterized in that The point cloud registration result between the current annotation frame to be corrected and the reference annotation frame includes: the posture transformation parameters of the current annotation frame to be corrected relative to the reference annotation frame; Based on the point cloud registration result, obtaining the second pose information of the current annotation frame to be corrected includes: The second pose information of the reference annotation frame is transformed using the pose transformation parameters to obtain the second pose information of the current annotation frame to be corrected.

9. The method according to claim 8, characterized in that The posture information includes: the coordinates and orientation of the center point of the annotation box, and the posture transformation parameters include: rotation parameters and translation parameters.

10. The method according to claim 1, characterized in that Before obtaining the point cloud registration result between corresponding annotation frames according to the first point cloud in each annotation frame in the annotation frame sequence, the method further includes: segmenting the annotation frame sequence to obtain a continuous annotation frame subsequence.

11. The method according to claim 10, characterized in that Segmenting the annotation box sequence to obtain continuous annotation box subsequences includes: Starting from the first marked frame in the marked frame sequence, the marked frame sequence is segmented once every preset number of marked frames to obtain a continuous marked frame subsequence.

12. The method according to claim 11, characterized in that The preset number is 10.

13. The method according to any one of claims 10 to 12, characterized in that According to the first point cloud in each annotation frame in the annotation frame sequence, obtaining a point cloud registration result between corresponding annotation frames, and based on the point cloud registration result, obtaining second pose information of each annotation frame in the annotation frame sequence, including: For each annotation box subsequence of the annotation box sequence, a point cloud registration result between corresponding annotation boxes is obtained according to the first point cloud in each annotation box in the annotation box subsequence, and second pose information of each annotation box in the annotation box subsequence is obtained based on the point cloud registration result.

14. A point cloud annotation device, characterized in that: The device comprises: An acquisition module is used to obtain a point cloud sequence to be annotated; A pre-annotation module is used to pre-annotate each frame point cloud in the sequence of point clouds to be annotated, and obtain pre-annotation information of each frame point cloud, wherein the pre-annotation information includes: identification information and first pose information of the annotation box for marking each target object in the corresponding frame point cloud; A processing module, configured to determine a sequence of annotation boxes corresponding to each target object according to the identification information, and determine a first point cloud in each annotation box in the sequence of annotation boxes according to the first position information of each annotation box in the sequence of annotation boxes; A registration module, configured to obtain a point cloud registration result between corresponding annotation frames according to a first point cloud in each annotation frame in the annotation frame sequence, and obtain second pose information of each annotation frame in the annotation frame sequence based on the point cloud registration result; A determination module is used to construct a pose graph for each annotation box sequence, using the second pose information of each annotation box in the annotation box sequence as a node and the pose transformation parameters of adjacent annotation boxes as edges, optimize the second pose information of each annotation box in the annotation box sequence based on the pose graph, obtain the third pose information of each annotation box in the annotation box sequence, and obtain the annotation result of the point cloud sequence to be annotated according to the third pose information of each annotation box in the annotation box sequence.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images