Point cloud data labeling method, system and device, medium, product and vehicle

Abstract

The embodiment of the invention discloses a point cloud data labeling method, system and device, a medium, a product and a vehicle, and relates to the technical field of data processing, the method comprises the following steps: dividing continuously collected M frames of point cloud data to obtain N point cloud sets, M and N being integers greater than 1; labeling a three-dimensional frame of each frame of point cloud data in the M frames of point cloud data to obtain pre-labeling information of each frame of point cloud data; according to the pre-labeling information and the pose information of each point cloud data in each point cloud set, merging at least a part of frames of point cloud data in each point cloud set to obtain N frames of first point cloud data corresponding to the N point cloud sets; and performing semantic segmentation on the N frames of first point cloud data to obtain semantic annotation information of each frame of first point cloud data. According to the method and the device, the three-dimensional frame of the point cloud data is labeled, and semantic segmentation is performed on the merged point cloud data to determine the semantic labeling information of the point cloud data, so that the labeling efficiency of the point cloud data can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, system, device, medium, product and vehicle for labeling point cloud data. Background Art

[0002] Point cloud data annotation is the process of annotating and describing point cloud data in three-dimensional space. It can be applied to fields such as autonomous driving, robot and drone navigation, architecture, medicine, and virtual and reality. For example, in the field of autonomous driving, autonomous vehicles need to rely on point cloud data annotation to identify key information such as roads, lanes, traffic signals, obstacles, etc., so as to accurately plan driving paths or make decisions such as deceleration and avoidance in advance. It can be seen that the importance of point cloud data annotation is self-evident. However, the annotation efficiency of point cloud data annotation methods in related technologies is low.

[0003] Application Contents

[0004] In view of this, one of the objects of the present application is to provide a point cloud data annotation method, system, device, medium, product and vehicle, which can improve the annotation efficiency of point cloud data.

[0005] To achieve the above purpose, the technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for labeling point cloud data, including:

[0007] The M frames of point cloud data collected continuously are divided and processed to obtain N point cloud sets, each of which includes several frames of point cloud data, where M and N are both integers greater than 1;

[0008] Annotate the three-dimensional frame of each frame of point cloud data in the M frames of point cloud data to obtain pre-annotated information of each frame of point cloud data;

[0009] Merging at least part of the frame point cloud data in each point cloud set according to the pre-annotated information and the position and posture information of each point cloud data in each point cloud set to obtain N frames of first point cloud data corresponding to the N point cloud sets;

[0010] Semantic segmentation is performed on the first point cloud data of N frames to obtain semantic annotation information of the first point cloud data of each frame.

[0011] In a possible implementation, at least part of the frame point cloud data in each point cloud set is merged according to the pre-annotated information and the pose information of each point cloud data in each point cloud set to obtain N frames of first point cloud data corresponding to the N point cloud sets, including:

[0012] Determine first intermediate frame point cloud data in at least part of the frame point cloud data in each point cloud set;

[0013] Convert each point cloud data into a world coordinate system according to pre-annotated information and pose information of each point cloud data in at least part of the frame point cloud data;

[0014] In the world coordinate system, at least part of the frame point cloud data except the first intermediate frame point cloud data is mapped to the first intermediate frame point cloud data to obtain N frames of first point cloud data corresponding to the N point cloud sets.

[0015] In a possible implementation, the pre-labeling information includes first information for indicating a static object in the corresponding point cloud data;

[0016] In the world coordinate system, mapping other frame point cloud data except the first intermediate frame point cloud data in at least a portion of the frame point cloud data onto the first intermediate frame point cloud data includes:

[0017] According to the first information of at least part of the frame point cloud data, in the world coordinate system, static objects of other frame point cloud data except the first intermediate frame point cloud data in at least part of the frame point cloud data are mapped onto the static objects of the first intermediate frame point cloud data.

[0018] In a possible implementation, the pre-labeled information includes second information for indicating a dynamic object in the corresponding point cloud data;

[0019] In the world coordinate system, mapping other frame point cloud data except the first intermediate frame point cloud data in at least a portion of the frame point cloud data onto the first intermediate frame point cloud data includes:

[0020] According to the second information of at least part of the frame point cloud data, in the world coordinate system, the dynamic objects of the frame point cloud data other than the first intermediate frame point cloud data in at least part of the frame point cloud data are mapped onto the dynamic objects of the first intermediate frame point cloud data.

[0021] In a possible implementation, after semantic segmentation is performed on N frames of first point cloud data to obtain semantic annotation information of each frame of first point cloud data, the method further includes:

[0022] Determine the second intermediate frame point cloud data of the N frames of first point cloud data;

[0023] According to the pre-annotated information and the pose information of each first point cloud data in the N frames of first point cloud data, each first point cloud data is converted into a world coordinate system;

[0024] In the world coordinate system, the first point cloud data of other frames except the second intermediate frame point cloud data in the N frames of the first point cloud data are mapped to the second intermediate frame point cloud data to obtain a frame of the second point cloud data corresponding to the N frames of the first point cloud data, wherein the point cloud in the second point cloud data includes a relationship identifier, and the relationship identifier is used to indicate the frame of the first point cloud data before the point cloud is merged;

[0025] Obtaining a verification result of the semantic annotation information of the second point cloud data;

[0026] When the verification result indicates that the accuracy of the semantic annotation information is greater than or equal to the preset accuracy threshold, all point clouds in the second point cloud data are mapped to the first point cloud data of the corresponding frame according to the relationship identifiers of each point cloud in the second point cloud data.

[0027] In a possible implementation, semantic segmentation is performed on N frames of first point cloud data to obtain semantic annotation information of each frame of first point cloud data, including:

[0028] When deduplication processing is performed on N frames of first point cloud data, semantic segmentation is performed on all frames of first point cloud data after deduplication processing to determine semantic annotation information of the first point cloud data of each frame.

[0029] In a possible implementation, performing deduplication processing on N frames of first point cloud data includes:

[0030] Rasterize the space where the first point cloud data of the N frames is located to obtain multiple grids;

[0031] The point cloud data in the target grid is deleted until the number of point clouds in the target grid is less than a preset number threshold, and the target grid is a grid in which the number of point clouds in multiple grids is greater than or equal to the preset number threshold.

[0032] In a possible implementation, the M frames of continuously collected point cloud data are divided and processed to obtain N point cloud sets, including:

[0033] Get the acquisition timestamp of M frames of point cloud data;

[0034] The M frames of point cloud data are divided and processed according to the acquisition timestamp to obtain N point cloud sets.

[0035] In a second aspect, an embodiment of the present application provides a point cloud data annotation system, the system comprising:

[0036] A division module is used to divide and process the M frames of point cloud data collected continuously to obtain N point cloud sets, each of which includes several frames of point cloud data, where M and N are both integers greater than 1;

[0037] The first annotation module is used to annotate the three-dimensional frame of each frame of point cloud data in the M frames of point cloud data to obtain pre-annotated information of each frame of point cloud data;

[0038] A merging module, configured to merge at least part of the frame point cloud data in each point cloud set according to the pre-annotated information and the position and posture information of each point cloud data in each point cloud set, to obtain N frames of first point cloud data corresponding to the N point cloud sets;

[0039] The second annotation module is used to perform semantic segmentation on the N frames of first point cloud data to obtain semantic annotation information of the first point cloud data of each frame.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, the point cloud data labeling method provided in the first aspect is implemented.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by one or more processors, the point cloud data annotation method provided in the first aspect is implemented.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the point cloud data labeling method provided in the first aspect.

[0043] In a sixth aspect, an embodiment of the present application provides a vehicle, the vehicle comprising at least one of the following:

[0044] Such as the annotation system of point cloud data in the second aspect;

[0045] Such as the electronic device of the third aspect;

[0046] A computer-readable storage medium as in the fourth aspect.

[0047] The embodiment of the present application provides a method for labeling point cloud data, which obtains N point cloud sets by dividing and processing M frames of continuously collected point cloud data. Pre-labeling information of each frame of point cloud data is obtained by labeling the three-dimensional frame of each frame of point cloud data in the M frames of point cloud data, and at least part of the frame point cloud data in each point cloud set is merged according to the pre-labeling information and posture information of each point cloud data in each point cloud set to obtain N frames of first point cloud data corresponding to N point cloud sets. Finally, semantic segmentation is performed on the N frames of first point cloud data to obtain semantic labeling information of each frame of first point cloud data. The present application can improve the labeling efficiency of point cloud data by labeling the three-dimensional frame of point cloud data and semantic segmenting the merged point cloud data to determine the semantic labeling information of point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. It should be understood that the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A flowchart of a method for labeling point cloud data provided in an embodiment of the present application;

[0050] Figure 2 A flowchart of a merging method included in a method for annotating point cloud data provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of functional modules of a point cloud data annotation system provided in an embodiment of the present application;

[0052] Figure 4 A diagram of the internal structure of an electronic device provided in an embodiment of the present application.

[0053] Description of reference numerals:

[0054] The point cloud data annotation system 300 includes a division module 310 , a first annotation module 320 , a merging module 330 , and a second annotation module 340 . DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0057] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0058] In various embodiments of the present application, the expression "or" or "at least one of A or / and B" includes any combination or all combinations of the words listed at the same time. For example, the expression "A or B" or "at least one of A or / and B" may include A, may include B, or may include both A and B.

[0059] In the description of the present application, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, it is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship in which the product of the invention is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0060] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0061] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0062] Furthermore, in the embodiments of the present application, the term "connection" may refer to "electrical connection" or "direct connection". "Electrical connection" may refer to direct electrical connection between two components or electrical connection between two components via one or more normally open tubes or other components.

[0063] To facilitate a better understanding of the solutions of the embodiments of the present application, the relevant technologies are first introduced below.

[0064] Point cloud annotation is the process of describing and labeling objects and scenes in three-dimensional space. Through annotation, objects, structures and features in the real world can be abstracted into digital point cloud data, which is convenient for computer processing and analysis. Point cloud annotation provides important training data for machine learning and deep learning algorithms. By annotating point cloud data, the computer system can establish the ability to recognize three-dimensional objects and scenes, thereby realizing automatic recognition, classification and analysis.

[0065] Point cloud is a data set consisting of a series of points in three-dimensional space. Each point usually contains three-dimensional coordinate information and may also include other attribute information such as color and normal. Semantic segmentation of point cloud is to determine the semantic category of each point based on the geometric features, spatial relationships and other related information of the point. For example, in practical applications, point clouds are divided into different categories such as buildings, roads, trees, vehicles, etc.

[0066] In order to solve the technical problems in the background technology, the embodiments of the present application provide a method, system, device, medium, product and vehicle for annotating point cloud data. The method for annotating point cloud data provided by the embodiments of the present application is first introduced below.

[0067] See also Figure 1 , Figure 1 A flowchart of a method for annotating point cloud data provided in an embodiment of the present application. The method for annotating point cloud data can be applied to a point cloud data annotation system, electronic device, and vehicle in the following embodiments, wherein the electronic device includes a personal computer, a server, a mobile device, a cloud computing platform, and a supercomputer, etc. The method for annotating point cloud data will be introduced from the perspective of application to electronic devices. The method for annotating point cloud data specifically includes the following steps:

[0068] Step 110 , dividing and processing the M frames of point cloud data collected continuously to obtain N point cloud sets, each point cloud set includes a number of frames of point cloud data, wherein M and N are both integers greater than 1.

[0069] Step 120 , annotate the three-dimensional frame of each frame of point cloud data in the M frames of point cloud data to obtain pre-annotated information of each frame of point cloud data.

[0070] Step 130 , based on the pre-labeled information and the pose information of each point cloud data in each point cloud set, merge at least part of the frame point cloud data in each point cloud set to obtain N frames of first point cloud data corresponding to the N point cloud sets.

[0071] Step 140 , performing semantic segmentation on the N frames of first point cloud data to obtain semantic annotation information of the first point cloud data of each frame.

[0072] The embodiment of the present application provides a method for labeling point cloud data, which obtains N point cloud sets by dividing and processing M frames of continuously collected point cloud data. Pre-labeling information of each frame of point cloud data is obtained by labeling the three-dimensional frame of each frame of point cloud data in the M frames of point cloud data, and at least part of the frame point cloud data in each point cloud set is merged according to the pre-labeling information and posture information of each point cloud data in each point cloud set to obtain N frames of first point cloud data corresponding to N point cloud sets. Finally, semantic segmentation is performed on the N frames of first point cloud data to obtain semantic labeling information of each frame of first point cloud data. The present application can improve the labeling efficiency of point cloud data by labeling the three-dimensional frame of point cloud data and semantic segmenting the merged point cloud data to determine the semantic labeling information of point cloud data.

[0073] The following will be Figure 1 Each step of the method is described in detail.

[0074] In step S110, the electronic device may divide and process the M frames of point cloud data collected continuously, and the electronic device may be a vehicle-mounted device. In some embodiments, the electronic device may receive in real time the M frames of point cloud data collected continuously by the point cloud collection device, and the point cloud collection device includes a laser radar or a depth camera, and the laser radar may be an airborne laser radar such as installed on an aircraft or a drone, a ground laser radar, or a vehicle-mounted laser radar such as installed on a vehicle, and may obtain point cloud data on both sides of the road and in front as the vehicle moves.

[0075] Taking the use of LiDAR to collect point cloud data as an example, one frame of point cloud data can be understood as a collection of point clouds formed by all laser beams surrounding a circle.

[0076] M frames of point cloud data are collected continuously by electronic devices. M frames of point cloud data can be regarded as M consecutive frames of point cloud data, such as 10 consecutive frames of point cloud data within 1 second. The number of frames of point cloud data is affected by the continuous collection time and the point cloud collection frequency. For example, the point cloud collection frequency of the vehicle-mounted laser radar is 10Hz, which means that the vehicle-mounted laser radar can collect 10 frames of point cloud data per second. The vehicle-mounted laser radar can collect 100 frames of point cloud data with a continuous collection time of 10 seconds.

[0077] Each point cloud set among the N point cloud sets is obtained by dividing and processing M frames of point cloud data by the electronic device. Each point cloud set may contain several frames of point cloud data, and the several frames of point cloud data contained in each point cloud set are all continuous frames of point cloud data.

[0078] In some embodiments, the electronic device can directly divide the point cloud data according to preset division parameters when acquiring the point cloud data. The preset division parameters include time parameters such as a time interval of 1 second.

[0079] In some embodiments, the electronic device may perform segmentation processing on the M frames of point cloud data collected continuously upon receiving a segmentation instruction. The segmentation instruction may include segmentation parameters such as a fixed time interval. Upon receiving the segmentation instruction, the electronic device may perform segmentation processing on the M frames of point cloud data according to the time interval in the segmentation instruction, such as 1 second.

[0080] The frames in each of the N point cloud sets may be the same or different.

[0081] Taking the example of a vehicle-mounted laser radar collecting data continuously for 10 seconds at a point cloud collection frequency of 10 Hz, 100 frames of point cloud data can be obtained. The electronic device divides and processes the 100 frames of point cloud data according to the division parameters in the division instruction, that is, the time interval such as 1 second, and 10 point cloud sets can be obtained, and each point cloud set includes 10 frames of point cloud data.

[0082] In some embodiments, the partitioning parameter in the partitioning instruction may also be a non-fixed time interval sequence, such as cyclic time intervals of 1 second and 2 seconds.

[0083] Taking the example of a vehicle-mounted laser radar collecting point cloud data continuously for 10 seconds at a point cloud collection frequency of 10 Hz, 100 frames of point cloud data can be obtained. The electronic device can obtain 6 point cloud sets based on the division parameters in the division instruction, that is, an unfixed time interval sequence of 1 second and 2 seconds. The 6 point cloud sets are arranged in time, including 10 frames, 20 frames, 10 frames, 20 frames, 10 frames, 20 frames and 10 frames of point cloud data.

[0084] In step S120, the pre-labeling information includes a three-dimensional frame of the point cloud data. The electronic device can use a three-dimensional object detection model to determine the three-dimensional frame of each frame of point cloud data and label it to obtain the pre-labeling information of each frame of point cloud data.

[0085] In some embodiments, the electronic device can obtain the acquisition timestamp corresponding to each frame of point cloud data in M ​​frames of point cloud data. The acquisition timestamp can be generated by the clock of the point cloud acquisition device such as the vehicle-mounted laser radar itself. When the electronic device uses a three-dimensional object detection model for annotation, it can be annotated in the form of a generated time window.

[0086] Exemplarily, when the electronic device annotates the three-dimensional boxes of each frame of point cloud data in M ​​frames, it simultaneously checks whether there are other frames of point cloud data that need to be annotated within the current time window. If so, the other frames can be put into the annotation queue. For example, if the time window is set to ±10 milliseconds, the point clouds in the frames within the range of ±10 milliseconds can be regarded as being annotated simultaneously.

[0087] In some embodiments, the three-dimensional object detection model includes any one of a VoxelNet model, a MV3D (Multi-View3D) model, and an AVOD (Aggregate View Object Detection) model.

[0088] In step S130, the pose information of the point cloud data can be represented by px, py, pz, ow, ox, oy and oz. px, py and pz represent the coordinate components of the point cloud P on the x-axis, y-axis and z-axis. ow, ox, oy and oz can represent the quaternion information of the acquisition viewpoint of the point cloud data, that is, the rotation information when converting from the world coordinate system to the vehicle-mounted laser radar coordinate system, ow represents the real part of the quaternion, and ox, oy and oz represent the imaginary part of the quaternion.

[0089] The electronic device determines the 4*4 transformation matrix Mi according to the posture information of the above point cloud data, and determines the world coordinate system Pi_w of the point cloud Pi: Pi_w=Pi@MiT, @ represents matrix multiplication, T represents matrix transpose, and according to the world coordinate system Pi_w of the point cloud Pi, the point cloud pi can be converted to the world coordinate system to realize the merging of the point clouds.

[0090] In some embodiments, the electronic device may select a portion of the frame point cloud data in a point cloud set for merging, or may select all the frame point cloud data in a point cloud set for merging. For example, if a point cloud set includes 10 frames of point cloud data, the electronic device may randomly select 5, 6 or 7 frames of point cloud data therein for merging, or may select all 10 frames of point cloud data for merging. The electronic device may select a portion of the frame point cloud data or all the frame point cloud data in a point cloud set for merging, which may be selected according to actual needs, and this embodiment is not limited here.

[0091] In step S140, the electronic device may use the trained semantic segmentation model to perform semantic segmentation on the N frames of first point cloud data, and determine the semantic annotation information of each frame of the first point cloud data.

[0092] In some embodiments, the trained semantic segmentation model includes any one of the FCN (Fully Convolutional Network) model, Patchwork++ and SegNet model, which is not limited here.

[0093] The three-dimensional physical detection model in the above embodiment can quickly identify objects in point cloud data and determine their position and approximate category in three-dimensional space. On this basis, the semantic segmentation model can further finely divide and label the various parts of the object, thereby providing more accurate and detailed initial labeling results for the labelers. For example, in an autonomous driving scenario, the three-dimensional object detection model can first detect the position of objects such as vehicles and pedestrians, and the semantic segmentation will further subdivide the vehicle into parts such as the body, windows, and wheels. In the subsequent operation process, the labelers only need to review and correct these preliminary annotations, which significantly reduces the workload of labeling from scratch.

[0094] When processing complex point cloud scenes, such as those with occlusion, overlap, or multi-object interaction, the 3D object detection model can use its understanding of the overall structure and spatial relationships of the object to assist semantic segmentation in better handling these complex situations. Even if part of the object is occluded, the 3D object detection model can make inferences based on contextual information and the overall features of the object, providing more reliable clues for semantic segmentation, thereby improving the accuracy and efficiency of annotation and reducing the difficulty and error of annotation caused by complex scenes.

[0095] Moreover, the semantic segmentation in the present embodiment is to perform semantic segmentation on N frames of first point cloud data corresponding to N point cloud sets that merge several frames of point cloud data. The number of frames of point cloud data that need to be processed for semantic segmentation is less than that of directly performing semantic segmentation on each frame of collected point cloud data in related technologies, which can improve the efficiency of semantic segmentation and obtaining semantic annotation information of the first point cloud data of each frame.

[0096] At the same time, reducing the number of frames of point cloud data during semantic segmentation processing can also reduce the processing cost of semantic segmentation.

[0097] See also Figure 2 , Figure 2 A flowchart of a merging method included in a point cloud data annotation method provided in an embodiment of the present application.

[0098] In a possible implementation, at least part of the frame point cloud data in each point cloud set is merged according to the pre-annotated information and the pose information of each point cloud data in each point cloud set to obtain N frames of first point cloud data corresponding to the N point cloud sets, including:

[0099] Step 210 , determining first intermediate frame point cloud data in at least a portion of the frame point cloud data in each point cloud set.

[0100] Step 220 , converting each point cloud data into a world coordinate system according to the pre-labeled information and the pose information of each point cloud data in at least a portion of the frame point cloud data.

[0101] Step 230, in the world coordinate system, map at least part of the frame point cloud data except the first intermediate frame point cloud data to the first intermediate frame point cloud data to obtain N frames of first point cloud data corresponding to the N point cloud sets.

[0102] The embodiment of the present application determines the first intermediate frame point cloud data, and maps at least part of the frame point cloud data that needs to be merged to the first intermediate frame point cloud data, thereby avoiding the ghosting phenomenon caused by the existence of dynamic point clouds during the point cloud merging process and further improving the annotation efficiency of the point cloud data.

[0103] Exemplarily, if a point cloud set includes 10 consecutive frames of point cloud data, the first intermediate frame point cloud data is the 5th or 6th frame point cloud data among the 10 consecutive frames of point cloud data; if a point cloud set includes 11 consecutive frames of point cloud data, the first intermediate frame point cloud data is the 6th frame point cloud data among the 11 consecutive frames of point cloud data.

[0104] According to the pre-annotated information and pose information of each point cloud data in at least part of the frame point cloud data, each point cloud data is converted to a world coordinate system, including the following two cases:

[0105] The first case: According to the pre-annotated information and pose information of each point cloud data in each point cloud set, each point cloud data is converted to the world coordinate system. For example, if a point cloud set includes 10 consecutive frames of point cloud data, the 1st, 2nd, 3rd, 8th, 9th and 10th frames are selected for merging.

[0106] The second scenario: According to the pre-expression information and pose information of each point cloud data in all the point cloud data in each point cloud set, each point cloud data is converted to the world coordinate system. For example, if a point cloud set includes 10 consecutive frames of point cloud data, all 10 frames of point cloud data are selected for merging.

[0107] According to the posture information, the point cloud data is converted into the world coordinate system. Please refer to the introduction of the above-mentioned embodiment, which will not be repeated here. It should be noted that this embodiment combines the posture information of the point cloud data and the pre-annotation information of the point cloud data to convert the point cloud data into the world coordinate system. Combining the pre-annotation information can avoid converting objects with the same pre-annotation information to different positions during the point cloud data conversion process, thereby increasing the situation of multiple marking of the same marked object.

[0108] In a possible implementation, the pre-labeling information includes first information for indicating a static object in the corresponding point cloud data;

[0109] In the world coordinate system, mapping other frame point cloud data except the first intermediate frame point cloud data in at least a portion of the frame point cloud data onto the first intermediate frame point cloud data includes:

[0110] According to the first information of at least part of the frame point cloud data, in the world coordinate system, static objects of other frame point cloud data except the first intermediate frame point cloud data in at least part of the frame point cloud data are mapped onto the static objects of the first intermediate frame point cloud data.

[0111] The embodiment of the present application uses the first information of static objects in the static point cloud in the point cloud data to map the static objects of at least part of the frame point cloud data except the first intermediate frame point cloud data to the static objects of the first intermediate frame point cloud data, thereby realizing the merging of the static objects in the point cloud data.

[0112] Specifically, the first information includes identification information of a three-dimensional frame of the static object.

[0113] In a possible implementation, the pre-labeled information includes second information for indicating a dynamic object in the corresponding point cloud data;

[0114] In the world coordinate system, mapping other frame point cloud data except the first intermediate frame point cloud data in at least a portion of the frame point cloud data onto the first intermediate frame point cloud data includes:

[0115] According to the second information of at least part of the frame point cloud data, in the world coordinate system, the dynamic objects of the frame point cloud data other than the intermediate frame point cloud data in at least part of the frame point cloud data are mapped onto the dynamic objects of the intermediate frame point cloud data.

[0116] The embodiment of the present application maps the dynamic objects of at least part of the frame point cloud data except the first intermediate frame point cloud data to the static objects of the first intermediate frame point cloud data through the second information of the dynamic objects in the dynamic point cloud in the point cloud data, thereby realizing the merging of the dynamic objects in the point cloud data.

[0117] Specifically, the second information includes the identification information of the three-dimensional frame of the dynamic object, as well as the center point of the three-dimensional frame and the angle information of the three-dimensional frame. During the process of merging dynamic objects, the electronic device needs to compare the center point of the three-dimensional frame and the angle information of the three-dimensional frame with the same identification information to avoid incorrect merging of dynamic objects in the point cloud data.

[0118] In a possible implementation, after semantic segmentation is performed on N frames of first point cloud data to obtain semantic annotation information of each frame of first point cloud data, the method further includes:

[0119] Determine the second intermediate frame point cloud data of the N frames of first point cloud data;

[0120] According to the pre-annotated information and the pose information of each first point cloud data in the N frames of first point cloud data, each first point cloud data is converted into a world coordinate system;

[0121] In the world coordinate system, the first point cloud data of other frames except the second intermediate frame point cloud data in the N frames of the first point cloud data are mapped to the second intermediate frame point cloud data to obtain a frame of the second point cloud data corresponding to the N frames of the first point cloud data, wherein the point cloud in the second point cloud data includes a relationship identifier, and the relationship identifier is used to indicate the frame of the first point cloud data before the point cloud is merged;

[0122] Obtaining a verification result of the semantic annotation information of the second point cloud data;

[0123] When the verification result indicates that the accuracy of the semantic annotation information is greater than or equal to the preset accuracy threshold, all point clouds in the second point cloud data are mapped to the first point cloud data of the corresponding frame according to the relationship identifiers of each point cloud in the second point cloud data.

[0124] The embodiment of the present application can merge the first point cloud data of all frames after removing the semantic annotation information of the first point cloud data of each frame to obtain a frame of second point cloud data. In the process of verifying the semantic annotation information of the point cloud data, only one frame of second point cloud data needs to be verified, thereby reducing the verification cost under manual verification and improving the efficiency of verification.

[0125] Specifically, the second intermediate frame point cloud data is similar to the first intermediate frame point cloud data in the aforementioned embodiment. The merging process of N frames of first point cloud data is implemented according to the second intermediate frame point cloud data. Please refer to the merging process of at least part of the frame point cloud data in any point cloud set according to the first intermediate frame point cloud data in the aforementioned embodiment, which will not be repeated here.

[0126] In some embodiments, the verification result may be determined through manual verification, such as determining the accuracy of the semantic annotation information based on the number of errors in the manually verified semantic annotation information.

[0127] In some embodiments, the verification result can be completed by the electronic device itself. For example, the electronic device can verify the number of errors in the semantic annotation information with the help of the pre-annotation information determined by the three-dimensional object detection model in the aforementioned embodiment and the semantic annotation information determined by the semantic segmentation model, and then determine the accuracy of the semantic annotation information. Specifically, for a certain point cloud data Pi, the electronic device can obtain the semantic annotation information of the point cloud data Pi and the annotation information of the three-dimensional box of the point cloud data Pi. If the two information are the same, the semantic annotation information of the point cloud data Pi is accurate, otherwise it means that the semantic annotation information is wrong.

[0128] The preset accuracy threshold can be set according to actual needs and is not limited in the embodiments of the present application.

[0129] In some embodiments, the preset accuracy threshold is 90%.

[0130] In some embodiments, the electronic device's associated database stores standard semantic annotation information and corresponding standard point cloud data. After determining the semantic annotation information, the electronic device can calculate the similarity between the point cloud data corresponding to the semantic annotation information and the standard point cloud data corresponding to the standard semantic annotation information, and if the similarity is greater than or equal to a preset similarity threshold, determine that the semantic annotation information is correct, otherwise determine that it is wrong.

[0131] Each point cloud in the second point cloud data includes a relationship identifier, which can be used to indicate which frame of the first point cloud data the point cloud belongs to before merging, so as to facilitate restoring the semantic annotation information to the corresponding frame of the first point cloud data after verifying a frame of the second point cloud data.

[0132] For example, if 20 frames of first point cloud data are merged to obtain 1 frame of second point cloud data, after verification or verification of modified semantic annotation information, the electronic device can remap each point cloud data in the second point cloud data back to the corresponding frame of the first point cloud data based on the relationship identifier of each point cloud data in the second point cloud data.

[0133] In a possible implementation, semantic segmentation is performed on N frames of first point cloud data to obtain semantic annotation information of each frame of first point cloud data, including:

[0134] When deduplication processing is performed on N frames of first point cloud data, semantic segmentation is performed on all frames of first point cloud data after deduplication processing to determine semantic annotation information of the first point cloud data of each frame.

[0135] The embodiment of the present application can reduce the amount of point cloud data processing for semantic segmentation by deduplicating the first point cloud data of N frames, while improving the accuracy and efficiency of semantic annotation of point cloud data.

[0136] In a possible implementation, performing deduplication processing on N frames of first point cloud data includes:

[0137] Rasterize the space where the first point cloud data of the N frames is located to obtain multiple grids;

[0138] The point cloud data in the target grid is deleted until the number of point clouds in the target grid is less than a preset number threshold, and the target grid is a grid in which the number of point clouds in multiple grids is greater than or equal to the preset number threshold.

[0139] The embodiment of the present application performs rasterization processing on the airborne point cloud data of the N frames, and then deletes the point cloud data in the grid based on a preset quantity threshold, thereby achieving deduplication of the first point cloud data of the N frames.

[0140] Specifically, the electronic device may delete the point cloud data by randomly selecting the point cloud data.

[0141] In some embodiments, after rasterizing the space where the first point cloud data of frames N are located, the electronic device can sort the point cloud data in each grid according to the order of coordinates x, y, and z in the three-dimensional space, and then delete the sorted point cloud data according to a preset interval.

[0142] In a possible implementation, the M frames of point cloud data collected continuously are divided and processed to obtain N point cloud sets, including:

[0143] Get the acquisition timestamp of M frames of point cloud data;

[0144] The M frames of point cloud data are divided and processed according to the acquisition timestamp to obtain N point cloud sets.

[0145] The embodiment of the present application obtains the acquisition timestamp of M frames of point cloud data, so that the M frames of point cloud data can be accurately divided according to the acquisition timestamp.

[0146] Corresponding to the above method embodiment, the present application embodiment also provides a point cloud data annotation system, see Figure 3 , Figure 3 A schematic diagram of functional modules of a point cloud data annotation system provided in an embodiment of the present application, wherein the point cloud data annotation system 300 includes:

[0147] A division module 310 is used to divide the M frames of point cloud data collected continuously to obtain N point cloud sets, each point cloud set includes a number of frames of point cloud data, wherein M and N are both integers greater than 1;

[0148] A first labeling module 320 is used to label the three-dimensional frame of each frame of point cloud data in the M frames of point cloud data to obtain pre-labeling information of each frame of point cloud data;

[0149] A merging module 330 is used to merge at least part of the frame point cloud data in each point cloud set according to the pre-annotated information and the pose information of each point cloud data in each point cloud set, so as to obtain N frames of first point cloud data corresponding to the N point cloud sets;

[0150] The second labeling module 340 is used to perform semantic segmentation on the N frames of first point cloud data to obtain semantic labeling information of each frame of the first point cloud data.

[0151] The point cloud data annotation system provided in the embodiment of the present application can achieve the following Figure 1 The various processes implemented in the embodiment of the method for labeling point cloud data can achieve similar or identical technical effects, and will not be described again here to avoid repetition.

[0152] In a possible implementation, the merging module 330 is further configured to:

[0153] Determine first intermediate frame point cloud data in at least part of the frame point cloud data in each point cloud set;

[0154] Convert each point cloud data into a world coordinate system according to pre-annotated information and pose information of each point cloud data in at least part of the frame point cloud data;

[0155] In the world coordinate system, at least part of the frame point cloud data except the first intermediate frame point cloud data is mapped to the first intermediate frame point cloud data to obtain N frames of first point cloud data corresponding to the N point cloud sets.

[0156] In a possible implementation, the merging module 330 further includes a first merging submodule, and the first merging submodule is used to:

[0157] According to the first information of at least part of the frame point cloud data, in the world coordinate system, static objects of other frame point cloud data except the intermediate frame point cloud data in at least part of the frame point cloud data are mapped onto the static objects of the intermediate frame point cloud data.

[0158] In a possible implementation, the pre-labeled information includes second information for indicating a dynamic object in the corresponding point cloud data;

[0159] The first qualified submodule is also used to:

[0160] According to the second information of at least part of the frame point cloud data, in the world coordinate system, the dynamic objects of the frame point cloud data other than the intermediate frame point cloud data in at least part of the frame point cloud data are mapped onto the dynamic objects of the intermediate frame point cloud data.

[0161] In a possible implementation, the point cloud data annotation system 300 further includes a verification module, which is used to:

[0162] Determine the second intermediate frame point cloud data of the N frames of first point cloud data;

[0163] According to the pre-annotated information and the pose information of each first point cloud data in the N frames of first point cloud data, each first point cloud data is converted into a world coordinate system;

[0164] In the world coordinate system, the first point cloud data of other frames except the second intermediate frame point cloud data in the N frames of the first point cloud data are mapped to the second intermediate frame point cloud data to obtain a frame of the second point cloud data corresponding to the N frames of the first point cloud data, wherein the point cloud in the second point cloud data includes a relationship identifier, and the relationship identifier is used to indicate the frame of the first point cloud data before the point cloud is merged;

[0165] Obtaining a verification result of the semantic annotation information of the second point cloud data;

[0166] When the verification result indicates that the accuracy of the semantic annotation information is greater than or equal to the preset accuracy threshold, all point clouds in the second point cloud data are mapped to the first point cloud data of the corresponding frame according to the relationship identifiers of each point cloud in the second point cloud data.

[0167] In a possible implementation, the second marking module 340 is further configured to:

[0168] When deduplication processing is performed on N frames of first point cloud data, semantic segmentation is performed on all frames of first point cloud data after deduplication processing to determine semantic annotation information of the first point cloud data of each frame.

[0169] In a possible implementation, the second marking module 340 includes a marking submodule, and the marking submodule is used to:

[0170] Rasterize the space where the first point cloud data of the N frames is located to obtain multiple grids;

[0171] The point cloud data in the target grid is deleted until the number of point clouds in the target grid is less than a preset number threshold, and the target grid is a grid in which the number of point clouds in multiple grids is greater than or equal to the preset number threshold.

[0172] In a possible implementation, the division module 310 is further configured to:

[0173] Get the acquisition timestamp of M frames of point cloud data;

[0174] The M frames of point cloud data are divided and processed according to the acquisition timestamp to obtain N point cloud sets.

[0175] The present application also provides an electronic device. The present application also provides an electronic device. Figure 4 , Figure 4 An internal structure diagram of an electronic device provided for an embodiment of the present application. The electronic device includes a processor, a memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the electronic device stores an operating system and may also store a computer program, which, when executed by the processor, enables the processor to implement the method for labeling point cloud data applied to the electronic device in the above-mentioned embodiment. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to execute the method for labeling point cloud data. Those skilled in the art will appreciate that Figure 4 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0176] The embodiment of the present application further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a method for labeling point cloud data as in the method embodiment is implemented.

[0177] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the embodiment of the point cloud data annotation method as described above, and can achieve similar or identical technical effects. To avoid repetition, it will not be described here.

[0178] An embodiment of the present application further discloses a vehicle, on which is installed a point cloud data annotation system, an electronic device or a computer-readable storage medium capable of implementing the point cloud data annotation method of any of the above embodiments.

[0179] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0180] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for labeling point cloud data, characterized in that: The method comprises: The M frames of point cloud data collected continuously are divided and processed to obtain N point cloud sets, each of which includes a plurality of frames of point cloud data, wherein M and N are both integers greater than 1; Annotating a three-dimensional frame of the point cloud data of each frame in the M frames of point cloud data to obtain pre-annotated information of the point cloud data of each frame; According to the pre-annotated information and the pose information of each of the point cloud data in each of the point cloud sets, at least part of the frame point cloud data in each of the point cloud sets are merged to obtain N frames of first point cloud data corresponding to the N point cloud sets; Semantic segmentation is performed on the N frames of first point cloud data to obtain semantic annotation information of each frame of the first point cloud data.

2. The method according to claim 1, characterized in that The step of merging at least part of the frame point cloud data in each of the point cloud sets according to the pre-annotated information and the pose information of each of the point cloud data in each of the point cloud sets to obtain N frames of first point cloud data corresponding to the N point cloud sets includes: Determine first intermediate frame point cloud data in at least part of the frame point cloud data in each of the point cloud sets; According to the pre-annotated information and the pose information of each of the point cloud data in the at least part of the frame point cloud data, converting each of the point cloud data into a world coordinate system; In the world coordinate system, other frame point cloud data in the at least part of the frame point cloud data except the first intermediate frame point cloud data are mapped to the first intermediate frame point cloud data to obtain the N frames of first point cloud data corresponding to the N point cloud sets.

3. The method according to claim 2, characterized in that The pre-labeled information includes first information for indicating a static object in the corresponding point cloud data; The step of mapping other frame point cloud data except the first intermediate frame point cloud data in the at least part of the frame point cloud data to the first intermediate frame point cloud data in the world coordinate system includes: According to the first information of the at least partial frame point cloud data, in the world coordinate system, static objects of other frame point cloud data in the at least partial frame point cloud data except the first intermediate frame point cloud data are mapped onto static objects of the first intermediate frame point cloud data.

4. The method according to claim 2, characterized in that The pre-labeled information includes second information for indicating a dynamic object in the corresponding point cloud data; The step of mapping other frame point cloud data except the first intermediate frame point cloud data in the at least part of the frame point cloud data to the first intermediate frame point cloud data in the world coordinate system includes: According to the second information of the at least partial frame point cloud data, in the world coordinate system, the dynamic objects of the frame point cloud data other than the first intermediate frame point cloud data in the at least partial frame point cloud data are mapped onto the dynamic objects of the first intermediate frame point cloud data.

5. The method according to claim 1, characterized in that After performing semantic segmentation on the N frames of first point cloud data to obtain semantic annotation information of each frame of the first point cloud data, the method further includes: Determine the second intermediate frame point cloud data of the N frames of first point cloud data; According to the pre-annotated information and the pose information of each of the first point cloud data in the N frames of first point cloud data, each of the first point cloud data is converted into a world coordinate system; In the world coordinate system, mapping the first point cloud data of other frames among the N frames of the first point cloud data except the second intermediate frame point cloud data to the second intermediate frame point cloud data, to obtain a frame of second point cloud data corresponding to the N frames of the first point cloud data, wherein the point cloud in the second point cloud data includes a relationship identifier, and the relationship identifier is used to indicate a frame of the first point cloud data before the point cloud is merged; Obtaining a verification result of the semantic annotation information of the second point cloud data; When the verification result indicates that the accuracy of the semantic annotation information is greater than or equal to a preset accuracy threshold, all point clouds in the second point cloud data are mapped to the first point cloud data of the corresponding frame according to the relationship identifiers of each point cloud in the second point cloud data.

6. The method according to claim 1, characterized in that The performing semantic segmentation on the N frames of first point cloud data to obtain semantic annotation information of each frame of the first point cloud data includes: In the case where deduplication processing is performed on the N frames of first point cloud data, semantic segmentation is performed on all frames of first point cloud data after deduplication processing to determine semantic annotation information of the first point cloud data of each frame.

7. The method according to claim 6, characterized in that The deduplication processing of the N frames of first point cloud data includes: Rasterizing the space where the N frames of first point cloud data are located to obtain a plurality of grids; The point cloud data in the target grid is deleted until the number of point clouds in the target grid is less than a preset number threshold, and the target grid is a grid in which the number of point clouds in the multiple grids is greater than or equal to the preset number threshold.

8. The method according to claim 1, characterized in that The M frames of point cloud data collected continuously are divided and processed to obtain N point cloud sets, including: Obtaining a collection timestamp for collecting the M frames of point cloud data; The M frames of point cloud data are divided and processed according to the acquisition timestamp to obtain the N point cloud sets.

9. A point cloud data annotation system, characterized in that: The system comprises: A division module is used to divide and process the M frames of point cloud data collected continuously to obtain N point cloud sets, each of which includes a number of frames of point cloud data, wherein M and N are both integers greater than 1; A first annotation module is used to annotate a three-dimensional frame of each frame of the point cloud data in the M frames of point cloud data to obtain pre-annotated information of each frame of the point cloud data; A merging module, configured to merge at least part of the frame point cloud data in each of the point cloud sets according to the pre-annotated information and the pose information of each of the point cloud data in each of the point cloud sets, to obtain N frames of first point cloud data corresponding to the N point cloud sets; The second annotation module is used to perform semantic segmentation on the N frames of first point cloud data to obtain semantic annotation information of the first point cloud data of each frame.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein a computer program is stored in the memory, and the computer program implements the method according to any one of claims 1 to 8 when executed by the processor.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The computer program product comprises a computer program, which implements the method of any one of claims 1 to 8 when executed by one or more processors.

13. A vehicle, characterized in that: Include at least one of the following: The system for controlling a vehicle as claimed in claim 9; The electronic device as claimed in claim 10; The computer readable storage medium of claim 11.

Images