Point cloud labeling method, device and system

Abstract

The invention discloses a point cloud labeling method, device and system, and relates to the field of intelligent driving, and the method comprises the steps: inputting a plurality of frames of point clouds into a neural network model, and outputting the plurality of frames of labeled point clouds; determining a static base map according to the multiple frames of point clouds; performing ground segmentation on the static base map, and determining ground points of the static base map; according to the ground points of the static base map, respectively filling the ground points of the multi-frame marked point clouds to determine multi-frame filled point clouds; and based on the ground points in the point clouds after the multi-frame filling, respectively adjusting the lower edge heights of the marking boxes in the point clouds after the multi-frame filling. According to the point cloud labeling method, the system is helped to automatically complete labeling of the dynamic target in the multi-frame point cloud, and automatic ground alignment of the target is realized for the labeled target frame, so that the manual labeling and the target ground alignment process are avoided, the accuracy of automatic ground alignment is improved, and the time and labor are saved. And the efficiency of point cloud labeling and automatic target ground alignment is also improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving, and more specifically, to a method, device and system for point cloud annotation. Background Art

[0002] In autonomous driving technology, sensor technology is crucial. Radar detection technology is an important part of sensor technology, and Lidar (light detection and ranging) is a commonly used radar system in automotive radars. Labeling dynamic targets based on Lidar is also an important topic in autonomous driving technology.

[0003] However, there are many challenges in labeling dynamic targets based on Lidar. The work of labeling dynamic targets on the ground requires the labelers to complete it manually based on the current point cloud data and experience. This process is relatively cumbersome, and the manual labeling method is not only difficult to guarantee accuracy, but also has low efficiency.

[0004] In view of this, a method that can achieve automatic labeling and automatic grounding for dynamic targets is urgently needed to be developed. Summary of the invention

[0005] The present application provides a method, device and system for point cloud annotation, which can realize automatic annotation of dynamic targets and automatically align the annotated target frame with the ground.

[0006] In a first aspect, a method for point cloud annotation is provided, the method comprising: inputting a plurality of frames of first point clouds into a first neural network model, and outputting a plurality of frames of second point clouds, the plurality of frames of first point clouds being point clouds acquired by a vehicle in a first time period for characterizing road information, the plurality of frames of second point clouds comprising point clouds for characterizing dynamic targets and a first target frame corresponding to the dynamic targets, the dynamic targets being located within the first target frame; determining a static base map based on the plurality of frames of first point clouds; performing ground segmentation on the static base map to determine ground points of the static base map; filling ground points of the plurality of frames of second point clouds respectively based on the ground points of the static base map to determine a plurality of frames of third point clouds; and adjusting the bottom edge height of the first annotation frame in the plurality of frames of third point clouds respectively based on the ground points in the plurality of frames of third point clouds.

[0007] For example, the first point cloud may be an environment point cloud around the vehicle collected by a sensor of the vehicle, or may be a multi-frame original point cloud that has not been processed in any way.

[0008] For example, the first time period may be a collection period in which the vehicle collects multiple frames of the first point cloud.

[0009] For example, the first target frame may be in the form of a cube, and the inner sides of each face of the first target frame may be tangent to the outer edge of the detected dynamic target.

[0010] For example, the road information includes the road surface morphology, the location information and geometric parameters of dynamic targets, and the location information and geometric parameters of stationary targets. Among them, dynamic targets may include moving vehicles, walking pedestrians, stationary vehicles, etc., and stationary targets may include trees planted beside the road, buildings beside the road, and traffic cones temporarily placed on the road. It can be seen that the dynamic targets mentioned in the embodiments of the present application are defined as targets that can move, even if the target is stationary, it is also classified as a dynamic target; and stationary targets are defined as targets that cannot move.

[0011] For example, the road section range covered by the static base map may be the union of the road section ranges respectively represented by the multiple frames of first point clouds.

[0012] For example, the static base map may be determined based on the multiple frames of first point clouds by using a simultaneous localization and mapping (SLAM) method.

[0013] For example, the above-mentioned ground segmentation method can be completed based on a point cloud ground filtering algorithm, or it can be completed based on a pre-trained deep learning model.

[0014] Based on the above technical solution, the ground points of each single-frame point cloud are supplemented based on the static base map obtained after superimposing the multi-frame point cloud to reconstruct the ground in the area without ground points on the single-frame point cloud, thereby solving the problem of sparse ground points at a long distance in the single-frame point cloud. In addition, the ground point filling operation is performed on the acquired multi-frame second point cloud based on the static base map, which helps the system to automatically complete the annotation of dynamic targets in the multi-frame point cloud, and realize automatic grounding of the target for the annotated target frame, thereby avoiding manual annotation and target grounding, improving the accuracy of automatic grounding, and increasing the efficiency of point cloud annotation and automatic target grounding.

[0015] In combination with the first aspect, in certain implementations of the first aspect, a first coordinate system is established based on a static base map, and the first coordinate system is mapped to multiple frames of second point clouds; based on the first coordinate system, the ground points of the static base map are respectively mapped to the multiple frames of second point clouds.

[0016] For example, the first coordinate system may be a three-dimensional Cartesian coordinate system or a spherical coordinate system.

[0017] Based on the above technical solution, it is helpful to realize automatic filling of ground points in multi-frame second point clouds, thereby solving the problem of sparse ground points at a long distance in a single-frame point cloud, and filling in ground points based on the first coordinate system established based on the static base map can ensure the accuracy of ground point filling.

[0018] In combination with the first aspect, in certain implementations of the first aspect, multiple frames of first point clouds are input into a second neural network model to output multiple frames of fourth point clouds, wherein the fourth point clouds eliminate point clouds used to characterize dynamic targets compared to the first point clouds; the multiple frames of fourth point clouds are superimposed to determine the static base map.

[0019] For example, the second neural network model can reuse the first neural network model.

[0020] Based on the above technical solution, considering that the point cloud corresponding to the dynamic target can affect the subsequent process of determining the ground point cloud, before determining the static base map, the point cloud corresponding to the dynamic target in the first point cloud of multiple frames is first eliminated, so as to better expose the ground points in the static base map, so as to facilitate the subsequent reconstruction of the ground point cloud.

[0021] In combination with the first aspect, in certain implementations of the first aspect, before superimposing multiple frames of fourth point clouds to determine a static base map, at least one fourth point cloud set is determined, and each fourth point cloud set includes two adjacent frames of the above-mentioned multiple frames of fourth point clouds; through a point cloud registration algorithm, the two frames of fourth point clouds included in the above-mentioned at least one fourth point cloud set are sequentially registered to determine a positioning calibration matrix; according to the positioning calibration matrix, the position information of the above-mentioned multiple frames of fourth point clouds is adjusted.

[0022] For example, if the number of frames of the fourth point cloud determined through the corresponding point cloud processing flow is N frames, then based on the N-frame fourth point cloud, N-1 fourth point cloud sets can be determined. Then, the i-th fourth point cloud set in the N-1 fourth point cloud sets can include the i-th frame and the i+1-th frame in the N-frame fourth point cloud, where i is an integer greater than 0 and less than N.

[0023] For example, the specific registration method may be to use a pre-trained neural network model for registration, or to use an iterative closest point (ICP) algorithm for registration.

[0024] For example, the position information of the fourth point cloud of the plurality of frames may be the coordinate value of the point cloud in the picture.

[0025] Based on the above technical solution, it is helpful to reduce the inter-frame point cloud positioning error of the fourth point cloud of multiple frames, and ensure the accuracy of the subsequent determination of the static base map.

[0026] In combination with the first aspect, in certain implementations of the first aspect, the first coordinate system is a three-dimensional Cartesian coordinate system, and the Z axis of the first coordinate system is used to represent the height; the Z axis coordinate of the lower edge of the first target frame is adjusted to the Z axis coordinate of the ground point closest to the lower edge of the first target frame.

[0027] Based on the above technical solution, automatic grounding for the first target frame can be achieved.

[0028] In combination with the first aspect, in certain implementations of the first aspect, when the number of identified dynamic targets is M, the number of first target frames is also M accordingly.

[0029] In combination with the first aspect, in certain implementations of the first aspect, before determining a static base map based on multiple frames of first point clouds, at least one first point cloud set is determined, each first point cloud set including two adjacent frames of the multiple frames of first point clouds; the similarity between the two frames of first point clouds included in the at least one first point cloud set is determined in turn; when the similarity between the two frames of first point clouds included in the above first point cloud set is greater than a preset first threshold, the first point cloud with a larger number of points in the above first point cloud set is retained.

[0030] By way of example, the similarity of two adjacent frames of first point clouds in the at least one first point cloud set can be identified separately based on the third neural network model; or, the first neural network model can be reused in the process of dynamic target recognition to simultaneously identify the similarity of two adjacent frames of first point clouds in the at least one first point cloud set; when the similarity of two adjacent frames of point clouds is greater than a preset first threshold, the frame of point cloud with more points will be selected for subsequent processing, and the other frame of point cloud can be discarded or stored in a temporary cache and will not participate in the subsequent point cloud processing process.

[0031] Based on the above technical solution, the first point clouds of two adjacent frames with high similarity can be simplified, which helps to reduce the subsequent calculation overhead of the method and thus increase the efficiency of point cloud annotation.

[0032] In a second aspect, a point cloud annotation device is provided, which is applied to a vehicle, and the device includes: a first processing unit and a second processing unit, wherein the first processing unit is used to: input a plurality of frames of first point clouds into a first neural network model, and output a plurality of frames of second point clouds, wherein the plurality of frames of first point clouds are point clouds acquired by the vehicle in a first time period for characterizing road information, and the plurality of frames of second point clouds include point clouds for characterizing dynamic targets and a first target frame annotated for the dynamic targets, and the dynamic targets are located in the first target frame; determine a static base map based on the plurality of frames of first point clouds; the second processing unit is used to: perform ground segmentation on the static base map to determine the ground points of the static base map; fill the ground points of the plurality of frames of second point clouds respectively according to the ground points of the static base map to determine a plurality of frames of third point clouds; and adjust the bottom edge height of the first annotation frame in the plurality of frames of third point clouds based on the ground points in the plurality of frames of third point clouds.

[0033] Based on the above technical solution, the ground points of each single-frame point cloud are supplemented based on the static base map obtained after superimposing the multi-frame point cloud to reconstruct the ground in the area without ground points on the single-frame point cloud, thereby solving the problem of sparse ground points at a long distance in the single-frame point cloud. In addition, the ground point filling operation is performed on the acquired multi-frame second point cloud based on the static base map, which helps the system to automatically complete the annotation of dynamic targets in the multi-frame point cloud, and also realize the automatic grounding of the target for the annotated target frame, thereby avoiding manual annotation and target grounding, improving the accuracy of automatic grounding, and increasing the efficiency of point cloud annotation and target automatic grounding.

[0034] In combination with the second aspect, in certain implementations of the second aspect, the second processing unit is specifically used to: establish a first coordinate system based on a static base map, and map the first coordinate system to the multi-frame second point cloud; based on the first coordinate system, map the ground points of the static base map to the multi-frame second point cloud respectively.

[0035] Based on the above technical solution, it is helpful to realize automatic filling of ground points in multi-frame second point clouds, thereby solving the problem of sparse ground points at a long distance in a single-frame point cloud, and filling in ground points based on the first coordinate system established based on the static base map can ensure the accuracy of ground point filling.

[0036] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned first processing unit is specifically used to: input the above-mentioned multiple frames of first point clouds into the second neural network model, output multiple frames of fourth point clouds, and the fourth point clouds eliminate point clouds used to characterize dynamic targets compared to the first point cloud; superimpose the above-mentioned multiple frames of fourth point clouds to determine the above-mentioned static base map.

[0037] Based on the above technical solution, considering that the point cloud corresponding to the dynamic target can affect the subsequent process of determining the ground point cloud, before determining the static base map, the point cloud corresponding to the dynamic target in the first point cloud of multiple frames is first eliminated, so as to better expose the ground points in the static base map, so as to facilitate the subsequent reconstruction of the ground point cloud.

[0038] In combination with the second aspect, in certain implementations of the second aspect, before the first processing unit superimposes multiple frames of fourth point clouds to determine the static base map, the first processing unit is also used to: determine at least one fourth point cloud set, each fourth point cloud set includes two adjacent frames in the multiple frames of fourth point clouds; through a point cloud registration algorithm, sequentially perform registration operations on two frames of fourth point clouds included in at least one fourth point cloud set to determine a positioning calibration matrix; and adjust the position information of the multiple frames of fourth point clouds according to the positioning calibration matrix.

[0039] Based on the above technical solution, it is helpful to reduce the inter-frame point cloud positioning error of the fourth point cloud of multiple frames, and ensure the accuracy of the subsequent determination of the static base map.

[0040] In combination with the second aspect, in certain implementations of the second aspect, the first coordinate system is a three-dimensional Cartesian coordinate system, the Z axis of the first coordinate system is used to represent the height, and the second processing unit is specifically used to: adjust the Z axis coordinate of the lower edge of the first target frame to the Z axis coordinate of the ground point closest to the lower edge of the first target frame.

[0041] Based on the above technical solution, automatic grounding for the first target frame can be achieved.

[0042] In combination with the second aspect, in certain implementations of the second aspect, when the number of identified dynamic targets is M, the number of first target frames is also M accordingly.

[0043] In combination with the second aspect, in certain implementations of the second aspect, before the first processing unit determines the static base map based on the multiple frames of first point clouds, the first processing unit is also used to: determine at least one first point cloud set, each first point cloud set includes two adjacent frames in the multiple frames of first point clouds; sequentially determine the similarity between the two frames of first point clouds included in the at least one first point cloud set; when the similarity between the two frames of first point clouds included in the first point cloud set is greater than a preset first threshold, retain the first point cloud with a larger number of points in the first point cloud set.

[0044] Based on the above technical solution, the first point clouds of two adjacent frames with high similarity can be simplified, which helps to reduce the subsequent calculation overhead of the method and thus increase the efficiency of point cloud annotation.

[0045] In a third aspect, a point cloud annotation device is provided, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute a method in any possible implementation mode of the method design of the first aspect above.

[0046] In a fourth aspect, a chip system is provided, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to echo a signal from a memory of the electronic device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes a method in any possible implementation of the method design of the first aspect above.

[0047] In a fifth aspect, a computer-readable storage medium is provided, storing a computer program or instructions, wherein the computer program or instructions are used to implement the method in any possible implementation manner in the method design of the first aspect.

[0048] In a sixth aspect, a computer program product is provided. When the computer program code or instructions are executed on a computer, the computer executes a method in any possible implementation of the method design of the first aspect above.

[0049] In a seventh aspect, a vehicle is provided, comprising a device as in any possible implementation of the second aspect or the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a functional schematic block diagram of a vehicle 100 provided in an embodiment of the present application.

[0051] Figure 2 It is a schematic diagram of the architecture of a point cloud annotation system 200 provided in an embodiment of the present application.

[0052] Figure 3 It is a flowchart of a method for point cloud annotation provided in an embodiment of the present application.

[0053] Figure 4 It is a schematic diagram of determining the coverage of a static base map provided in an embodiment of the present application.

[0054] Figure 5 This is another schematic diagram for determining the coverage of a static base map provided in an embodiment of the present application.

[0055] Figure 6 It is a flowchart of a method for filling ground points of multiple frames of second point clouds provided in an embodiment of the present application.

[0056] Figure 7 It is a flow chart of a method for determining a static base map provided in an embodiment of the present application.

[0057] Figure 8 It is a flowchart of a method for pre-processing a plurality of first point clouds provided in an embodiment of the present application.

[0058] Fig. 9 It is a flowchart of a method for inter-frame positioning and registration provided in an embodiment of the present application.

[0059] Fig.10 It is a schematic diagram of a first range provided in an embodiment of the present application.

[0060] Fig.11 This is a schematic diagram of another first range provided in an embodiment of the present application.

[0061] Fig.12 It is a schematic diagram of a static base map provided in an embodiment of the present application.

[0062] Fig.13 It is a schematic diagram of another static base map provided in an embodiment of the present application.

[0063] Fig.14 It is a schematic block diagram of a point cloud annotation device 1400 provided in an embodiment of the present application.

[0064] Fig.15 It is a schematic block diagram of a computer device 1500 provided in an embodiment of the present application.

[0065] Fig.16 It is a schematic block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is a kind of association relationship that describes associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In this application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0067] The prefixes such as "first" and "second" used in the embodiments of the present application are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers used to distinguish description objects in the embodiments of the present application does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0068] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0069] Figure 1 It is a functional schematic block diagram of a vehicle 100 provided in an embodiment of the present application.

[0070] The vehicle 100 may include a perception system 120, a display device 130, and a computing platform 150, wherein the perception system 120 may include several sensors for sensing information about the environment around the vehicle 100. For example, the perception system 120 may include a positioning system, which may be a global positioning system (GPS), a Beidou system or other positioning systems, an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera device.

[0071] Some or all functions of the vehicle 100 may be controlled by a computing platform 150. The computing platform 150 may include processors 151 to 15n (n is a positive integer). The processor is a circuit with signal processing capability. In one implementation, the processor may be a circuit with instruction reading and execution capability, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which may be understood as a microprocessor), or a digital signal processor (DSP). In another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as a hardware circuit implemented by a processor such as an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field programmable gate array (FPGA). In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration may be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. In addition, the computing platform 150 can also include a memory, the memory is used to store instructions, and some or all of the processors 151 to 15n can call the instructions in the memory to implement the corresponding functions described in the embodiments of the present application.

[0072] The vehicles involved in this application may include road vehicles, water vehicles, air vehicles, industrial equipment, agricultural equipment, or entertainment equipment, etc. For example, the vehicle may include an unmanned vehicle, which is a vehicle in a broad sense, and may be a vehicle (such as a commercial vehicle, a passenger car, a motorcycle, a flying car, a train, etc.), an industrial vehicle (such as a forklift, a trailer, a tractor, etc.), an engineering vehicle (such as an excavator, a bulldozer, a crane, etc.), agricultural equipment (such as a mower, a harvester, etc.), amusement equipment, a toy vehicle, etc. The embodiment of this application does not specifically limit the type of vehicle. For the convenience of description, the embodiment of this application is described in detail by taking an autonomous driving vehicle as an example.

[0073] There are many challenges in the dynamic target annotation technology based on Lidar. For example, the number of annotated targets is large: a training session requires tens to hundreds of thousands of annotated single-frame point clouds, and each frame of point cloud needs to be annotated with hundreds to thousands of 3D annotation boxes one by one. In addition, each 3D annotation box also has a lot of annotation attributes. For example, each 3D annotation box needs to be annotated with the length, width, height, spatial position, orientation, type, and logo of the box. Among them, the 3D annotation box is also called the 3D bounding box, also referred to as the annotation box or target box; in addition, the annotation rules are also very complex, for example, the target needs to be close to the edge, the target needs to be on the ground, and the target's motion trajectory needs to be smooth.

[0074] From the above examples, it can be seen that dynamic target annotation based on Lidar is a very labor-intensive and time-consuming task. Among them, automatic target alignment is one of the most time-consuming steps. It requires the annotator to first determine the ground points near the target, and then use the corresponding annotation tools to adjust the height of the target center point in the annotation visualization interface so that the lower edge of the frame is close to the ground point cloud to complete the target alignment. Annotators usually perform dynamic target annotation on a single-frame point cloud. When the annotated target is very close to the vehicle, the point cloud is relatively dense. Therefore, the ground point cloud is easy to accurately identify, and based on this, the annotated target alignment can also be achieved more accurately.

[0075] However, when the labeled target is far away from the vehicle, for example, more than 60 meters away, the ground point cloud at the location of the labeled target is very sparse, or even nonexistent. At this time, the annotator needs to judge the height of the center point of the labeled target based on the labeling experience and the environment around the target, such as trees, light poles, etc., and then complete the target alignment. In this case, it is necessary to manually judge whether the height of the target box is correct one by one, which greatly reduces the labeling efficiency.

[0076] In view of this, the embodiments of the present application propose a method, device and system for point cloud annotation, which automatically complete point cloud annotation. When the distance of the annotated target is far, the ground point cloud around the target is sparse or even there is no ground point cloud. By acquiring multiple frames of point cloud, a static base map that can be used as a reference is established, and then automatic ground alignment for each dynamic target is completed based on the static base map.

[0077] It should be understood that the above-mentioned process of automatically grounding a target generally refers to marking a dynamic target and then completing the grounding operation for a target frame corresponding to the dynamic target.

[0078] Figure 2 It is a schematic diagram of the architecture of a point cloud annotation system 200 proposed in an embodiment of the present application.

[0079] refer to Figure 2As shown, the point cloud annotation system proposed in the embodiment of the present application can be deployed on the vehicle side, including: a data acquisition module, a perception module, a data mining module, a data annotation module and a data feedback module. The system is also referred to as the first system in this application.

[0080] Among them, the data acquisition module can obtain the point cloud data collected by Lidar, specifically the road test data, and transmit this data to the perception module.

[0081] After acquiring the above data, the perception module performs real-time detection on the collected data based on the corresponding perception algorithm. The detected attributes include: dynamic target information, such as vehicles, pedestrians and other moving targets, and road structure information, such as lane lines, road rocks, traffic signs, etc. The detected information is then transmitted to the data mining module.

[0082] The data mining module analyzes and selects high-value data based on the corresponding feature extraction algorithm, such as the type of dynamic targets, the location information of dynamic targets, and the spatial collection parameters of dynamic targets, etc. The selected data is then transmitted to the data annotation module.

[0083] The data annotation module generates a static base map based on the point cloud annotation method proposed in the embodiment of the present application, automatically annotates the target based on the static base map, and automatically aligns the target. Finally, the static base map after the target annotation and target alignment and other relevant information are transmitted to the data return module.

[0084] In some possible embodiments, the data annotation module may include two submodules: a static base map reconstruction module and an automatic ground alignment module. The static base map reconstruction module is specifically used to generate a static base map of the current scene based on multiple frame point clouds; the automatic ground alignment module is specifically used to automatically annotate targets based on the static base map and complete automatic ground alignment of the targets.

[0085] After receiving the above target annotations and the static base map of the target on the ground and other related information, the data return module uploads these data information to the server background.

[0086] It should be understood that the point cloud annotation system proposed in the above embodiment can automatically perform target annotation based on the collected point cloud data, thereby completing automatic grounding of the target.

[0087] In some possible embodiments, the embodiments of the present application also propose a system for labeling laser point cloud data, which is also referred to as the second system in the present application. The system can obtain a large amount of point cloud data collected by the vehicle and the labeling data corresponding to each frame of the point cloud, and then display the point cloud data and labeling data in the specified scene based on the main interface provided by the system, and provide a front-end labeling tool. The user completes further labeling work on the static base map through the corresponding front-end labeling tool, that is, the second system can serve as a supplement to the above-mentioned first system. For example, after the first system completes the automatic labeling of dynamic targets and automatic grounding, the result data will be presented to the user through the main interface of the second system. The user can verify whether the results of the target labeling and automatic grounding of the first system are accurate. If not, the user can correct the output results of the first system based on the second system to ensure the accuracy of dynamic target labeling and grounding.

[0088] Based on the above-mentioned system architecture of point cloud annotation, an embodiment of the present application proposes a method of point cloud annotation, which can be specifically applied to the above-mentioned data annotation module to complete the corresponding functions.

[0089] Figure 3 1 is a flow chart of a method for point cloud annotation proposed in an embodiment of the present application. The method can be applied to an autonomous driving vehicle or a smart car, assuming that the vehicle is traveling on a first road section.

[0090] S310: Input multiple frames of first point clouds into the first neural network model, and output multiple frames of second point clouds.

[0091] The multi-frame first point cloud is a point cloud used to characterize road information acquired by the vehicle in the first time period, and the multi-frame second point cloud includes a point cloud used to characterize a dynamic target and a first target frame corresponding to the dynamic target, and the dynamic target is located in the first target frame. The first time period may be a collection period during which the vehicle collects the multi-frame first point cloud.

[0092] In some possible embodiments, the first point cloud may be an environment point cloud around the vehicle collected by the vehicle through a sensor, or may be a multi-frame original point cloud that has not been processed in any way.

[0093] In some possible embodiments, the road information includes the road surface morphology, the location information and geometric parameters of dynamic targets, and the location information and geometric parameters of stationary targets. Among them, dynamic targets may include moving vehicles, walking pedestrians, stationary vehicles, etc., and stationary targets may include trees planted beside the road, buildings beside the road, and traffic cones temporarily placed on the road. It can be seen that the dynamic targets mentioned in the embodiments of the present application are defined as targets that can move, even if the target is stationary, it is also classified as a dynamic target; and stationary targets are defined as targets that cannot move.

[0094] In some possible embodiments, the first target frame may be in the form of a cube, and the inner sides of each face of the first target frame may be tangent to the outer edge of the detected dynamic target.

[0095] In some possible embodiments, when the number of identified dynamic targets is M, the number of target frames is also M accordingly.

[0096] In some possible embodiments, when multiple dynamic targets of the same type and adjacent to each other on the same plane are identified, these dynamic targets may be framed in a first target frame. Based on this, the number of first target frames may be reduced to reduce subsequent computational overhead.

[0097] S320: Determine a static base map according to the first point clouds of the multiple frames.

[0098] In some possible embodiments, assume that N frames of first point clouds are acquired during S310, where N is greater than 1. Based on the coverage of the first point cloud acquired by the vehicle sensor, the range of the first road section covered by the static base map can be determined by the following two methods.

[0099] Figure 4 This is a schematic diagram of determining the coverage range of a static base map proposed in an embodiment of the present application. In this diagram, the coverage range of the first point cloud acquired by the vehicle is a circular range with the vehicle as the center and a radius of the first distance, wherein the first distance is the detection distance of the vehicle sensor, and the first distance can be adjusted appropriately, and this embodiment of the present application does not limit this.

[0100] In some possible embodiments, reference Figure 4 As shown in the figure, when the vehicle is in the first position, the first point cloud of the first frame is obtained. Based on the above corresponding description, it can be seen that the point cloud closer to the first position is denser, so the ground point cloud is easy to accurately identify, but the position farther from the first position, such as the farthest position that the sensor can detect, is Figure 4 The point cloud near the second position in the image is very sparse, and even the ground point cloud cannot be detected. However, when the vehicle travels to the second position, the first point cloud of the Nth frame collected by the vehicle sensor can obtain the densely distributed point cloud near the second position. Therefore, in the first point cloud of the Nth frame, the ground point cloud near the second position is easy to accurately identify. By superimposing the first point clouds of the 1st frame to the Nth frame, a static base map can be obtained. The coverage range of the static base map can be the union of the road sections represented by the first point clouds of the 1st frame and the Nth frame respectively.

[0101] Figure 5This is another schematic diagram of determining the coverage of a static base map proposed in an embodiment of the present application. In this diagram, the coverage of the first point cloud acquired by the vehicle is the road range at a first distance in front of the vehicle. Similarly, the first distance is the detection distance of the vehicle sensor. The first distance can be adjusted appropriately, and this embodiment of the present application does not limit this.

[0102] In some possible embodiments, reference Figure 5 As shown in the figure, when the vehicle is in the first position, the first point cloud of the first frame is obtained. Based on the above corresponding description, it can be seen that the point cloud closer to the first position is denser, so the ground point cloud is easy to accurately identify, but the position farther from the first position, such as the farthest position that the sensor can detect, is Figure 5 The point cloud near the second position in the image is very sparse, and even the ground point cloud cannot be detected. However, when the vehicle travels to the third position, the first point cloud of the Nth frame collected by the vehicle sensor can obtain the point cloud with a denser distribution near the second position. Therefore, in the first point cloud of the Nth frame, the ground point cloud near the second position is easy to accurately identify. By superimposing the first point clouds from the 1st frame to the Nth frame, a static base map can be obtained. The coverage range of the static base map can be the union of the road sections represented by the first point clouds of the 1st frame and the Nth frame respectively.

[0103] In some possible embodiments, the static base map may be determined based on the multiple frames of first point clouds by using a SLAM method.

[0104] S330: Perform ground segmentation on the static base map to determine ground points of the static base map.

[0105] In some possible embodiments, the above-mentioned ground segmentation method can be completed based on a point cloud ground filtering algorithm, or can be completed based on a pre-trained deep learning model.

[0106] S340: Filling ground points of the multiple frames of second point clouds respectively according to the ground points of the static base map to determine the multiple frames of third point clouds.

[0107] With the above Figure 4 Taking the scene shown as an example, based on the above S340, at least the ground points of the road section with the first distance length in front (the direction of vehicle travel in the figure) corresponding to the first point cloud of the first frame can be supplemented, that is, the ground points of the road section shown in the first point cloud of the first frame to the Nth frame can be supplemented more accurately. However, the ground points of the road section farther ahead corresponding to the point cloud of the Nth frame are also sparse. For the determination and supplement of this part of the ground points, it can be supplemented by superimposing the point clouds of the N+1th frame to the N+Nth frame obtained subsequently, and determining another static base map. And so on.

[0108] S350: Based on the ground points in the third point cloud of the multiple frames, respectively adjust the bottom edge heights of the first annotation boxes in the third point cloud of the multiple frames.

[0109] It should be understood that the above adjustment of the lower edge height of the first annotation box in the third point cloud of multiple frames specifically adjusts the lower edge height of the first annotation box to be flush with the height of the plane where the ground point below it is located.

[0110] Based on the above technical solution, the ground points of each single-frame point cloud are supplemented based on the static base map obtained after superimposing the multi-frame point cloud to reconstruct the ground in the area without ground points on the single-frame point cloud, thereby solving the problem of sparse ground points at a long distance in the single-frame point cloud. In addition, the ground point filling operation is performed on the acquired multi-frame second point cloud based on the static base map, which helps the system to automatically complete the annotation of dynamic targets in the multi-frame point cloud, and also realize the automatic grounding of the target for the annotated target frame, thereby avoiding manual annotation and target grounding, improving the accuracy of automatic grounding, and increasing the efficiency of point cloud annotation and target automatic grounding.

[0111] In some possible embodiments, the above S340 may be specifically implemented by the following method.

[0112] Figure 6 It is a flowchart of a method for filling ground points of multiple frames of second point clouds proposed in an embodiment of the present application.

[0113] S610: Establish a first coordinate system based on the static base map, and map the first coordinate system to multiple frames of second point clouds.

[0114] In some possible embodiments, the first coordinate system may be a three-dimensional Cartesian coordinate system or a spherical coordinate system, which is not limited in the embodiments of the present application.

[0115] In some possible embodiments, the first coordinate system may be proportionally mapped to multiple frames of second point clouds. If the static base map and the multiple frames of second point clouds are not in proportional relationship, the first coordinate system may be mapped to the multiple frames of second point clouds with a corresponding proportional coefficient according to the proportional relationship between the static base map and the multiple frames of second point clouds.

[0116] S620: Based on the first coordinate system, the ground points of the static base map are mapped to multiple frames of second point clouds respectively.

[0117] In some possible embodiments, if the first coordinate system is a three-dimensional Cartesian coordinate system, the Z axis of the first coordinate system is used to represent the height. Then, when there is no slope in the first section, the plane formed by the X axis and the Y axis is parallel to or coincides with the plane where the first section is located.

[0118] For example, taking the first coordinate system as a three-dimensional Cartesian coordinate system, it is known that the static base map includes a ground point with coordinates (20,20,10) in the first coordinate system, but in the second point cloud that has been mapped to the first coordinate system, there is no ground point at the corresponding position of the coordinates (20,20,10), so it is necessary to fill the ground point at this position. Specifically, the method of filling the ground point in the second point cloud can be to use the first coordinate system as a reference and map the ground points of the static base map to the second point clouds of the multiple frames at the same scale.

[0119] In some possible embodiments, when the above-mentioned first coordinate system is a three-dimensional Cartesian coordinate system, the operation of adjusting the height of the lower edge of the first target box in the third point cloud of multiple frames in the above-mentioned S350 can specifically be to adjust the Z-axis coordinate of the lower edge of the first target box to the Z-axis coordinate of the ground point closest to the lower edge of the first target box.

[0120] Similarly, when the first coordinate system is a spherical coordinate system, the operation of adjusting the height of the lower edge of the first target frame in the third point cloud of multiple frames in S350 may specifically be to adjust the distance value and elevation value corresponding to the midpoint of the lower edge of the first target frame so that the lower edge of the first target frame is on the same horizontal plane as the ground point. The operation of filling the ground points of the second point cloud may also be performed using the coordinate system mapping method in the above example to fill the ground points of the second point cloud.

[0121] Based on the above technical solution, it is helpful to realize automatic filling of ground points in multiple frames of the second point cloud, thereby solving the problem of sparse ground points at a long distance in a single frame point cloud, and helping to realize subsequent automatic grounding for the first target frame and improve the accuracy of automatic grounding.

[0122] In some possible embodiments, the above S320 may be specifically implemented by the following method.

[0123] Figure 7 It is a flow chart of a method for determining a static base map proposed in an embodiment of the present application.

[0124] S710: Input the above-mentioned multi-frame first point cloud into the second neural network model, and output multiple frames of fourth point cloud. Compared with the above-mentioned first point cloud, the fourth point cloud eliminates the point cloud used to characterize the dynamic target.

[0125] In some possible embodiments, the second neural network model may reuse the first neural network model in S310.

[0126] S720: Overlay the above-mentioned multiple frames of fourth point clouds to determine a static base map.

[0127] Based on the above technical solution, considering that the point cloud corresponding to the dynamic target can affect the subsequent process of determining the ground point cloud, before determining the static base map, the point cloud corresponding to the dynamic target in the first point cloud of multiple frames is first eliminated, so as to better expose the ground points in the static base map, so as to facilitate the subsequent reconstruction of the ground point cloud.

[0128] It should be understood that the ground point described in the embodiment of the present application refers to a single point used to represent the ground, while the ground point cloud refers to a collection of these ground points.

[0129] In some possible embodiments, before the above S320, the acquired multiple frames of first point clouds may be pre-processed to achieve deduplication of the multiple frames of first point clouds.

[0130] Figure 8 It is a flowchart of a method for pre-processing a plurality of first point clouds proposed in an embodiment of the present application.

[0131] S810: Determine at least one first point cloud set, each first point cloud set including two adjacent frames in the multiple frames of first point clouds.

[0132] In some possible embodiments, when the number of frames of the first point cloud acquired by the vehicle in the first time period is N frames, correspondingly, based on the N frames of the first point cloud, N-1 first point cloud sets can be determined. For example, the i-th first point cloud set in the N-1 first point cloud sets may include the i-th frame and the i+1-th frame in the N frames of the first point cloud, where i is an integer greater than 0 and less than N.

[0133] S820: Determine in sequence the similarity between two frames of first point clouds included in at least one first point cloud set.

[0134] S830: When the similarity between the two frames of first point clouds included in the first point cloud set is greater than a preset first threshold, retain the first point cloud with a larger number of points in the first point cloud set.

[0135] In some possible embodiments, the similarity of two adjacent frames of first point clouds in the at least one first point cloud set can be identified based on the third neural network model; or, in the process of S310, the similarity of two adjacent frames of first point clouds in the at least one first point cloud set can be identified based on the first neural network model; when the similarity of two adjacent frames of point clouds is greater than a preset first threshold, a frame of point cloud with more points will be selected for subsequent processing, and the other frame of point cloud can be discarded or stored in a temporary cache and does not participate in the subsequent point cloud processing process. Based on this scheme, it can be seen that the specific number of frames of the above-mentioned multi-frame first point cloud and the above-mentioned multi-frame second point cloud can be different, that is, after N frames of first point cloud are input into the above-mentioned first neural network model, the number of frames of the second point cloud finally output after processing by the first neural network model can be M frames, where M is not equal to N.

[0136] In some possible embodiments, the above-mentioned first threshold corresponds to the similarity between two adjacent frames of first point clouds in each first point cloud set, and the similarity between the two frames of first point clouds can be determined by performing an intersection and union ratio operation on the data of the two frames of first point clouds. The closer the operation result is to 1, the higher the similarity between the two frames of first point clouds. Correspondingly, the first threshold can be preset to 0.9.

[0137] Based on the above technical solution, the first point clouds of two adjacent frames with high similarity can be simplified, which helps to reduce the subsequent calculation overhead of the method and thus increase the efficiency of point cloud annotation.

[0138] In some possible embodiments, considering that when the vehicle sensor is collecting point clouds, there may be errors in the positioning of point clouds between frames of point clouds in the picture, the following method may be executed to eliminate the positioning errors of point clouds between frames.

[0139] Fig. 9 1 is a flow chart of a method for inter-frame positioning and registration proposed in an embodiment of the present application. The method can be executed before the above S720.

[0140] S910: Determine at least one fourth point cloud set, each fourth point cloud set including two adjacent frames in the multiple frames of fourth point clouds.

[0141] In some possible embodiments, if the number of frames of the fourth point cloud determined by the corresponding point cloud processing flow is N frames, then based on the N-frame fourth point cloud, N-1 fourth point cloud sets can be determined. For example, the i-th fourth point cloud set in the N-1 fourth point cloud sets may include the i-th frame and the i+1-th frame in the N-frame fourth point cloud, where i is an integer greater than 0 and less than N.

[0142] S920: Performing registration operations on two frames of fourth point clouds included in at least one fourth point cloud set in sequence through a point cloud registration algorithm to determine a positioning calibration matrix.

[0143] In some possible embodiments, when the determined fourth point cloud is N frames, the above registration operation may be as follows: align the fourth point cloud of the i-th frame with the fourth point cloud of the i+1-th frame, where i is an integer greater than or equal to 1 and less than N. The specific registration method may use a pre-trained neural network model for registration, or may adopt an ICP algorithm for registration. Based on the above registration method, it can be known that the above N frames of fourth point clouds are all registered based on the positioning of the fourth point cloud of the first frame, and the positioning calibration matrix determined after the registration operation is used to represent the coordinate conversion coefficients of the fourth point clouds of each frame except the fourth point cloud of the first frame compared to the point cloud of the first frame.

[0144] S930: Adjust the position information of the fourth point cloud of the multiple frames according to the positioning calibration matrix.

[0145] In some possible embodiments, the position information of the fourth point cloud in the multi-frames may be the coordinate value of the point cloud in the picture.

[0146] Based on the above technical solution, it is helpful to reduce the inter-frame point cloud positioning error of the fourth point cloud of multiple frames, and ensure the accuracy of the subsequent determination of the static base map.

[0147] In some possible embodiments, the above-mentioned method of inter-frame positioning and registration can also be executed before the above-mentioned S310, and inter-frame positioning and registration are performed on the first point cloud of multiple frames, but the first point cloud includes some point clouds of dynamic targets, which will cause corresponding interference to the inter-frame positioning and registration, and the point cloud data processed in the positioning and registration process is relatively large, which also increases the complexity of the method. Therefore, under normal circumstances, after determining the fourth point cloud of multiple frames, inter-frame positioning and registration can be performed on the fourth point cloud of multiple frames.

[0148] In some possible embodiments, considering that the point cloud collected by the vehicle is usually affected by noise, the following operations may be performed before the above S350: determine the first ground point cloud in the third point cloud of each frame, and denoise the first ground point cloud in the third point cloud of each frame, wherein the first ground point cloud is the ground point cloud within the first range around the first target frame. Of course, if the first target frame is not marked in the third point cloud of a certain frame, it means that there is no dynamic target in the third point cloud of the frame, and there is no need to mark the first target frame, and there is no need to determine the first ground point cloud of the third point cloud of the frame and denoise the point cloud.

[0149] In some possible embodiments, the first range may be a preset reasonable interval range.

[0150] Fig.10Schematic diagram of a first range proposed in an embodiment of the present application. The first range is illustrated based on a three-dimensional Cartesian coordinate system.

[0151] refer to Fig.10 As shown, the first range may be the range of the first target frame projected onto the ground in the Z-axis direction, and the ground points within the range are the first ground point cloud.

[0152] Fig.11 2 is a schematic diagram of another first range proposed in the embodiment of the present application. The first range is illustrated based on a three-dimensional Cartesian coordinate system.

[0153] refer to Fig.11 As shown, the first range may be a further expanded range based on the range of the first target frame projected onto the ground in the Z-axis direction, and the ground points within the range are the first ground point cloud.

[0154] Based on the above technical solution, denoising operations can be performed on the ground point cloud of key ground areas, which can reduce the impact of noise on point cloud annotation while taking into account the complexity of the algorithm.

[0155] In some possible embodiments, in the process of labeling the dynamic targets in the above-mentioned multi-frame second point cloud, it is also necessary to identify the motion state of the labeled dynamic targets, and the motion state includes a moving state and a stationary state.

[0156] It should be understood that although the identified targets, such as vehicles, pedestrians, etc., have corresponding attribute labels of dynamic targets, the motion state of these dynamic targets is not necessarily in a mobile state. For example, some vehicles are parked on the side of the road. In this case, dynamic targets in this state can be identified as stationary dynamic targets by the above-mentioned dynamic target detection model.

[0157] It should be understood that, because in S710, the point cloud used to characterize the dynamic target in the first point cloud of multiple frames needs to be removed through the second neural network model, and the dynamic target will block the ground points below it, so when the first point cloud of the i-th frame includes a dynamic target, assuming that the dynamic target is located at the first position, after the point cloud of the dynamic target is removed from the first point cloud of the i-th frame, the corresponding fourth point cloud of the i-th frame will have a point cloud gap at the above-mentioned first position, which originally includes not only the point cloud of the dynamic target, but also the ground points blocked by the dynamic target. Then, when the motion states of the dynamic targets included in the first point cloud of multiple frames are different, the ground point integrity of the static base map finally obtained based on the fourth point cloud of multiple frames is also different.

[0158] Fig.12 It is a schematic diagram of a static base map proposed in an embodiment of the present application.

[0159] Assume that the static base map is determined by superimposing the fourth point cloud of the i-th frame and the fourth point cloud of the i+1-th frame, and for the convenience of description, Fig.12 The first and fourth point clouds of the i-th frame, the first and fourth point clouds of the i+1-th frame, and the static base map shown in are all captured from point clouds corresponding to the same part of the road, and the motion state of the identified dynamic target is moving. It can be seen from the fourth point cloud of the i-th frame that the dynamic target is located at the first position and blocks the ground point at the first position. However, in the fourth point cloud of the i+1-th frame, since the position of the dynamic target moves to the second position, the originally blocked ground point will be exposed again. Therefore, after superimposing the fourth point clouds of the i-th frame and the i+1-th frame, there will be no point cloud gaps at the ground point at the corresponding first position in the static base map.

[0160] Fig.13 It is a schematic diagram of another static base map proposed in an embodiment of the present application.

[0161] Assume that the static base map is determined by superimposing the fourth point cloud of the i-th frame and the fourth point cloud of the i+1-th frame, and for the convenience of description, Fig.13 The first point cloud and the fourth point cloud of the i-th frame, the first point cloud and the fourth point cloud of the i+1-th frame, and the static base map shown in are all intercepted from the point clouds corresponding to the same part of the road, and the motion state of the identified dynamic target is static. It can be seen from the fourth point cloud of the i-th frame that the dynamic target is located at the first position and blocks the ground point at the first position. In the first point cloud of the i+1-th frame, the position of the dynamic target remains unchanged, and the originally blocked ground point will not be exposed. Therefore, after superimposing the fourth point cloud of the i-th frame and the i+1-th frame, there will be a point cloud gap at the ground point at the first position. The static base map obtained by superimposing the fourth point cloud with point cloud gaps will also have missing ground points at the first position corresponding to the static base map. Even after executing the above S340, the ground points of the obtained multi-frame third point cloud will have point cloud gaps at the first position. Then, there will be no ground point below the first target frame corresponding to the static dynamic target in the multi-frame third point cloud. Therefore, in this scenario, the Z-axis coordinate of the lower edge of the first target frame can also be adjusted to the Z-axis coordinate of the ground point closest to the lower edge of the first target frame, so as to complete the automatic grounding of the first target frame.

[0162] In some possible embodiments, when the dynamic target in the first target frame is in a stationary state, the geometric parameters of the first target frame can be expanded, for example, by increasing the length, width and height of the first target frame, thereby expanding the volume of the first target frame. When the first target frame is expanded to include surrounding ground points, the first target frame is automatically aligned to the ground based on these ground points. The specific automatic alignment method is detailed in the above corresponding embodiment, which will not be repeated here.

[0163] In addition, an embodiment of the present application also provides a device for implementing any of the above methods. For example, a point cloud annotation device is provided, which includes a unit (or means) for implementing any of the above point cloud annotation methods.

[0164] Fig.14 1 is a schematic block diagram of a point cloud annotation device 1400 provided in an embodiment of the present application. Fig.14 As shown, the smart device 1400 includes: a first processing unit 1410 and a second processing unit 1420, wherein the first processing unit 1410 is used to: input multiple frames of first point clouds into a first neural network model, and output multiple frames of second point clouds, wherein the multiple frames of first point clouds are point clouds acquired by the vehicle in a first period of time for representing road information, and the multiple frames of second point clouds include point clouds for representing dynamic targets and first target frames annotated for the dynamic targets, and the dynamic targets are located in the first target frames; and determine a static base map according to the multiple frames of first point clouds;

[0165] The second processing unit 1420 is used to: perform ground segmentation on the static base map to determine the ground points of the static base map; fill in the ground points of multiple frames of second point clouds according to the ground points of the static base map to determine multiple frames of third point clouds; and adjust the bottom edge height of the first annotation box in the multiple frames of third point clouds based on the ground points in the multiple frames of third point clouds.

[0166] In some possible embodiments, the second processing unit is specifically used to: establish a first coordinate system based on a static base map, and proportionally map the first coordinate system to multiple frames of second point clouds; based on the first coordinate system, map the ground points of the static base map to multiple frames of second point clouds respectively.

[0167] In some possible embodiments, the above-mentioned first processing unit is specifically used to: input multiple frames of first point clouds into a second neural network model, output multiple frames of fourth point clouds, the fourth point clouds eliminating point clouds used to characterize dynamic targets compared to the first point cloud; superimpose multiple frames of fourth point clouds to determine a static base map.

[0168] In some possible embodiments, before the first processing unit 1410 superimposes multiple frames of fourth point clouds to determine a static base map, the first processing unit 1410 is also used to: determine at least one fourth point cloud set, each fourth point cloud set including two adjacent frames in the multiple frames of fourth point clouds; perform registration operations on two frames of fourth point clouds included in at least one fourth point cloud set in turn through a point cloud registration algorithm to determine a positioning calibration matrix; and adjust the position information of the multiple frames of fourth point clouds according to the positioning calibration matrix.

[0169] In some possible embodiments, the first coordinate system is a three-dimensional Cartesian coordinate system, the Z axis of the first coordinate system is used to represent the height, and the second processing unit 1420 is specifically used to adjust the Z axis coordinate of the lower edge of the first target frame to the Z axis coordinate of the ground point closest to the lower edge of the first target frame.

[0170] In some possible embodiments, when the number of identified dynamic targets is M, the number of first target frames is also M accordingly.

[0171] In some possible embodiments, before the first processing unit 1410 determines the static base map based on the multiple frames of first point clouds, the first processing unit 1410 is also used to: determine at least one first point cloud set, each first point cloud set includes two adjacent frames in the multiple frames of first point clouds; sequentially determine the similarity between the two frames of first point clouds included in the at least one first point cloud set; when the similarity between the two frames of first point clouds included in the first point cloud set is greater than a preset first threshold, retain the first point cloud with a larger number of points in the first point cloud set.

[0172] Fig.15 It is a schematic block diagram of a computer device 1500 provided in an embodiment of the present application. Fig.15 The computer device 1500 shown includes a memory 1510, a processor 1520 and a bus 1540. Optionally, the computer device 1500 also includes a communication interface 1530. The memory 1510, the processor 1520 and the communication interface 1530 are connected to each other through the bus 1540.

[0173] The memory 1510 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1510 may store a program. When the program stored in the memory 1510 is executed by the processor 1520, the processor 1520 is used to execute each step of the point cloud annotation method of the embodiment of the present application. Specifically, the processor 1520 may execute the above Figure 3 The method shown.

[0174] Processor 1520 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the point cloud annotation method proposed in the method embodiment of the present application.

[0175] The processor 1520 may also be an integrated circuit chip having signal processing capabilities. During implementation, each step of the point cloud annotation method of the present application may be completed through an integrated logic circuit of hardware in the processor 1520 or instructions in software form.

[0176] The processor 1520 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method involved in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1510, and the processor 1520 reads the information in the memory 1510 and completes it in combination with its hardware. Fig.14 The functions required to be performed by the units included in the device shown, or the functions required to perform the method embodiments of the present application Figure 3 The method shown.

[0177] The communication interface 1530 uses a transceiver device such as but not limited to a transceiver to implement communication between the apparatus 1500 and other devices or a communication network.

[0178] The bus 1540 may include a path for transmitting information between the various components of the device 1500 (eg, the memory 1510 , the processor 1520 , and the communication interface 1530 ).

[0179] It should be understood that although the above device 1500 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device 1500 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 1500 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 1500 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Fig.15 All devices shown in .

[0180] It should be understood that the processor in the embodiment of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0181] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0182] Fig.16 A schematic block diagram of a computer-readable storage medium provided in an embodiment of the present application. Fig.16The computer-readable storage medium 1600 shown in the figure stores computer instructions 1610. When the computer instructions 1610 are executed by a processor, the method corresponding to the above embodiment can be implemented.

[0183] In some possible embodiments, the computer-readable storage medium 1600 may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media sets. The available media may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0184] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0189] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory, a magnetic disk or an optical disk.

[0190] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for point cloud annotation, characterized in that: Applied to a vehicle, the method comprises: Inputting the multi-frame first point cloud into a first neural network model, and outputting a multi-frame second point cloud, wherein the multi-frame first point cloud is a point cloud acquired by the vehicle in a first period of time and used to represent road information, and the multi-frame second point cloud includes a point cloud used to represent a dynamic target and a first target frame corresponding to the dynamic target, and the dynamic target is located in the first target frame; Determine a static base map according to the multiple frames of first point clouds; Performing ground segmentation on the static base map to determine ground points of the static base map; Filling the ground points of the multiple frames of second point clouds respectively according to the ground points of the static base map to determine the multiple frames of third point clouds; Based on the ground points in the multi-frame third point cloud, the bottom edge heights of the first annotation boxes in the multi-frame third point cloud are adjusted respectively.

2. The method according to claim 1, characterized in that: The filling of the ground points of the plurality of frames of the second point cloud respectively according to the ground points of the static base map comprises: Establishing a first coordinate system based on the static base map, and mapping the first coordinate system to the multi-frame second point cloud; Based on the first coordinate system, the ground points of the static base map are respectively mapped to the multiple frames of second point clouds.

3. The method according to claim 1 or 2, characterized in that: Determining the static base map according to the first point clouds of the multiple frames includes: Inputting the multiple frames of first point clouds into a second neural network model, and outputting multiple frames of fourth point clouds, wherein the fourth point clouds exclude point clouds used to characterize the dynamic target compared to the first point clouds; The multiple frames of fourth point clouds are superimposed to determine the static base map.

4. The method according to claim 3, characterized in that Before superimposing the multiple frames of fourth point clouds to determine the static base map, the method further includes: Determine at least one fourth point cloud set, each of the fourth point cloud sets includes two adjacent frames in the multiple frames of fourth point clouds; By using a point cloud registration algorithm, sequentially performing registration operations on two frames of fourth point clouds included in the at least one fourth point cloud set to determine a positioning calibration matrix; According to the positioning calibration matrix, position information of the multi-frame fourth point cloud is adjusted.

5. The method according to claim 4, characterized in that The first coordinate system is a three-dimensional Cartesian coordinate system, the Z axis of the first coordinate system is used to represent the height, and the adjusting the lower edge height of the first annotation box in the multi-frame third point cloud based on the ground points in the multi-frame third point cloud comprises: The Z-axis coordinate of the lower edge of the first target frame is adjusted to the Z-axis coordinate of the ground point closest to the lower edge of the first target frame.

6. The method according to any one of claims 1 to 5, characterized in that When the number of identified dynamic targets is M, the number of the first target frames is also M accordingly.

7. The method according to any one of claims 1 to 6, characterized in that Before determining the static base map according to the first point clouds of the multiple frames, the method further includes: Determine at least one first point cloud set, each of the first point cloud sets includes two adjacent frames in the multiple frames of first point clouds; sequentially determining the similarity between two frames of first point clouds included in the at least one first point cloud set; When the similarity between two frames of first point clouds included in the first point cloud set is greater than a preset first threshold, the first point cloud with a larger number of points in the first point cloud set is retained.

8. A point cloud annotation device, characterized in that: Applied to a vehicle, the device comprises: a first processing unit and a second processing unit, The first processing unit is used to: input the multi-frame first point cloud into a first neural network model, and output a multi-frame second point cloud, wherein the multi-frame first point cloud is a point cloud acquired by the vehicle in a first period of time and used to characterize road information, and the multi-frame second point cloud includes a point cloud used to characterize a dynamic target and a first target frame annotated for the dynamic target, and the dynamic target is located in the first target frame; determine a static base map according to the multi-frame first point cloud; The second processing unit is used to: perform ground segmentation on the static base map to determine the ground points of the static base map; fill in the ground points of the multiple frames of second point clouds according to the ground points of the static base map to determine the multiple frames of third point clouds; and adjust the bottom edge height of the first annotation box in the multiple frames of third point clouds based on the ground points in the multiple frames of third point clouds.

9. The device according to claim 8, characterized in that The second processing unit is specifically used for: Establishing a first coordinate system based on the static base map, and mapping the first coordinate system to the multi-frame second point cloud; Based on the first coordinate system, the ground points of the static base map are respectively mapped to the multiple frames of second point clouds.

10. The device according to claim 8 or 9, characterized in that The first processing unit is specifically configured to: Inputting the multiple frames of first point clouds into a second neural network model, and outputting multiple frames of fourth point clouds, wherein the fourth point clouds exclude point clouds used to characterize the dynamic target compared to the first point clouds; The multiple frames of fourth point clouds are superimposed to determine the static base map.

11. The device according to claim 10, characterized in that Before the first processing unit superimposes the multiple frames of fourth point clouds to determine the static base map, the first processing unit is further used to: Determine at least one fourth point cloud set, each of the fourth point cloud sets includes two adjacent frames in the multiple frames of fourth point clouds; By using a point cloud registration algorithm, sequentially performing registration operations on two frames of fourth point clouds included in the at least one fourth point cloud set to determine a positioning calibration matrix; According to the positioning calibration matrix, position information of the multi-frame fourth point cloud is adjusted.

12. The device according to claim 11, characterized in that The first coordinate system is a three-dimensional Cartesian coordinate system, the Z axis of the first coordinate system is used to represent the height, and the second processing unit is specifically used to: The Z-axis coordinate of the lower edge of the first target frame is adjusted to the Z-axis coordinate of the ground point closest to the lower edge of the first target frame.

13. The device according to any one of claims 8 to 12, characterized in that When the number of identified dynamic targets is M, the number of the first target frames is also M accordingly.

14. The device according to any one of claims 8 to 13, characterized in that Before the first processing unit determines the static base map according to the multiple frames of the first point cloud, the first processing unit is further used to: Determine at least one first point cloud set, each of the first point cloud sets includes two adjacent frames in the multiple frames of first point clouds; sequentially determining the similarity between two frames of first point clouds included in the at least one first point cloud set; When the similarity between two frames of first point clouds included in the first point cloud set is greater than a preset first threshold, the first point cloud with a larger number of points in the first point cloud set is retained.

15. A point cloud annotation device, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute the method according to any one of claims 1 to 7.

16. A chip system, characterized in that: The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method as described in any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 7.

18. A computer program product, characterized in that When the computer program code or instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 7.

19. A vehicle, characterized in that: Comprising a device as claimed in any one of claims 8 to 15.

Images