Point cloud generation method, electronic equipment and storage medium

Abstract

The invention discloses a point cloud generation method, electronic equipment and a storage medium, and the method comprises the steps: determining static points in each point cloud frame for continuous point cloud frames; obtaining a global mapping result based on static point mapping of the continuous point cloud frame; expanding the points of the dynamic target in each point cloud frame to obtain the expanded points of the dynamic target; and on the basis of the global mapping result, combining the static points in each point cloud frame and the expanded points of the dynamic target to obtain a final point cloud of each point cloud frame. The purpose of accurately determining the complete point cloud in the point cloud frame aiming at the self-defined data set is achieved, a basis is provided for simply and rapidly generating dense grid unit occupation labels, manual labeling is not needed, and labeling cost is saved.

Description

Technical Field

[0001] This application relates to the technical field of occupancy label generation, and more particularly to a method for generating point clouds, an electronic device, and a storage medium. Background Art

[0002] In autonomous driving perception tasks, most traditional 3D scene understanding methods focus on 3D object detection, making it difficult to describe real-world objects with arbitrary shapes and infinite categories. The 3D Occupancy Network is a new type of perception network. Briefly speaking, it divides the space around the vehicle into a series of grid cells, then defines which grid cells are occupied and which are free, and obtains a simple 3D space representation by predicting the occupancy probability of each grid cell in the 3D space, so as to more comprehensively achieve 3D scene perception.

[0003] Therefore, how to determine the occupancy of each grid cell in the 3D space is the core problem to be solved. Before solving this core problem, it is necessary to accurately determine the complete point cloud in the point cloud frame, that is, accurately determining the complete point cloud in the point cloud frame is a prerequisite for determining the occupancy of each grid cell in the 3D space.

[0004] In view of this, this application is specifically proposed. Summary of the Invention

[0005] The following gives a brief overview of one or more aspects to provide a basic understanding of these aspects. This overview is not an exhaustive survey of all contemplated aspects, and is neither intended to identify key or decisive elements of all aspects nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that follows.

[0006] This application provides a method for generating point clouds, an electronic device, and a storage medium, achieving the purpose of accurately determining the complete point cloud in the point cloud frame for a custom dataset, providing a basis for simply and quickly generating dense grid cell occupancy labels, without manual annotation, and saving annotation costs.

[0007] In a first aspect, this application provides a method for generating point clouds, including the following steps:

[0008] For continuous point cloud frames, determine the static points in each point cloud frame;

[0009] Based on the static points of the continuous point cloud frames, build a map to obtain a global mapping result;

[0010] Expand the points of the dynamic objects in each point cloud frame to obtain the expanded points of the dynamic objects;

[0011] Based on the global mapping result, merge the static points in each point cloud frame and the points of the expanded dynamic objects to obtain the final point cloud of each point cloud frame.

[0012] Further, the expansion of the points of the dynamic objects in each point cloud frame to obtain the points of the expanded dynamic objects includes:

[0013] Traverse each point cloud frame in the continuous point cloud frames, and determine a reference point cloud frame group associated with each point cloud frame respectively, where the number of point cloud frames in the reference point cloud frame group is less than the number of point cloud frames in the continuous point cloud frames;

[0014] For the current point cloud frame, expand the points of the dynamic objects in the current point cloud frame based on the points of the dynamic objects in the reference point cloud frame group associated with the current point cloud frame to obtain the points of the expanded dynamic objects corresponding to the current point cloud frame.

[0015] Further, the traversing each point cloud frame in the continuous point cloud frames and determining a reference point cloud frame group associated with each point cloud frame respectively includes:

[0016] For a point cloud frame in the continuous point cloud frames, determine a set number of point cloud frames with scanning times before the scanning time of the point cloud frame and closest to the scanning time of the point cloud frame as the reference point cloud frame group associated with the point cloud frame;

[0017] Or, determine a set number of point cloud frames with scanning times after the scanning time of the point cloud frame and closest to the scanning time of the point cloud frame as the reference point cloud frame group associated with the point cloud frame;

[0018] Or, centered on the scanning time of the point cloud frame, determine a set number of point cloud frames from the point cloud frames with scanning times before and after the scanning time of the point cloud frame as the reference point cloud frame group associated with the point cloud frame.

[0019] Further, the expanding the points of the dynamic objects in the current point cloud frame based on the points of the dynamic objects in the reference point cloud frame group associated with the current point cloud frame to obtain the points of the expanded dynamic objects corresponding to the current point cloud frame includes:

[0020] For a dynamic object in the current point cloud frame, project the points belonging to the dynamic object in each point cloud frame in the reference point cloud frame group associated with the current point cloud frame to the target coordinate system of the dynamic object to obtain the intermediate points of the dynamic object;

[0021] Reproject the intermediate points of the dynamic object from the target coordinate system to the ego-vehicle coordinate system of the current point cloud frame to obtain the reprojection points;

[0022] Take the points among the reprojection points that fall within the detection box of the one dynamic target corresponding to the current point cloud frame as the extended points of the one dynamic target;

[0023] Take the original points within the detection box and the extended points as the points of the extended one dynamic target corresponding to the current point cloud frame.

[0024] Further, based on the global mapping result, merge the static points in each point cloud frame and the points of the extended dynamic targets in each point cloud frame to obtain the final point cloud of each point cloud frame, including:

[0025] For the current point cloud frame, project the points in the global mapping result into the ego-vehicle coordinate system of the current point cloud frame to obtain projected points;

[0026] Determine the projected points that fall within the set spatial range as the final static points in the current point cloud frame;

[0027] Determine the final static points and the points of the extended dynamic targets corresponding to the current frame as the final point cloud of the current point cloud frame.

[0028] Further, for the continuous point cloud frames, determining the static points in each point cloud frame includes:

[0029] Input each point cloud frame in the continuous point cloud frames into a 3D object detection model to obtain the detection boxes of the dynamic targets in each point cloud frame;

[0030] Remove the points that fall within the detection boxes of the dynamic targets to obtain the static points in each point cloud frame.

[0031] Further, it further includes: determining the occupancy labels of each grid cell of the 3D occupancy network based on the final point cloud of each point cloud frame.

[0032] Further, determining the occupancy labels of each grid cell of the 3D occupancy network based on the final point cloud of each point cloud frame includes:

[0033] Based on the final point cloud of each point cloud frame, obtain the reconstructed 3D points through the Poisson reconstruction algorithm;

[0034] Map the reconstructed 3D points to the grid cells of the 3D occupancy network, and add class labels to the grid cells occupied by points through the nearest neighbor algorithm.

[0035] In a second aspect, the present application also provides a point cloud generation device, including:

[0036] A first determination module, configured to determine the static points in each point cloud frame for continuous point cloud frames;

[0037] A mapping module, configured to map based on the static points of the continuous point cloud frames to obtain a global mapping result;

[0038] An expansion module for expanding the points of dynamic objects in each point cloud frame to obtain the points of the expanded dynamic objects;

[0039] A merging module for merging the static points in each point cloud frame and the points of the expanded dynamic objects based on the global mapping result to obtain the final point cloud of each point cloud frame.

[0040] In a third aspect, the present application further provides an electronic device, which includes:

[0041] One or more processors;

[0042] A storage device for storing one or more programs;

[0043] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating a point cloud as described above.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method for generating a point cloud as described above is implemented.

[0045] The method for generating a point cloud disclosed in the present application, by determining the static points in each point cloud frame for continuous point cloud frames; constructing a map based on the static points of the continuous point cloud frames to obtain a global mapping result; expanding the points of dynamic objects in each point cloud frame to obtain the points of the expanded dynamic objects; and merging the static points in each point cloud frame and the points of the expanded dynamic objects based on the global mapping result to obtain the final point cloud of each point cloud frame, realizes the purpose of accurately determining the complete point cloud in the point cloud frame for a custom dataset, provides a basis for simply and quickly generating dense grid cell occupancy labels, and saves the annotation cost without manual annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0047] Figure 1 It is a schematic flowchart of a method for generating a point cloud provided by an embodiment of the present application;

[0048] Figure 2 It is a schematic diagram of determining a reference point cloud frame group associated with each point cloud frame provided by an embodiment of the present application;

[0049] Figure 3 Schematic diagram of a process for aggregating dynamic target point clouds provided by an embodiment of the present application;

[0050] Figure 4 Schematic diagram of a process for aggregating dynamic point clouds and static point clouds provided by an embodiment of the present application;

[0051] Figure 5 Overall framework diagram of occupancy label generation provided by an embodiment of the present application;

[0052] Figure 6 Schematic diagram of the structure of a point cloud generation device provided by an embodiment of the present application;

[0053] Figure 7 Schematic diagram of the structure of an electronic device in an embodiment of the present application. Detailed implementation manners

[0054] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention, rather than limiting the invention. Additionally, it should be noted that for the sake of description, only parts related to the invention are shown in the drawings.

[0055] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and embodiments.

[0056] Figure 1 Schematic diagram of a process for a point cloud generation method proposed by the present application. This method is applicable to the scenario of accurately determining the complete point cloud in a point cloud frame for a custom dataset, providing a basis for simply and quickly generating dense grid cell occupancy labels without manual annotation, saving the annotation cost. Specifically, the point cloud generation method can be executed by a point cloud generation device, and the point cloud generation device can be implemented in the form of software and / or hardware and integrated into an electronic device.

[0057] As Figure 1 shown, the point cloud generation method includes the following steps:

[0058] S110. For continuous point cloud frames, determine the static points in each point cloud frame.

[0059] Among them, the continuous point cloud frames can refer to a set of 3D point clouds within a certain range above, below, front, back, left, and right of the vehicle itself, and can be obtained by a vehicle-mounted lidar scanning at a certain scanning frequency during vehicle driving.

[0060] The static points in the point cloud frame refer to the scanning points for static objects. For example, the scanning points on the road surface, the scanning points of trees, the scanning points of utility poles, etc. belong to static points.

[0061] Exemplarily, the determining of the static points in each point cloud frame for the continuous point cloud frames includes:

[0062] Input each point cloud frame in the continuous point cloud frame into the 3D target detection model to obtain the detection frame of the dynamic target in each point cloud frame; remove the points falling in the detection frame of the dynamic target to obtain the static points in each point cloud frame. That is, input the point cloud frames one by one into the 3D target detection model to obtain the detection frame of the dynamic target. The points within the detection frame are the points of the dynamic target, and the points outside the detection frame are the points of the static target, that is, the static points. Therefore, by removing the points falling in the detection frame of the dynamic target, the static points can be obtained.

[0063] Specifically, the detection frame of the dynamic target output by the 3D target detection model may include: the position coordinates of the center of the detection frame, the length, width, height and other information of the detection frame, so that a specific detection frame can be formed in the three-dimensional space, and the point cloud points falling inside the detection frame are removed, and the remaining are the static points of the static object.

[0064] S120, constructing a static point map based on the continuous point cloud frames to obtain a global mapping result.

[0065] Exemplarily, based on the static points of the continuous point cloud frames, three-dimensional reconstruction is performed through the SLAM (Simultaneous Localization and Mapping) algorithm to obtain a global mapping result.

[0066] The essence of 3D reconstruction of static points in continuous point cloud frames through SLAM algorithm is to calculate the pose of each frame relative to the initial frame (i.e. relative pose), and then project the static points of other frames to the initial frame according to the relative pose to obtain the global mapping result. Therefore, the static points in the global mapping result are relatively dense, laying the foundation for generating dense occupancy labels of 3D occupancy network.

[0067] S130 , expanding the points of the dynamic target in each point cloud frame to obtain the expanded points of the dynamic target.

[0068] In order to generate the occupancy labels of the dense 3D occupancy network, in addition to obtaining denser static points based on static points, denser dynamic points can also be obtained based on dynamic points. Therefore, the points of the dynamic target in each point cloud frame can be expanded to obtain the expanded points of the dynamic target.

[0069] Exemplarily, the step of expanding the points of the dynamic target in each point cloud frame to obtain the expanded points of the dynamic target includes the following sub-steps:

[0070] 131. Traverse each point cloud frame in the continuous point cloud frames, and determine a reference point cloud frame group associated with each point cloud frame respectively. The number of point cloud frames in the reference point cloud frame group is less than the number of point cloud frames in the continuous point cloud frames.

[0071] Specifically, for a point cloud frame in the continuous point cloud frames, determine a set number of point cloud frames with scanning times before the scanning time of the point cloud frame and closest to the scanning time of the point cloud frame as the reference point cloud frame group associated with the point cloud frame. For example, for the 10th point cloud frame, a total of 9 frames including the 1st point cloud frame, the 2nd point cloud frame, the 3rd point cloud frame, the 4th point cloud frame, the 5th point cloud frame, the 6th point cloud frame, the 7th point cloud frame, the 8th point cloud frame, and the 9th point cloud frame can be formed into the reference point cloud frame group associated with the 10th point cloud frame.

[0072] Alternatively, determine a set number of point cloud frames with scanning times after the scanning time of the point cloud frame and closest to the scanning time of the point cloud frame as the reference point cloud frame group associated with the point cloud frame. For example, for the 10th point cloud frame, a total of 9 frames including the 11th point cloud frame, the 12th point cloud frame, the 13th point cloud frame, the 14th point cloud frame, the 15th point cloud frame, the 16th point cloud frame, the 17th point cloud frame, the 18th point cloud frame, and the 19th point cloud frame can be formed into the reference point cloud frame group associated with the 10th point cloud frame.

[0073] Alternatively, with the scanning time of the point cloud frame as the center, determine a set number of point cloud frames from the point cloud frames with scanning times before and after the scanning time of the point cloud frame as the reference point cloud frame group associated with the point cloud frame. For example, for the 10th point cloud frame, a total of 9 frames including the 11th point cloud frame, the 12th point cloud frame, the 13th point cloud frame, the 14th point cloud frame, the 15th point cloud frame, the 9th point cloud frame, the 8th point cloud frame, the 7th point cloud frame, and the 6th point cloud frame can be formed into the reference point cloud frame group associated with the 10th point cloud frame.

[0074] Particularly, the point cloud frames in the reference point cloud frame group associated with a point cloud frame can also be discontinuous point cloud frames.

[0075] Generally speaking, assume that the point cloud acquisition vehicle has driven on the road for 5 minutes and has collected a total of 2000 point cloud frames (i.e., there are 2000 continuous point cloud frames in total). By determining the reference point cloud frame group associated with each frame respectively, assuming that the reference point cloud frame group associated with a point cloud frame includes 9 point cloud frames, plus this point cloud frame, there are a total of 10 point cloud frames. Then, generating corresponding occupancy labels for each point cloud frame only requires performing relevant processing (such as the processing represented by step 132) on 10 point cloud frames, rather than on 2000 point cloud frames. In this way, the computational complexity of the relevant processing can be reduced, the memory occupancy can be reduced, and the processing speed can be improved.

[0076] By determining a reference point cloud frame group associated with each frame respectively from the continuous point cloud frames, the computational amount of subsequent processing can be reduced, the memory occupation can be reduced, and the processing speed can be improved.

[0077] Correspondingly, referring to Figure 2 As shown in a schematic diagram of determining a reference point cloud frame group associated with each point cloud frame respectively, for the dynamic target tracking results 210 of M frames and the continuous point cloud frames 220 of M frames, a frame division operation 230 is performed, and they are divided into groups of N point cloud frames each, where N is much smaller than M, so as to reduce the computational amount of related processing, reduce the memory occupation, and improve the processing speed.

[0078] 132. For the current point cloud frame, based on the points of the dynamic target in the reference point cloud frame group associated with the current point cloud frame, the points of the dynamic target in the current point cloud frame are expanded to obtain the points of the expanded dynamic target corresponding to the current point cloud frame.

[0079] Exemplarily, for a dynamic target in the current point cloud frame, the points of the dynamic target belonging to the dynamic target in each point cloud frame in the reference point cloud frame group associated with the current point cloud frame are projected onto the target coordinate system of the dynamic target to obtain the intermediate points of the dynamic target. That is, the points of the same dynamic target in other point cloud frames are projected onto the target coordinate system of the dynamic target in the current point cloud frame, and the points obtained by projection are marked as the intermediate points. Among them, the points in the current point cloud frame are represented in the ego-vehicle coordinate system (usually, the ego-vehicle coordinate system takes the center of the rear axle of the vehicle as the origin, the direction pointing to the front of the vehicle as the positive x-axis, the left as the positive y-axis, and the up as the positive z-axis). By the center point coordinates and the heading angle of the detection box of the dynamic target, the translation vector and rotation matrix of the ego-vehicle relative to the dynamic target can be determined. Through the translation vector and rotation matrix, the points can be transformed from the ego-vehicle coordinate system to the target coordinate system of the dynamic target (the target coordinate system takes the center point of the detection box of the corresponding dynamic target as the origin, and the orientation of the dynamic target as the positive x-axis, the left as the positive y-axis, and the up as the positive z-axis).

[0080] The intermediate points of the dynamic target are reprojected from the target coordinate system to the ego-vehicle coordinate system of the current point cloud frame to obtain reprojection points. The points obtained by reprojection are marked as reprojection points.

[0081] The points among the reprojection points that fall within the detection box of the dynamic target corresponding to the current point cloud frame are used as the expansion points of the dynamic target; the original points within the detection box and the expansion points are used as the points of the expanded dynamic target corresponding to the current point cloud frame. In this way, the purpose of expanding the points of the dynamic target is achieved, making the points of the dynamic target denser, laying a foundation for generating dense occupancy labels.

[0082] Generally speaking, the purpose of step 132 is to increase the number of points of a dynamic target by matching the point clouds of the same dynamic target that appear in consecutive frames in the case of sparse point cloud points of the dynamic target. Since the dynamic target will move, this solution proposes a conversion solution using the target coordinate system of an instance target (i.e., a specific dynamic target) as a medium. As Figure 3 shown in a schematic diagram of the process of aggregating the point cloud of a dynamic target, first, perform spatial calculations on the points in each point cloud frame (in the ego vehicle coordinate system of the current point cloud frame) 310 and the dynamic target tracking results (in the ego vehicle coordinate system of the current point cloud frame, including the center point coordinates, length, width, height, heading angle, target instance ID, and target category of the target detection box) 320; determine whether the points in the point cloud frame are within the detection box of the dynamic target (step 330), if not, it means this point is not a dynamic point and is directly discarded (step 340). The obtained dynamic points after screening are corresponded one by one according to the target tracking results to obtain the points, categories, instance IDs, and instance detection boxes of N-frame dynamic instance targets (step 350). At this time, the points are in the position in the ego vehicle coordinate system. Through the center point coordinates and heading angle of the detection box, the translation vector T and rotation matrix R of the ego vehicle relative to the target can be obtained, and the dynamic points are transformed from the ego vehicle coordinate system to the target coordinate system through the translation vector T and rotation matrix R. Since the instance ID is unique, the points belonging to the same instance ID can be merged. At this time, the point sets of all different instance ID dynamic targets within the N-frame period (in the target coordinate system) are obtained (step 360). Traverse the N frames again. For example, take the nth frame, compare the dynamic target instance ID in the nth frame with the IDs of all different instances within the N-frame period just obtained. If they belong to the same ID, then project all the dynamic sets of this ID back to the ego vehicle coordinate system of the nth frame (ego_n) through the translation vector T' and rotation matrix R' of the ego vehicle relative to the target obtained from the center point coordinates and heading angle of the detection box in the nth frame (step 370), calculate again whether the projected points are within the detection box of the current instance target (step 380), perform refined screening, discard if not (step 390), and finally retain all the points belonging to the dynamic target in the nth frame and the category of this point (step 3100). The category of the point is the category of the corresponding target. After the traversal ends, the information of all dynamic target points in the N frames (including the coordinate positions of the points and the categories of the targets to which they belong) is obtained (step 3200).

[0083] S140. Based on the global mapping result, merge the static points in each point cloud frame and the points of the expanded dynamic target to obtain the final point cloud of each point cloud frame.

[0084] Specifically, the dynamic point cloud and static points after the expansion of the current frame are aggregated to obtain the entire point cloud of the current frame. The n-th frame pose is obtained. The pose is obtained through static point cloud mapping. It records the pose change of the ego-vehicle coordinates of each frame (ego_n) relative to the initial frame (ego_0) during mapping. The global mapping result (map) is also obtained through static point cloud mapping (based on the coordinate system of the ego_0 frame). The static point cloud obtained by the global mapping result has a higher density than the single-frame static point cloud. Through the n-th frame pose (ego_n), the points in the global mapping result are converted to the ego_n coordinate system. Since the distance range in the global mapping result is large, static points that are too far away are not required for the current frame. Therefore, the distance range of the front, back, left, right, up, down (x, y, z axis) of the ego-vehicle can be customized, such as 50 meters front and back, 30 meters left and right, 5 meters up and down, etc. Finally, the points in the range are assigned to the same static category and merged with the points of the dynamic target of the current frame to obtain the final point cloud, including the position coordinates of the points and the category they belong to. Figure 4 A schematic diagram of a process for aggregating dynamic point clouds and static points is shown, including: 410, all dynamic target points in N frames. 420, obtaining the pose of the corresponding frame. 430, inter-frame pose. 440, converting the global mapping point to the current frame ego-vehicle coordinate system according to the pose. 450, global mapping result. 460, range filtering and setting the static category. That is, marking all static points as static categories without subdividing the category attributes. 470, point cloud merging.

[0085] In general, for the current point cloud frame, the points in the global mapping result are projected to the vehicle coordinate system of the current point cloud frame to obtain projection points; the projection points falling within the set spatial range are determined as the final static points in the current point cloud frame; the points of the expanded dynamic target corresponding to the final static points and the current frame are determined as the final point cloud of the current point cloud frame.

[0086] Furthermore, based on the final point cloud of each point cloud frame, the purpose of quickly generating dense occupancy labels can be achieved without manual labeling, and the method can be fully automated. Specifically, the method further includes: determining the occupancy labels of each grid unit of the 3D occupancy network based on the final point cloud of each point cloud frame.

[0087] Exemplarily, based on the final point cloud of each point cloud frame, reconstructed three-dimensional points are obtained through a Poisson reconstruction algorithm; the reconstructed three-dimensional points are mapped to grid cells of a 3D occupancy network, and category labels are added to the grid cells occupied by the points through a nearest neighbor algorithm.

[0088] Among them, the Poisson reconstruction algorithm is an algorithm for reconstructing a continuous three-dimensional surface from a discrete point cloud. Specifically, based on the point cloud and its normal direction, it infers the shape of the continuous surface formed by these point clouds. The surface model formed by the reconstructed three-dimensional points can more intuitively display the shape of an object or a scene and can better reflect the true geometric shape of the object.

[0089] Optionally, in addition to using the Poisson reconstruction algorithm, a deep learning-based reconstruction algorithm, an octree reconstruction algorithm, or a moving least squares method can also be used for three-dimensional reconstruction of the point cloud.

[0090] The method for generating a point cloud disclosed in the embodiments of the present application, on the basis of saving the annotation cost, uses the target basic information to realize the simple and rapid generation of dense occupancy labels on a custom dataset. There is no need for manual annotation and it is completely automated.

[0091] Based on the above embodiments, referring to Figure 5 the overall framework diagram of an occupancy label generation shown as follows, specifically including: 510, continuous point cloud frames; 520, dynamic target deduction; 530, static point cloud; 540, static point cloud mapping; 550, global mapping result; 560, inter-frame pose; 570, point cloud 3D detection model; 580, target tracking; 590, dynamic target tracking result; 5100, dynamic target point cloud aggregation; 5110, static and dynamic point cloud merging; 5120, filling holes; 5130, occupancy label generation.

[0092] As Figure 5 shown, a dynamic target detection box is obtained through the point cloud 3D detection model, and the points of the laser point cloud in the dynamic target detection box are deducted from the original laser point cloud frame to obtain a static point cloud frame. Through static point cloud mapping, a global mapping result and the pose information of each frame are generated. The dynamic target detection box obtains the ID information of the target instance tracking through target tracking. Then, the original laser point cloud and the dynamic target tracking result pass through the dynamic selection module to obtain grouped data. The grouped data is subjected to dynamic target point cloud aggregation to obtain the aggregated dynamic target point cloud of each frame and the category information of the corresponding points. At the same time, the global mapping result is subjected to pose transformation using the pose of the corresponding frame to obtain the static point cloud of each frame. The categories of the static point cloud are not distinguished. After merging the dynamic point cloud and the static point cloud, the blank voxel space is filled through the Poisson reconstruction and the nearest neighbor algorithm, and finally a dense occupancy label is obtained.

[0093] Figure 6 is a schematic structural diagram of a device for generating a point cloud provided by an embodiment of the present application. The device includes: a first determination module 610, a mapping module 620, an expansion module 630, and a merging module 640.

[0094] Among them, the first determination module 610 is used to determine the static points in each point cloud frame for consecutive point cloud frames; the mapping module 620 is used to construct a map based on the static points of the consecutive point cloud frames to obtain a global mapping result; the expansion module 630 is used to expand the points of the dynamic targets in each point cloud frame to obtain the expanded points of the dynamic targets; the merging module 640 is used to merge the static points in each point cloud frame and the expanded points of the dynamic targets based on the global mapping result to obtain the final point cloud of each point cloud frame.

[0095] Further, the expansion module 630 includes a determination unit, which is used to traverse each point cloud frame in the consecutive point cloud frames to determine a reference point cloud frame group associated with each point cloud frame respectively, and the number of point cloud frames in the reference point cloud frame group is less than the number of point cloud frames in the consecutive point cloud frames; an expansion unit, which is used to expand the points of the dynamic target in the current point cloud frame based on the points of the dynamic target in the reference point cloud frame group associated with the current point cloud frame to obtain the expanded points of the dynamic target corresponding to the current point cloud frame.

[0096] Further, the determination unit is specifically used for: for a point cloud frame in the consecutive point cloud frames, determining a set number of point cloud frames with scanning times before the scanning time of the point cloud frame and closest to the scanning time of the point cloud frame as the reference point cloud frame group associated with the point cloud frame;

[0097] Or, determining a set number of point cloud frames with scanning times after the scanning time of the point cloud frame and closest to the scanning time of the point cloud frame as the reference point cloud frame group associated with the point cloud frame;

[0098] Or, taking the scanning time of the point cloud frame as the center, determining a set number of point cloud frames from the point cloud frames with scanning times before and after the scanning time of the point cloud frame as the reference point cloud frame group associated with the point cloud frame.

[0099] Further, the expansion unit is specifically used for: for a dynamic target in the current point cloud frame, projecting the points belonging to the dynamic target in each point cloud frame in the reference point cloud frame group associated with the current point cloud frame into the target coordinate system of the dynamic target to obtain the intermediate points of the dynamic target; re-projecting the intermediate points of the dynamic target from the target coordinate system into the ego-vehicle coordinate system of the current point cloud frame to obtain re-projected points; taking the points among the re-projected points that fall within the detection box of the dynamic target corresponding to the current point cloud frame as the expansion points of the dynamic target; taking the original points within the detection box and the expansion points as the expanded points of the dynamic target corresponding to the current point cloud frame.

[0100] Further, the merging module 640 is specifically configured to project the points in the global mapping result onto the ego-vehicle coordinate system of the current point cloud frame for the current point cloud frame to obtain projected points; determine the projected points falling within a set spatial range as the final static points in the current point cloud frame; and determine the final static points and the points of the augmented dynamic targets corresponding to the current frame as the final point cloud of the current point cloud frame.

[0101] Further, the first determination module 610 is specifically configured to input each point cloud frame in the continuous point cloud frames into a 3D object detection model respectively to obtain detection boxes of dynamic targets in each point cloud frame; remove the points falling within the detection boxes of the dynamic targets to obtain the static points in each point cloud frame.

[0102] Further, it further includes: a second determination module, configured to determine the occupancy labels of each grid cell of the 3D occupancy network based on the final point cloud of each point cloud frame.

[0103] Further, the second determination module is specifically configured to: based on the final point cloud of each point cloud frame, obtain the reconstructed three-dimensional points through the Poisson reconstruction algorithm; map the reconstructed three-dimensional points to the grid cells of the 3D occupancy network, and add class labels to the grid cells occupied by points through the nearest neighbor algorithm.

[0104] The point cloud generation device provided by the embodiments of the present disclosure can execute the steps in the point cloud generation method provided by the method embodiments of the present disclosure, and the implementation steps and beneficial effects are not described herein again.

[0105] Figure 7 It is a schematic structural diagram of an electronic device in the embodiments of the present disclosure. Specifically refer to Figure 7 , which shows a schematic structural diagram of the electronic device 500 suitable for implementing the embodiments of the present disclosure. Figure 7 The electronic device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0106] As Figure 7 shown, the electronic device 500 may include a processing device (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) or the program loaded from the storage device 508 into the random access memory (RAM) to implement the method of the embodiments as described in the present disclosure. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. The I / O interface 505 is also connected to the bus 504. The input device 506, the output device 507, the storage device 508, and the communication device 509 are all connected to the I / O interface 505.

[0107] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium. The computer program includes program code for performing the method shown in the flowchart, thereby implementing the method for generating a point cloud as described above. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by a processing device 501, the above functions defined in the method of the embodiment of the present disclosure are executed.

[0108] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0109] The above computer-readable medium may be included in the above electronic device; or it may exist separately without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device executes the method steps in the present application.

[0110] Optionally, when one or more of the above programs are executed by the electronic device, the electronic device may further execute the other steps described in the above embodiments.

[0111] In the context of the present disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0112] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features having similar functions disclosed in the present disclosure.

[0113] Specific examples are used herein to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. The above is only the preferred implementation manner of the present application. It should be noted that due to the limited nature of written expression and the objectively infinite specific structures, for those of ordinary skill in the art, without departing from the principles of the present application, several improvements, refinements, or changes can be made, or the above technical features can be combined in an appropriate manner; these improvements, refinements, changes, or combinations, or directly applying the inventive concept and technical solution to other occasions without improvement, should all be regarded as the protection scope of the present application.

Claims

1. A method for generating a point cloud, characterized in that: include: For continuous point cloud frames, determine the static points in each point cloud frame; Based on static point mapping of continuous point cloud frames, global mapping results are obtained; Expanding the points of the dynamic target in each point cloud frame to obtain the expanded points of the dynamic target; Based on the global mapping result, the static points in each point cloud frame and the points of the expanded dynamic target are merged to obtain the final point cloud of each point cloud frame.

2. The method for generating a point cloud according to claim 1, characterized in that: The step of expanding the points of the dynamic target in each point cloud frame to obtain the expanded points of the dynamic target includes: Traversing each point cloud frame in the continuous point cloud frames, determining a reference point cloud frame group associated with each point cloud frame, wherein the number of point cloud frames in the reference point cloud frame group is less than the number of point cloud frames in the continuous point cloud frames; For the current point cloud frame, the points of the dynamic target in the current point cloud frame are expanded based on the points of the dynamic target in the reference point cloud frame group associated with the current point cloud frame to obtain the expanded points of the dynamic target corresponding to the current point cloud frame.

3. The method for generating a point cloud according to claim 2, characterized in that: The traversing each point cloud frame in the continuous point cloud frames to determine a reference point cloud frame group respectively associated with each point cloud frame includes: For a point cloud frame in the continuous point cloud frames, determining a set number of point cloud frames whose scanning time is before the scanning time of the point cloud frame and closest to the scanning time of the point cloud frame as a reference point cloud frame group associated with the point cloud frame; Alternatively, a set number of point cloud frames whose scanning time is after the scanning time of the point cloud frame and closest to the scanning time of the point cloud frame are determined as a reference point cloud frame group associated with the point cloud frame; Alternatively, taking the scanning time of the point cloud frame as the center, a set number of point cloud frames are determined from point cloud frames whose scanning times are before and after the scanning time of the point cloud frame as a reference point cloud frame group associated with the point cloud frame.

4. The method for generating a point cloud according to claim 2, characterized in that: The method of expanding the points of the dynamic target in the current point cloud frame based on the points of the dynamic target in the reference point cloud frame group associated with the current point cloud frame to obtain the expanded points of the dynamic target corresponding to the current point cloud frame includes: For a dynamic target in the current point cloud frame, projecting points belonging to the dynamic target in each point cloud frame in the reference point cloud frame group associated with the current point cloud frame to the target coordinate system of the dynamic target to obtain the middle point of the dynamic target; Reprojecting the middle point of the dynamic target from the target coordinate system to the vehicle coordinate system of the current point cloud frame to obtain a reprojection point; Taking points among the reprojected points that fall within the detection frame of the dynamic target corresponding to the current point cloud frame as expansion points of the dynamic target; The original point in the detection frame and the expanded point are used as the expanded point of the dynamic target corresponding to the current point cloud frame.

5. The method for generating a point cloud according to claim 1, characterized in that: Based on the global mapping result, the static points in each point cloud frame and the expanded points of each dynamic target are merged to obtain the final point cloud of each point cloud frame, including: For the current point cloud frame, project the points in the global mapping result to the self-vehicle coordinate system of the current point cloud frame to obtain projection points; The projection point falling within the set spatial range is determined as the final static point in the current point cloud frame; The final static point and the point of the expanded dynamic target corresponding to the current frame are determined as the final point cloud of the current point cloud frame.

6. The method for generating a point cloud according to claim 1, characterized in that: The step of determining the static points in each point cloud frame for the continuous point cloud frames includes: Inputting each point cloud frame in the continuous point cloud frames into the 3D target detection model to obtain a detection frame of the dynamic target in each point cloud frame; The points falling within the detection frame of the dynamic target are removed to obtain the static points in each point cloud frame.

7. The method for generating a point cloud according to claim 1, characterized in that: Also includes: The occupancy label of each grid cell of the 3D occupancy network is determined based on the final point cloud of each point cloud frame.

8. The method for generating a point cloud according to claim 7, characterized in that: The determining of the occupancy labels of each grid unit of the 3D occupancy network based on the final point cloud of each point cloud frame includes: Based on the final point cloud of each point cloud frame, the reconstructed 3D points are obtained through the Poisson reconstruction algorithm; The reconstructed 3D points are mapped to the grid cells of the 3D occupancy network, and category labels are added to the grid cells occupied by the points through the nearest neighbor algorithm.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the point cloud generation method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the point cloud generation method as described in any one of claims 1 to 8 is implemented.

Images