Point cloud processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to a point cloud processing method and apparatus, an electronic device and a computer readable storage medium. The method comprises the steps of determining multi-frame point cloud data of a target scene; the multi-frame point cloud data comprises semantic categories of points in multi-frame point clouds and a plurality of dynamic target areas; on the basis of semantic categories of the points, static point cloud data in the multiple frames of point cloud data are spliced, point cloud data corresponding to the multiple dynamic target areas are aggregated, and fused point cloud data are determined; and performing voxelization processing on the fused point cloud data, and determining voxel information corresponding to the fused point cloud data. According to the invention, high-efficiency automatic universal occupied grid production can be realized.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a point cloud processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] The universal occupancy grid is an approximation of the real physical world. It divides the three-dimensional space into small cubes with a certain accuracy, and annotates each cube with its semantic information, speed information and other attributes, thereby providing the ego vehicle with information such as whether the surrounding area is passable, the speed and category of dynamic objects, etc., to assist the ego vehicle in path planning. The universal occupancy grid can reflect the space where general obstacles are occupied but not annotated, and perceive the impact of speed changes in some areas of large targets on the passable area. Therefore, research on algorithms such as perception and planning of the universal occupancy grid is attracting more and more attention in the industry.

[0003] The generation of a universal occupancy grid is a complex and time-consuming process with complex dependencies between related modules, a long data flow process, and intensive calculations, which reduces the generation efficiency of the universal occupancy grid and the efficiency of generating voxelized information of the point cloud. Summary of the invention

[0004] One of the objects of the present invention is to provide a point cloud processing method, which can improve the generation efficiency of the general occupancy grid and improve the efficiency of generating voxel information of the point cloud; the second object is to provide a point cloud processing device; the third object is to provide an electronic device; the fourth object is to provide a computer-readable storage medium.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A point cloud processing method, comprising:

[0007] Determine multi-frame point cloud data of the target scene; the multi-frame point cloud data includes semantic categories of points and multiple dynamic target areas;

[0008] Based on the semantic categories of the points, the static point cloud data in the multiple frames of point cloud data are spliced ​​and the point cloud data corresponding to the multiple dynamic target areas are aggregated to determine the fused point cloud data;

[0009] The fused point cloud data is voxelized to determine voxel information corresponding to the fused point cloud data.

[0010] By determining multi-frame point cloud data of the target scene; the multi-frame point cloud data includes semantic categories of points and multiple dynamic target areas; based on the semantic categories of points, the static point cloud data in the multi-frame point cloud data is spliced ​​and the point cloud data corresponding to multiple dynamic target areas are aggregated to determine the fused point cloud data; the fused point cloud data is voxelized to determine the voxel information corresponding to the fused point cloud data, thereby realizing the efficient generation of streaming point cloud voxelization information, thereby improving the production efficiency of the general occupancy grid and the voxelization efficiency of the point cloud data.

[0011] Further, based on the semantic category of the points, the static point cloud data in the multiple frames of point cloud data are spliced ​​and the point cloud data corresponding to the multiple dynamic target areas are aggregated to determine the fused point cloud data, including:

[0012] According to the semantic categories of the points, dynamic semantic points are removed from the multiple frames of point cloud data to determine multiple frames of static point cloud data;

[0013] Splicing the multiple frames of static point cloud data to determine static spliced ​​point cloud data;

[0014] as well as,

[0015] Based on the semantic categories of the points, target aggregation is performed on the point cloud data corresponding to the multiple dynamic target areas to determine at least one dynamically aggregated point cloud data;

[0016] The fused point cloud data is determined by combining the at least one dynamically aggregated point cloud data with the statically spliced ​​point cloud data.

[0017] It realizes the extraction of static scenes and dynamic target areas based on multi-frame point cloud data, and performs corresponding stitching and target aggregation to generate fused point cloud data, thus realizing a universal point cloud static and dynamic fusion solution. Based on the fused point cloud data, it further realizes the full automation of the point cloud voxelization process and improves the voxelization efficiency.

[0018] Further, the multiple frames of static point cloud data include: current frame static point cloud data and at least one frame of non-current frame static point cloud data; the step of splicing the multiple frames of static point cloud data to determine the static spliced ​​point cloud data includes:

[0019] Determine at least one pose transformation matrix of the at least one frame of non-current frame static point cloud data corresponding to the current frame static point cloud data;

[0020] According to the at least one pose transformation matrix, the at least one frame of non-current frame static point cloud data is pose transformed, and the at least one non-current frame static point cloud data after pose transformation is spliced ​​with the current frame static point cloud data to determine the static spliced ​​point cloud data.

[0021] By performing pose transformation on at least one frame of point cloud data that is not the current frame and then splicing it with the static point cloud data of the current frame, the distortion of the static scene between the point cloud data collected at different times can be reduced, the accuracy of the static spliced ​​point cloud data can be improved, and the accuracy of the subsequent voxelization process can be further improved.

[0022] Further, the multiple dynamic target areas include multiple areas corresponding to at least one dynamic target object in the multiple frames of point cloud data; the at least one dynamic target object corresponds to at least one target identifier; and the point cloud data corresponding to the multiple dynamic target areas are subjected to target aggregation based on the semantic category of the point to determine at least one dynamically aggregated point cloud data, including:

[0023] Determine, according to the at least one target identifier, a target identifier of a point in the point cloud data corresponding to each of the multiple dynamic target areas;

[0024] Aggregating points with the same target identifier in the point cloud data corresponding to the multiple dynamic target areas to determine at least one initial dynamic aggregated point cloud data corresponding to the at least one dynamic target object;

[0025] For each dynamically aggregated point cloud data in the at least one initial dynamically aggregated point cloud data, semantic category normalization is performed on the points in the each initial dynamically aggregated point cloud data according to the semantic category of the point to determine the at least one dynamically aggregated point cloud data.

[0026] By grouping the dynamic target areas in the point cloud data of different frames into the same category according to the target identification, the fusion of different point cloud data of dynamic target objects collected at different times is achieved, thereby improving the accuracy of the point cloud data corresponding to the dynamic target objects in the subsequent fused point cloud data and the comprehensiveness of the information expression.

[0027] Furthermore, before combining the at least one dynamically aggregated point cloud data with the statically spliced ​​point cloud data to determine the fused point cloud data, the method further includes:

[0028] Symmetry completion and / or enhancement processing is performed on the at least one dynamically aggregated point cloud data.

[0029] Symmetry completion and / or enhancement processing is performed on at least one dynamically aggregated point cloud data, so as to improve the accuracy of the at least one dynamically aggregated point cloud data, and further improve the accuracy of a subsequent voxelization process.

[0030] Further, the combining the at least one dynamically aggregated point cloud data with the statically spliced ​​point cloud data to determine the fused point cloud data includes:

[0031] Determine the position of at least one target area in the static spliced ​​point cloud data according to the position of at least one dynamic target area in the current frame point cloud data in the multiple frames of point cloud data;

[0032] According to the at least one target area position, the at least one dynamically aggregated point cloud data is fused into the static spliced ​​point cloud data to determine the fused point cloud data.

[0033] By fusing at least one dynamically aggregated point cloud data into the static spliced ​​point cloud data according to the position of at least one target area, fused point cloud data is obtained, and the static scene and dynamic target area in the multi-frame point cloud data are fused into one fused point cloud data, so that the fused point cloud data can be further voxelized, thereby improving the comprehensiveness and accuracy of the information of the voxelized point cloud.

[0034] Further, the voxelization processing is performed on the fused point cloud data to determine the voxel information corresponding to the fused point cloud data, including:

[0035] Performing voxel division on the fused point cloud data to determine a plurality of voxels;

[0036] synchronously calculating a unique voxel coordinate corresponding to each voxel in the plurality of voxels;

[0037] Based on the voxel unique coordinates corresponding to each voxel, the voxel information corresponding to each voxel is determined, thereby determining the voxel information corresponding to the fused point cloud data.

[0038] By synchronously calculating the unique voxel coordinates corresponding to each voxel in multiple voxels, parallel voxel coordinate calculation is achieved, which improves processing efficiency and further improves the efficiency of point cloud voxelization.

[0039] Further, determining the voxel information corresponding to each voxel based on the unique voxel coordinates corresponding to each voxel includes:

[0040] Synchronously processing each voxel in the plurality of voxels, calculating a mean value corresponding to each non-integer data according to at least one non-integer data corresponding to a point contained in each voxel, and determining at least one non-integer voxel information corresponding to each voxel;

[0041] The voxel information corresponding to each voxel is determined according to the voxel unique coordinates corresponding to each voxel and at least one non-integer voxel information.

[0042] By synchronously calculating at least one non-integer voxel information corresponding to each voxel in a plurality of voxels, parallel voxel information calculation is achieved, processing efficiency is improved, and thus point cloud voxelization efficiency is improved.

[0043] Further, determining the voxel information corresponding to each voxel based on the unique voxel coordinates corresponding to each voxel includes:

[0044] Synchronously processing each voxel in the plurality of voxels, and determining the integer data with the largest number of occurrences in each type of integer data according to at least one type of integer data corresponding to the points contained in each voxel, thereby determining at least one type of integer voxel information corresponding to each voxel;

[0045] The voxel information corresponding to each voxel is determined according to the voxel unique coordinates corresponding to each voxel and at least one integer voxel information.

[0046] By synchronously calculating at least one integer voxel information corresponding to each voxel in a plurality of voxels, parallel voxel information calculation is achieved, processing efficiency is improved, and thus point cloud voxelization efficiency is improved.

[0047] Furthermore, at least one non-integer data includes: at least one of speed and / or reflection intensity; at least one integer data includes: at least one of semantic category and / or target identification.

[0048] Further, based on the semantic category of the points, the static point cloud data in the multiple frames of point cloud data are spliced ​​and the point cloud data corresponding to the multiple dynamic target areas are aggregated to determine the fused point cloud data, including:

[0049] Aggregating point cloud data corresponding to the plurality of dynamic target areas through a first computing process, and writing the obtained at least one dynamically aggregated point cloud data into a shared storage space;

[0050] Through the main process, the static point cloud data in the multiple frames of point cloud data are spliced, and the at least one dynamically aggregated point cloud data is obtained from the shared storage space, and is merged with the spliced ​​static spliced ​​point cloud data to determine the merged point cloud data;

[0051] The voxelization processing of the fused point cloud data to determine the voxel information corresponding to the fused point cloud data includes:

[0052] By means of the main process, the fused point cloud data is divided into voxels, and a plurality of voxels obtained by the division are written into the shared storage space;

[0053] Through the second calculation process, the multiple voxels are obtained from the shared storage space, and the voxel information corresponding to each voxel in the multiple voxels is determined, so as to determine the voxel information corresponding to the fused point cloud data.

[0054] Different processes are used to handle dynamic target aggregation, dynamic and static point cloud data fusion, and voxelization of fused point cloud data respectively, decoupling parallel data-free and operation-dependent processes, so that data processing can make full use of computing resources, reduce unnecessary waiting, data calculation and data copying, improve processing efficiency, and thus improve the efficiency of point cloud voxelization.

[0055] Furthermore, the method further comprises:

[0056] Sending a first processing progress to a first monitoring process through the first computing process;

[0057] Sending a second processing progress to a second monitoring process through the second computing process;

[0058] managing the first computing process according to the first processing schedule through the first monitoring process;

[0059] The second computing process is managed according to the second processing schedule through the second supervisory process.

[0060] By monitoring the processing of the computing process through the supervisory process, the situation where the computing process is blocked or the processing is completed can be discovered and handled in a timely manner, thereby improving the processing efficiency and further improving the efficiency of point cloud voxelization.

[0061] A point cloud processing device, comprising:

[0062] A determination module, used to determine multi-frame point cloud data of a target scene; the multi-frame point cloud data includes semantic categories of points and multiple dynamic target areas;

[0063] A fusion module, configured to splice the static point cloud data in the multiple frames of point cloud data and aggregate the point cloud data corresponding to the multiple dynamic target areas based on the semantic categories of the points, to determine fused point cloud data;

[0064] The voxelization module is used to perform voxelization processing on the fused point cloud data to determine the voxel information corresponding to the fused point cloud data.

[0065] By determining multi-frame point cloud data of the target scene; the multi-frame point cloud data includes semantic categories of points and multiple dynamic target areas; based on the semantic categories of points, the static point cloud data in the multi-frame point cloud data is spliced ​​and the point cloud data corresponding to multiple dynamic target areas are aggregated to determine the fused point cloud data; the fused point cloud data is voxelized to determine the voxel information corresponding to the fused point cloud data, thereby realizing the efficient generation of streaming point cloud voxelization information, thereby improving the production efficiency of the general occupancy grid and the voxelization efficiency of the point cloud data.

[0066] Furthermore, the fusion module is further used to remove dynamic semantic points from the multiple frames of point cloud data according to the semantic categories of the points to determine multiple frames of static point cloud data; splice the multiple frames of static point cloud data to determine static spliced ​​point cloud data; and, based on the semantic categories of the points, perform target aggregation on the point cloud data corresponding to the multiple dynamic target areas to determine at least one dynamically aggregated point cloud data;

[0067] The fused point cloud data is determined by combining the at least one dynamically aggregated point cloud data with the statically spliced ​​point cloud data.

[0068] It realizes the extraction of static scenes and dynamic target areas based on multi-frame point cloud data, and performs corresponding stitching and target aggregation to generate fused point cloud data, thus realizing a universal point cloud static and dynamic fusion solution. Based on the fused point cloud data, it further realizes the full automation of the point cloud voxelization process and improves the voxelization efficiency.

[0069] Furthermore, the multi-frame static point cloud data includes: current frame static point cloud data and at least one frame of non-current frame static point cloud data; the fusion module is further used for:

[0070] Determine at least one pose transformation matrix of the static point cloud data of the current frame corresponding to the static point cloud data of the at least one frame of the non-current frame; perform pose transformation on the static point cloud data of the at least one frame of the non-current frame according to the at least one pose transformation matrix, and splice the static point cloud data of the non-current frame after at least one pose transformation with the static point cloud data of the current frame to determine the static spliced ​​point cloud data.

[0071] By performing pose transformation on at least one frame of point cloud data that is not the current frame and then splicing it with the static point cloud data of the current frame, the distortion of the static scene between the point cloud data collected at different times can be reduced, the accuracy of the static spliced ​​point cloud data can be improved, and the accuracy of the subsequent voxelization process can be further improved.

[0072] Furthermore, the multiple dynamic target areas include multiple areas corresponding to at least one dynamic target object in the multiple frames of point cloud data; the at least one dynamic target object corresponds to at least one target identifier; and the fusion module is further used to:

[0073] According to the at least one target identifier, the target identifier of the points in the point cloud data corresponding to each of the multiple dynamic target areas is determined; the points with the same target identifier in the point cloud data corresponding to the multiple dynamic target areas are aggregated to determine at least one initial dynamic aggregated point cloud data corresponding to the at least one dynamic target object; for each dynamic aggregated point cloud data in the at least one initial dynamic aggregated point cloud data, the points in each initial dynamic aggregated point cloud data are normalized according to the semantic category of the points to determine the at least one dynamic aggregated point cloud data.

[0074] By grouping the dynamic target areas in the point cloud data of different frames into the same category according to the target identification, the fusion of different point cloud data of dynamic target objects collected at different times is achieved, thereby improving the accuracy of the point cloud data corresponding to the dynamic target objects in the subsequent fused point cloud data and the comprehensiveness of the information expression.

[0075] Furthermore, the fusion module is also used to combine the at least one dynamically aggregated point cloud data with the static spliced ​​point cloud data, and before determining the fused point cloud data, perform symmetry completion and / or enhancement processing on the at least one dynamically aggregated point cloud data.

[0076] Symmetry completion and / or enhancement processing is performed on at least one dynamically aggregated point cloud data, so as to improve the accuracy of the at least one dynamically aggregated point cloud data, and further improve the accuracy of a subsequent voxelization process.

[0077] Furthermore, the fusion module is also used for:

[0078] According to the position of at least one dynamic target area in the current frame point cloud data in the multi-frame point cloud data, determine the position of at least one target area in the static spliced ​​point cloud data; according to the at least one target area position, fuse the at least one dynamic aggregated point cloud data into the static spliced ​​point cloud data to determine the fused point cloud data.

[0079] By fusing at least one dynamically aggregated point cloud data into the static spliced ​​point cloud data according to the position of at least one target area, fused point cloud data is obtained, and the static scene and dynamic target area in the multi-frame point cloud data are fused into one fused point cloud data, so that the fused point cloud data can be further voxelized, thereby improving the comprehensiveness and accuracy of the information of the voxelized point cloud.

[0080] Furthermore, the voxelization module is further used for:

[0081] The fused point cloud data is divided into voxels to determine a plurality of voxels; the unique voxel coordinates corresponding to each voxel in the plurality of voxels are simultaneously calculated; based on the unique voxel coordinates corresponding to each voxel, the voxel information corresponding to each voxel is determined, thereby determining the voxel information corresponding to the fused point cloud data.

[0082] By synchronously calculating the unique voxel coordinates corresponding to each voxel in multiple voxels, parallel voxel coordinate calculation is achieved, which improves processing efficiency and further improves the efficiency of point cloud voxelization.

[0083] Furthermore, the voxelization module is further used for:

[0084] Each voxel among the multiple voxels is processed synchronously, and based on at least one non-integer data corresponding to the points contained in each voxel, a mean value corresponding to each non-integer data is calculated to determine at least one non-integer voxel information corresponding to each voxel; and based on the voxel unique coordinates corresponding to each voxel and at least one non-integer voxel information, the voxel information corresponding to each voxel is determined.

[0085] By synchronously calculating at least one non-integer voxel information corresponding to each voxel in a plurality of voxels, parallel voxel information calculation is achieved, processing efficiency is improved, and thus point cloud voxelization efficiency is improved.

[0086] Furthermore, the voxelization module is further used for:

[0087] Each voxel among the multiple voxels is processed synchronously, and the integer data with the largest number of occurrences in each integer data is determined according to at least one integer data corresponding to the points contained in each voxel, so as to determine at least one integer voxel information corresponding to each voxel; and the voxel information corresponding to each voxel is determined according to the voxel unique coordinates and at least one integer voxel information corresponding to each voxel.

[0088] By synchronously calculating at least one integer voxel information corresponding to each voxel in a plurality of voxels, parallel voxel information calculation is achieved, processing efficiency is improved, and thus point cloud voxelization efficiency is improved.

[0089] Furthermore, at least one non-integer data includes: at least one of speed and / or reflection intensity; at least one integer data includes: at least one of semantic category and / or target identification.

[0090] Furthermore, the fusion module is also used for:

[0091] Aggregate the point cloud data corresponding to the multiple dynamic target areas through the first computing process, and write the obtained at least one dynamically aggregated point cloud data into a shared storage space; splice the static point cloud data in the multiple frames of point cloud data through the main process, obtain the at least one dynamically aggregated point cloud data from the shared storage space, and fuse it with the spliced ​​static spliced ​​point cloud data to determine the fused point cloud data;

[0092] The voxelization module is further used for:

[0093] Through the main process, the fused point cloud data is voxel-divided, and the multiple voxels obtained by the division are written into the shared storage space; through the second calculation process, the multiple voxels are obtained from the shared storage space, and the voxel information corresponding to each voxel in the multiple voxels is determined, thereby determining the voxel information corresponding to the fused point cloud data.

[0094] Different processes are used to handle dynamic target aggregation, dynamic and static point cloud data fusion, and voxelization of fused point cloud data respectively, decoupling parallel data-free and operation-dependent processes, so that data processing can make full use of computing resources, reduce unnecessary waiting, data calculation and data copying, improve processing efficiency, and thus improve the efficiency of point cloud voxelization.

[0095] Furthermore, the fusion module is also used to send the first processing progress to the first supervision process through the first computing process; send the second processing progress to the second supervision process through the second computing process; manage the first computing process according to the first processing progress through the first supervision process; and manage the second computing process according to the second processing progress through the second supervision process.

[0096] By monitoring the processing of the computing process through the supervisory process, the situation where the computing process is blocked or the processing is completed can be discovered and handled in a timely manner, thereby improving the processing efficiency and further improving the efficiency of point cloud voxelization.

[0097] The present invention provides an electronic device, comprising:

[0098] A memory for storing computer executable instructions;

[0099] The processor is used to implement any one of the point cloud processing methods provided by the present invention when executing the computer executable instructions stored in the memory.

[0100] The present invention provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions or computer programs are executed by a processor, any one of the above-mentioned point cloud processing methods provided by the present invention is implemented.

[0101] Beneficial effects of the present invention:

[0102] By determining multi-frame point cloud data of the target scene; the multi-frame point cloud data includes semantic categories of points and multiple dynamic target areas; based on the semantic categories of points, the static point cloud data in the multi-frame point cloud data is spliced ​​and the point cloud data corresponding to multiple dynamic target areas are aggregated to determine the fused point cloud data; the fused point cloud data is voxelized to determine the voxel information corresponding to the fused point cloud data, thereby realizing the efficient generation of streaming point cloud voxelization information, thereby improving the production efficiency of the general occupancy grid and the voxelization efficiency of the point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 An optional flow chart of the point cloud processing method provided by the present invention;

[0104] Figure 2 It is a schematic diagram of the annotation information in the point cloud data of the present invention;

[0105] Figure 3 An optional flow chart of the point cloud processing method provided by the present invention;

[0106] Figure 4 A schematic diagram of a multi-process streaming processing of the point cloud processing method provided by the present invention;

[0107] Figure 5 A schematic diagram of an optional structure of the point cloud processing device provided by the present invention;

[0108] Figure 6 An optional structural diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0109] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0110] The universal occupancy grid is an approximation of the real physical world. It divides the three-dimensional space into small cubes with a certain accuracy, and annotates each cube with its semantic information, speed information and other attributes, thereby providing the self-driving car with information such as whether the surrounding area is passable, the speed and category of dynamic objects, etc., to assist the self-driving car in path planning. The biggest advantage of the universal occupancy grid is that it can reflect the space where general obstacles are occupied but not annotated, and perceive the impact of speed changes in some areas of large targets on the passable area. Therefore, research on algorithms such as perception and planning of the universal occupancy grid is attracting more and more attention from the industry.

[0111] LiDAR is an important sensor that can be equipped for autonomous vehicles. Its mechanism of emitting lasers and receiving return beams to determine the target position can provide accurate spatial geometric information and reflect the geometric properties of the object. LiDAR collects information and parses it to obtain point cloud data, which is the main truth basis for current 3D target annotation. The dynamic and static information of vehicles, pedestrians, obstacles, etc. on the road are annotated on the point cloud data to obtain their real-world position, posture, and motion information on continuous frames. Combined with point-level semantics, speed and other information, the properties of the corresponding small cubes can be obtained through the point cloud, thereby generating the aforementioned general occupancy grid. The occupancy grid obtained based on point cloud data naturally has strong spatial accuracy and rich semantic information, and therefore has great research value.

[0112] The generation of a universal occupancy grid is a complex and time-consuming process, which includes data acquisition, pose estimation, dynamic target annotation, static target annotation, dynamic target tracking, dynamic target trajectory optimization, point cloud dynamic and static compensation, dynamic point cloud stitching, voxelization, grid post-processing and other modules. The modules have complex dependencies, long data transfer process and intensive calculation. At present, there is no process-based universal occupancy grid production solution for reference, which reduces the generation efficiency of the universal occupancy grid and the efficiency of generating voxel information of point clouds.

[0113] In order to achieve efficient automated universal occupancy grid production, serve automated data production, optimize data flow, and accelerate computing, the present invention provides an efficient occupancy grid truth value production link that can be used for large-scale production of universal occupancy grids and spliced ​​point clouds. The present invention can be applied to complete pose compensation, point cloud splicing, semantic segmentation, target detection, generation of occupancy grid data, etc. on collected and annotated continuous frame data, and can be used in scene reconstruction, training and verification of universal obstacle perception models, as well as the development of spliced ​​point cloud models and high-precision map generation.

[0114] The electronic device provided in the embodiment of the present invention may be implemented as a terminal or a server. In some embodiments, the terminal may include an on-board terminal or a driving computer on a vehicle. The server may include an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, etc., which is selected according to the actual situation and is not limited by the embodiment of the present invention.

[0115] An embodiment of the present invention provides a point cloud processing method, such as Figure 1 As shown, including S101-S103, as follows:

[0116] S101, determining multi-frame point cloud data of a target scene; the multi-frame point cloud data includes semantic categories of points and multiple dynamic target areas.

[0117] In an embodiment of the present invention, an electronic device can obtain multiple frames of point cloud data obtained by collecting a target scene; perform semantic segmentation and target detection on the multiple frames of point cloud data through a semantic segmentation model and a target detection model, obtain the semantic category of each point in each frame of point cloud data, and at least one dynamic target area in each frame of point cloud data, thereby obtaining the semantic category of the points in the multiple frames of point cloud data and multiple frames of point cloud data of dynamic target areas. Among them, the semantic segmentation model can perform semantic category division on the points in each frame of point cloud data to determine the semantic category corresponding to each point. Exemplarily, the semantic category can include the object category of the object in the target scene corresponding to each point, such as the semantic category can include buildings, vehicles, ground, etc. It can be seen that according to the semantic category of the point, it can be indicated whether the object corresponding to the point is a static object or a dynamic object, thereby indicating that the point belongs to a dynamic semantic point or a static semantic point. The target detection model is used to detect dynamic target objects on each frame of point cloud data, detect at least one dynamic target object in each frame of point cloud data, and determine the area occupied by at least one dynamic target object, such as the occupied rectangular area, as at least one dynamic target area, thereby determining that the multiple frames of point cloud data contain multiple dynamic target areas. Exemplarily, the dynamic target object may include a moving object included in the point cloud data, and the dynamic target region may include a label box region corresponding to the moving object generated by the target detection model. Exemplarily, multiple dynamic target regions may be as follows: Figure 2 The area around the vehicle is shown in the box.

[0118] Alternatively, in an embodiment of the present invention, multiple frames of point cloud data of the target scene may be manually annotated by manually annotating the semantic categories of the points in each frame of point cloud data and at least one dynamic target area, thereby acquiring multiple frames of point cloud data.

[0119] S102, based on the semantic category of the points, splicing the static point cloud data in the multi-frame point cloud data and aggregating the point cloud data corresponding to the multiple dynamic target areas to determine the fused point cloud data.

[0120] In an embodiment of the present invention, it can be determined whether a point belongs to a dynamic semantic point or a static semantic point based on the semantic category of the point. For each frame of point cloud data in multiple frames of point cloud data, the static semantic points in each frame of point cloud data can be determined based on the semantic category of the point, and the point cloud data containing the static semantic points can be used as static point cloud data, so that multiple frames of static point cloud data corresponding to the multiple frames of point cloud data can be obtained. According to at least one dynamic target area in each frame of point cloud, the point cloud data corresponding to at least one dynamic target area can be determined, so that the point cloud data corresponding to multiple dynamic target areas can be determined based on the multiple frames of point cloud data. In this way, multiple frames of static point cloud data can be spliced, and point cloud data corresponding to multiple dynamic target areas can be aggregated, so that the static objects and dynamic target objects in the multiple frames of point cloud data can be fused into one frame of point cloud data to obtain fused point cloud data.

[0121] In some embodiments, the processing of static point cloud data and the processing of point cloud data corresponding to the dynamic target area can be performed simultaneously or in any order, which is not limited in the embodiments of the present invention.

[0122] In some embodiments, S102 may be implemented by the following process, including:

[0123] According to the semantic categories of the points, dynamic semantic points are removed from the multi-frame point cloud data to determine the multi-frame static point cloud data;

[0124] Splicing multiple frames of static point cloud data to determine static splicing point cloud data;

[0125] as well as,

[0126] Based on the semantic category of the points, target aggregation is performed on the point cloud data corresponding to the multiple dynamic target areas to determine at least one dynamically aggregated point cloud data;

[0127] At least one dynamically aggregated point cloud data and the statically spliced ​​point cloud data are combined to determine fused point cloud data.

[0128] Among them, for each frame of point cloud data in the multi-frame point cloud data, according to the semantic category of the point, the dynamic semantic points are removed from the multi-frame point cloud data, and the static semantic points are retained as static point cloud data, thereby determining the multi-frame static point cloud data. It can be understood that static point cloud data usually contains information such as the background and static objects in the target scene. Multi-frame static point cloud data includes information such as the background and static objects in the target scene collected at multiple times. The multi-frame static point cloud data is spliced ​​to obtain static spliced ​​point cloud data.

[0129] Among them, for multiple dynamic target areas in multi-frame point cloud data, based on the semantic category of the points, the point cloud data corresponding to the multiple dynamic target areas are subjected to data aggregation and semantic aggregation of the same dynamic target object, thereby completing target aggregation and determining at least one dynamically aggregated point cloud data.

[0130] In some embodiments, the multi-frame static point cloud data includes: current frame static point cloud data, and at least one frame of non-current frame static point cloud data. The current frame static point cloud data includes point cloud data obtained by removing dynamic semantic points from the current frame point cloud data in the multi-frame point cloud data; the at least one frame of non-current frame static point cloud data includes point cloud data obtained by removing dynamic semantic points from at least one frame of non-current frame point cloud data in the multi-frame point cloud data. The above process of splicing the multi-frame static point cloud data and determining the static spliced ​​point cloud data includes:

[0131] Determine at least one pose transformation matrix of at least one frame of static point cloud data other than the current frame corresponding to the static point cloud data of the current frame;

[0132] According to at least one pose transformation matrix, pose transformation is performed on at least one frame of static point cloud data of a non-current frame, and at least one pose transformed static point cloud data of a non-current frame is spliced ​​with the static point cloud data of a current frame to determine static spliced ​​point cloud data.

[0133] In the embodiment of the present invention, since the static scene will move relative to the vehicle coordinate system as the vehicle moves, the Doppler effect will occur, resulting in the distortion of the target shape. In order to correct the distortion, the static point cloud data can be compensated for posture by a posture compensation algorithm. Exemplarily, the posture compensation algorithm can include batch spherical linear interpolation and Einstein summation. The specific selection is made according to the actual situation, and the embodiment of the present invention is not limited.

[0134] Exemplarily, spherical linear interpolation can be used to calculate the pose transformation of each point in the non-current frame static point cloud data relative to the current frame static point cloud data for each frame of non-current frame static point cloud data. In some embodiments, the above algorithm can be batch processed, that is, batch spherical linear interpolation is performed. Through the Einstein summation algorithm, for each frame of non-current frame static point cloud data, a pose transformation matrix corresponding to the non-current frame static point cloud data is generated. Furthermore, according to the pose transformation matrix corresponding to each frame of non-current frame static point cloud data, a pose transformation is performed on each frame of non-current frame static point cloud data, thereby determining at least one non-current frame static point cloud data after pose transformation. At least one non-current frame static point cloud data after pose transformation is spliced ​​with the current frame static point cloud data to determine static spliced ​​point cloud data.

[0135] In some embodiments, the multiple dynamic target areas include multiple areas corresponding to at least one dynamic target object in multiple frames of point cloud data; at least one dynamic target object corresponds to at least one target identifier; that is, at least one dynamic target area corresponds to at least one target identifier. The above process of performing target aggregation on the point cloud data corresponding to the multiple dynamic target areas based on the semantic category of the point and determining at least one dynamically aggregated point cloud data can be implemented by the following process, including:

[0136] According to at least one target identifier, the target identifier of the points in the point cloud data corresponding to each of the multiple dynamic target areas is determined; for the point cloud data corresponding to the multiple dynamic target areas, the points with the same target identifier are aggregated to determine at least one initial dynamic aggregated point cloud data corresponding to at least one dynamic target object; for each dynamic aggregated point cloud data in the at least one initial dynamic aggregated point cloud data, the semantic categories of the points in each initial dynamic aggregated point cloud data are normalized according to the semantics of the points to determine at least one dynamic aggregated point cloud data.

[0137] Among them, the points with the same target identifier in different dynamic target areas represent the points corresponding to the same dynamic target object in different dynamic target areas. For the point cloud data corresponding to multiple dynamic target areas, the points with the same target identifier are aggregated, and the information of the points of the same dynamic target object in multiple dynamic target areas can be aggregated to obtain the initial dynamic aggregated point cloud data corresponding to each dynamic target object, and then obtain at least one initial dynamic aggregated point cloud data corresponding to at least one dynamic target object. Furthermore, for the points contained in each dynamically aggregated point cloud data, the points in each initial dynamically aggregated point cloud data are normalized according to the semantic category of the points: illustratively, the semantic category of each point in each initial dynamically aggregated point cloud data is normalized to the semantic category with the largest number of occurrences in the initial dynamically aggregated point cloud data, thereby completing the semantic category normalization of each dynamically aggregated point cloud data and obtaining at least one dynamically aggregated point cloud data.

[0138] Exemplarily, all dynamic target objects in the multi-frame point cloud data are determined based on the target boxes (dynamic target areas) and semantic labels of the points (semantic categories of the points) annotated in the multi-frame point cloud data, and the semantic categories of the points contained in each dynamic target object are marked based on the semantic labels of the points; the point instance identifier (target identifier of the points) contained in each dynamic target object is determined based on the target identifier of the target box (such as the dynamic box id); the target box category of the target box corresponding to each dynamic target object is determined based on the semantic labels of the points; thus, for each dynamic target object, the semantic categories of the points contained therein, the point instance identifiers and their corresponding target box categories are determined, and a dynamic target truth library corresponding to multiple dynamic target objects is generated based on the above data.

[0139] According to the dynamic target truth library, for all instances with the same points, the corresponding point cloud data are aggregated together to obtain the initial dynamic aggregate point cloud data corresponding to each dynamic target object, and the category normalization is completed according to the semantic label of each point in each initial dynamic aggregate point cloud data: for an initial dynamic aggregate point cloud data, the static semantic points are deleted, and the semantic label of each point is unified into the same semantic label to obtain the dynamic aggregate point cloud data. It can be understood that the semantic category corresponding to the dynamic aggregate point cloud data is the normalized semantic category. Furthermore, the semantic category of the dynamic aggregate point cloud data can also be mapped to the semantic category of the target box corresponding to the dynamic aggregate point cloud data, so as to realize the instance construction of the dynamic target object in the whole life cycle.

[0140] In some embodiments, before combining at least one dynamically aggregated point cloud data with statically spliced ​​point cloud data to determine the fused point cloud data, the method further includes:

[0141] Symmetry completion and / or enhancement processing is performed on at least one dynamically aggregated point cloud data.

[0142] That is to say, after determining at least one dynamic aggregate point cloud data corresponding to at least one dynamic target object, before combining at least one dynamic aggregate point cloud data with static splicing point cloud data to determine the fused point cloud data, the at least one dynamic aggregate point cloud data can be further symmetry completed and / or enhanced to achieve symmetry completion and other enhancement operations on dynamic objects in the point cloud data, so as to improve the data representation capability of the fused point cloud data.

[0143] In some embodiments, based on the above process, at least one target area position can be determined in the static spliced ​​point cloud data according to the position of at least one dynamic target area in the current frame point cloud data in the multi-frame point cloud data; based on the at least one target area position, at least one dynamic aggregated point cloud data is fused into the static spliced ​​point cloud data to determine the fused point cloud data.

[0144] Among them, the position of at least one dynamic target area in the current frame point cloud data represents the respective positions of at least one dynamic target object in the current frame. Exemplarily, the position of at least one dynamic target area may include the center point coordinates or the upper left corner point coordinates of at least one dynamic target area, which is specifically selected according to the actual situation and is not limited in the embodiment of the present invention. According to the position of at least one dynamic target area in the current frame point cloud data, at least one target area position can be correspondingly determined in the spliced ​​static spliced ​​point cloud data, that is, the position of at least one dynamic target object can be determined in the spliced ​​static spliced ​​point cloud data. In this way, according to the position of each dynamic target object (the target area position corresponding to each dynamic target object), at least one dynamic aggregated point cloud data corresponding to at least one dynamic target object can be fused at the corresponding position in the static spliced ​​point cloud data to obtain fused point cloud data.

[0145] It should be noted that in the above-mentioned S101-S102 process, when the semantic category of the point and at least one dynamic target area are obtained by processing multi-frame point cloud data using a semantic segmentation model and a target detection model, the process processing corresponding to S101 and S102 can be placed in the same processing node, and the semantic category of the point and at least one dynamic target area determined in S101 can be directly reused in S102.

[0146] S103: voxelize the fused point cloud data to determine voxel information corresponding to the fused point cloud data.

[0147] At present, for the voxelization of point cloud data, the processing flow of related technologies usually includes:

[0148] 1. Calculate the voxel coordinates of all points in the point cloud and arrange them in ascending order according to the voxel coordinates.

[0149] 2. Scan the voxel coordinates in ascending order, and record the instance id, semantic information, and speed information of points with the same voxel coordinates.

[0150] 3. Each time a different coordinate is scanned, the following operations are performed on the data in the previous coordinate: voting to determine the category information of the voxel, including instance ID and semantics; taking the average, maximum value, etc. to determine the floating-point information of the voxel, such as speed.

[0151] It can be seen that the algorithms of current related technologies must traverse each voxel in the point cloud data in turn, and the calculations are not parallelized, resulting in extremely slow production speeds.

[0152] The embodiment of the present invention provides a point cloud processing method, which can generate accurate voxel information while performing parallel calculations, thus greatly reducing the calculation time. Figure 3As shown, including:

[0153] S201 , performing voxel division on the fused point cloud data to determine a plurality of voxels.

[0154] In S201, the fused point cloud data is divided into voxels to determine a plurality of voxels, each of which includes at least one point in the fused point cloud data.

[0155] S202 , synchronously calculating the unique voxel coordinates corresponding to each voxel in the plurality of voxels obtained by division.

[0156] In S202, for each voxel obtained by division, the voxel unique coordinates corresponding to each voxel can be calculated in parallel, wherein the voxel unique coordinates represent the voxel coordinates of each voxel, and the voxel unique coordinates of each voxel are unique coordinate values, that is, coordinate values ​​that are not repeated with other voxel coordinates in multiple voxels.

[0157] In some embodiments, for each point contained in each voxel, the local coordinates of the point within the voxel can be calculated, and the local coordinates of the points in different voxels can be the same or different. In this way, the position of a point in the fused point cloud data can be determined based on the unique voxel coordinates and the local coordinates of the point.

[0158] It is understandable that in the embodiment of the present application, since the unique voxel coordinates of each voxel are not repeated with other voxels, the voxel information of each voxel can be calculated in parallel to improve processing efficiency. For example, the voxel information of each voxel can be calculated in parallel by using a divergent operator.

[0159] S203 . Based on the unique voxel coordinates corresponding to each voxel, determine the voxel information corresponding to each voxel, thereby determining the voxel information corresponding to the fused point cloud data.

[0160] In the embodiment of the present application, the voxel information corresponding to each voxel at least includes the unique voxel coordinates corresponding to each voxel. Therefore, according to the unique voxel coordinates corresponding to each voxel, the voxel information corresponding to each voxel can be determined, and then the voxel information corresponding to multiple voxels in the fused point cloud data can be determined, that is, the voxel information corresponding to the fused point cloud data can be determined.

[0161] In some embodiments, other voxel information corresponding to each voxel, such as speed, semantic category, reflection intensity, etc., may also be determined based on the points contained in each voxel.

[0162] In some embodiments, the points contained in each voxel correspond to at least one non-integer data. Exemplarily, the at least one non-integer data may include, but is not limited to, at least one of speed and / or reflection intensity. According to the at least one non-integer data corresponding to the points contained in each voxel, the mean corresponding to each non-integer data is calculated to obtain at least one mean corresponding to at least one non-integer data as at least one non-integer voxel information corresponding to each voxel. In this way, each voxel in a plurality of voxels is processed synchronously to determine at least one non-integer voxel information corresponding to each voxel. In this way, for each voxel, the voxel information corresponding to the voxel is determined based on the voxel unique coordinates corresponding to the voxel and at least one non-integer voxel information.

[0163] In some embodiments, the points contained in each voxel correspond to at least one integer data. Exemplarily, the at least one integer data may include, but is not limited to, at least one of a semantic category and / or a target identifier. According to the at least one integer data corresponding to the points contained in each voxel, the integer data with the most occurrences in each integer data can be counted and determined. Exemplarily, in a voting manner, the integer data with the most occurrences in each integer data corresponding to at least one point contained in each voxel, such as the semantic category with the most occurrences, is determined as the non-integer voxel information corresponding to each integer data. Thus, at least one integer voxel information corresponding to each voxel is obtained. In this way, each voxel in a plurality of voxels is processed synchronously to determine at least one integer voxel information corresponding to each voxel. In this way, for each voxel, the voxel information corresponding to the voxel is determined based on the voxel unique coordinates corresponding to the voxel and at least one integer voxel information.

[0164] Exemplarily, at least one piece of non-integer voxel information and / or at least one piece of integer voxel information corresponding to each voxel, as well as the voxel unique coordinates, may be used as the voxel information corresponding to each voxel.

[0165] Exemplarily, the voxel unique coordinates of all points in the fused point cloud data (i.e., the coordinate values ​​obtained by taking all voxel coordinates without repetition) and the subscript corresponding to the unique coordinate of the voxel (the local coordinate of the point in the voxel) are calculated for each point in the fused point cloud data; the mean of the floating-point elements (non-integer data) in the unique coordinate of each voxel is calculated in parallel through the divergence operator; the number of occurrences of each category of the category elements (integer data) in each unique coordinate is calculated in parallel through the divergence operator; the voting category (integer voxel information) in each unique coordinate is obtained according to the maximum number of occurrences of the category elements; the voting category is used as the category information of each point in the voxel through the embedding operator.

[0166] In some embodiments, after determining at least one integer voxel information corresponding to each voxel in the fused point cloud data, at least one integer data of at least one point contained in the voxel may be updated accordingly according to the at least one integer voxel information corresponding to each voxel. In this way, the integer voxel information of the points in each voxel, such as the target identifier or the semantic category, may be unified into the same value.

[0167] It can be understood that the embodiment of the present application determines multi-frame point cloud data of the target scene; the multi-frame point cloud data includes semantic categories of points and multiple dynamic target areas; based on the semantic categories of the points, the static point cloud data in the multi-frame point cloud data are spliced ​​and the point cloud data corresponding to at least one dynamic target area are aggregated to determine the fused point cloud data; the fused point cloud data is voxelized to determine the voxel information corresponding to the fused point cloud data, thereby achieving efficient generation of streaming point cloud voxelization information, thereby improving the production efficiency of the general occupancy grid and the voxelization efficiency of the point cloud data.

[0168] In some embodiments, based on the above point cloud processing method, an embodiment of the present invention provides a multi-process streaming stitching point cloud processing link, which can decouple parallel processes without data and operation dependencies, so that data processing can make full use of computing resources and reduce unnecessary waiting, data calculation and data copying. Among them, the process of stitching static point cloud data in multiple frames of point cloud data and aggregating point cloud data corresponding to multiple dynamic target areas based on the semantic category of the points in the above S102 to determine the fused point cloud data may include:

[0169] Aggregating point cloud data corresponding to a plurality of dynamic target areas through a first computing process, and writing at least one dynamically aggregated point cloud data obtained into a shared storage space;

[0170] Through the main process, the static point cloud data in the multi-frame point cloud data is spliced, and at least one dynamically aggregated point cloud data is obtained from the shared storage space, and is fused with the spliced ​​static spliced ​​point cloud data to determine the fused point cloud data.

[0171] Accordingly, the above-mentioned step S103 performs voxel processing on the fused point cloud data to determine the voxel information corresponding to the fused point cloud data, including:

[0172] Through the main process, the fused point cloud data is divided into voxels, and the multiple voxels obtained by the division are written into the shared storage space;

[0173] Through the second calculation process, a plurality of voxels are obtained from the shared storage space, and voxel information corresponding to each voxel in the plurality of voxels is determined, thereby determining the voxel information corresponding to the fused point cloud data.

[0174] In some embodiments, the first computing process and the second computing process may also be managed by a supervisory process as follows:

[0175] The first processing progress is sent to the first supervision process through the first computing process; the second processing progress is sent to the second supervision process through the second computing process; the first computing process is managed according to the first processing progress through the first supervision process; the second computing process is managed according to the second processing progress through the second supervision process.

[0176] Among them, the first supervisory process can report the processing progress of the point cloud data corresponding to the multiple dynamic target areas aggregated by itself to the first supervisory process through the first task lock. Exemplarily, the processing process of aggregating the point cloud data corresponding to the multiple dynamic target areas may include at least one first task node. Each time the first supervisory process completes a first task node, it reports the currently completed first task node to the first supervisory process through the first task lock. When the first supervisory process determines through the first task lock that the current first task node is processed, it instructs the first computing process to process the next task node. If the first computing process reports that the first task node has timed out, such as the first computing process has abnormally stagnated or been cleared, the first supervisory process reallocates computing resources, clears shared data, and initiates the next processing until at least one first task node is processed. Correspondingly, the second computing process can report the processing progress of at least one second task node corresponding to the process of calculating voxel information to the second supervisory process through the second task lock. The process of the second supervisory process managing the second computing process through the second task lock is similar to the process of the first supervisory process managing the first computing process through the first task lock, which will not be repeated here.

[0177] For example, Figure 4 As shown, the multi-process streaming processing in the point cloud processing method provided by the present invention can be implemented by the following modules: supervisory process 1 (equivalent to the first supervisory process) and supervisory process 2 (equivalent to the second supervisory process), computing process 1 (equivalent to the first computing process) and computing process 2 (equivalent to the second computing process), shared memory (equivalent to shared storage space) and task lock. Among them, supervisory process 1 and supervisory process 2 are responsible for initiating the processing of computing process 1 and computing process 2 respectively. Calculation process 1 and computing process 2 share input data and output data in shared memory through data sharing mechanism. After the processing is completed, they report their own progress to supervisory process 1 and supervisory process 2 respectively through task lock 1 (first task lock) and task lock 2 (second task lock). If supervisory process 1 or supervisory process 2 finds that the task lock is completed, it will instruct computing process 1 or computing process 2 to perform the next step of calculation; if supervisory process 1 or supervisory process 2 finds that the computing process is abnormally stagnant, timed out or killed, it will reallocate computing resources, clear shared data, and initiate the next calculation until all tasks are processed.

[0178] It can be understood that the embodiment of the present invention realizes the accurate and reliable production of occupancy grid true value, which can serve downstream related tasks; through node optimization, single-node high throughput is achieved, data exchange between memory and hard disk is reduced, memory utilization is improved, and data production is efficient. In addition, through the high-performance occupancy grid voxelization solution: the production efficiency of occupancy grid is greatly improved, which is conducive to large-scale production.

[0179] The embodiment of the present invention provides a point cloud processing device 1, such as Figure 5 As shown, including:

[0180] A determination module 11 is used to determine multi-frame point cloud data of a target scene; the multi-frame point cloud data includes semantic categories of points and multiple dynamic target areas;

[0181] A fusion module 12 is used to splice the static point cloud data in the multiple frames of point cloud data and aggregate the point cloud data corresponding to the multiple dynamic target areas based on the semantic categories of the points to determine fused point cloud data;

[0182] The voxelization module 13 is used to perform voxelization processing on the fused point cloud data to determine voxel information corresponding to the fused point cloud data.

[0183] In some embodiments, the fusion module 12 is further used to remove dynamic semantic points from the multiple frames of point cloud data according to the semantic categories of the points to determine multiple frames of static point cloud data; splice the multiple frames of static point cloud data to determine static spliced ​​point cloud data; and, based on the semantic categories of the points, perform target aggregation on the point cloud data corresponding to the multiple dynamic target areas to determine at least one dynamically aggregated point cloud data;

[0184] The fused point cloud data is determined by combining the at least one dynamically aggregated point cloud data with the statically spliced ​​point cloud data.

[0185] In some embodiments, the multiple frames of static point cloud data include: current frame static point cloud data and at least one frame of non-current frame static point cloud data; the fusion module 12 is further used to:

[0186] Determine at least one pose transformation matrix of the static point cloud data of the current frame corresponding to the static point cloud data of the at least one frame of the non-current frame; perform pose transformation on the static point cloud data of the at least one frame of the non-current frame according to the at least one pose transformation matrix, and splice the static point cloud data of the non-current frame after at least one pose transformation with the static point cloud data of the current frame to determine the static spliced ​​point cloud data.

[0187] In some embodiments, the multiple dynamic target areas include multiple areas corresponding to at least one dynamic target object in the multiple frames of point cloud data; the at least one dynamic target object corresponds to at least one target identifier; the fusion module 12 is further used to:

[0188] According to the at least one target identifier, the target identifier of the points in the point cloud data corresponding to each of the multiple dynamic target areas is determined; the points with the same target identifier in the point cloud data corresponding to the multiple dynamic target areas are aggregated to determine at least one initial dynamic aggregated point cloud data corresponding to the at least one dynamic target object; for each dynamic aggregated point cloud data in the at least one initial dynamic aggregated point cloud data, the points in each initial dynamic aggregated point cloud data are normalized according to the semantic category of the points to determine the at least one dynamic aggregated point cloud data.

[0189] In some embodiments, the fusion module 12 is further used to combine the at least one dynamically aggregated point cloud data with the static spliced ​​point cloud data, and perform symmetry completion and / or enhancement processing on the at least one dynamically aggregated point cloud data before determining the fused point cloud data.

[0190] In some embodiments, the fusion module 12 is further configured to:

[0191] According to the position of at least one dynamic target area in the current frame point cloud data in the multi-frame point cloud data, determine the position of at least one target area in the static spliced ​​point cloud data; according to the at least one target area position, fuse the at least one dynamic aggregated point cloud data into the static spliced ​​point cloud data to determine the fused point cloud data.

[0192] In some embodiments, the voxelization module 13 is further configured to:

[0193] The fused point cloud data is divided into voxels to determine a plurality of voxels; the unique voxel coordinates corresponding to each voxel in the plurality of voxels are simultaneously calculated; based on the unique voxel coordinates corresponding to each voxel, the voxel information corresponding to each voxel is determined, thereby determining the voxel information corresponding to the fused point cloud data.

[0194] In some embodiments, the voxelization module 13 is further configured to:

[0195] Each voxel among the multiple voxels is processed synchronously, and based on at least one non-integer data corresponding to the points contained in each voxel, a mean value corresponding to each non-integer data is calculated to determine at least one non-integer voxel information corresponding to each voxel; and based on the voxel unique coordinates corresponding to each voxel and at least one non-integer voxel information, the voxel information corresponding to each voxel is determined.

[0196] In some embodiments, the voxelization module 13 is further configured to:

[0197] Each voxel among the multiple voxels is processed synchronously, and the integer data with the largest number of occurrences in each integer data is determined according to at least one integer data corresponding to the points contained in each voxel, so as to determine at least one integer voxel information corresponding to each voxel; and the voxel information corresponding to each voxel is determined according to the voxel unique coordinates and at least one integer voxel information corresponding to each voxel.

[0198] In some embodiments, at least one non-integer data includes: at least one of speed and / or reflection intensity; at least one integer data includes: at least one of semantic category and / or target identification.

[0199] In some embodiments, the fusion module 12 is further configured to:

[0200] Aggregate the point cloud data corresponding to the multiple dynamic target areas through the first computing process, and write the obtained at least one dynamically aggregated point cloud data into a shared storage space; splice the static point cloud data in the multiple frames of point cloud data through the main process, obtain the at least one dynamically aggregated point cloud data from the shared storage space, and fuse it with the spliced ​​static spliced ​​point cloud data to determine the fused point cloud data;

[0201] The voxelization module 13 is further used for:

[0202] Through the main process, the fused point cloud data is voxel-divided, and the multiple voxels obtained by the division are written into the shared storage space; through the second calculation process, the multiple voxels are obtained from the shared storage space, and the voxel information corresponding to each voxel in the multiple voxels is determined, thereby determining the voxel information corresponding to the fused point cloud data.

[0203] In some embodiments, the fusion module 12 is also used to send a first processing progress to a first supervision process through the first computing process; to send a second processing progress to a second supervision process through the second computing process; to manage the first computing process according to the first processing progress through the first supervision process; and to manage the second computing process according to the second processing progress through the second supervision process.

[0204] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0205] An embodiment of the present invention further provides an electronic device, Figure 6 This is an optional structural diagram of an electronic device 3 provided in an embodiment of the present invention. Figure 6 As shown, the electronic device 3 includes: a memory 32 and a processor 33. The memory 32 and the processor 33 are connected via a communication bus 34; the memory 32 is used to store executable instructions; the processor 33 is used to execute the executable instructions stored in the memory 32 to implement the point cloud processing method provided by the embodiment of the present invention.

[0206] An embodiment of the present invention provides a vehicle, the vehicle comprising: Figure 5 The point cloud processing device shown, or, comprising Figure 6 Electronic equipment shown.

[0207] An embodiment of the present invention provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by the processor, the processor will be caused to execute the torque determination method provided by the embodiment of the present invention.

[0208] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0209] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0210] As examples, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As examples, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network.

[0211] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0212] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0213] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0214] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0215] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. A point cloud processing method, characterized in that: include: Determine multi-frame point cloud data of the target scene; The multi-frame point cloud data includes semantic categories of points and multiple dynamic target areas; Based on the semantic categories of the points, the static point cloud data in the multiple frames of point cloud data are spliced ​​and the point cloud data corresponding to the multiple dynamic target areas are aggregated to determine the fused point cloud data; The fused point cloud data is voxelized to determine voxel information corresponding to the fused point cloud data.

2. The method according to claim 1, characterized in that The step of splicing the static point cloud data in the multiple frames of point cloud data and aggregating the point cloud data corresponding to the multiple dynamic target areas based on the semantic categories of the points to determine the fused point cloud data includes: According to the semantic categories of the points, dynamic semantic points are removed from the multiple frames of point cloud data to determine multiple frames of static point cloud data; Splicing the multiple frames of static point cloud data to determine static spliced ​​point cloud data; as well as, Based on the semantic categories of the points, target aggregation is performed on the point cloud data corresponding to the multiple dynamic target areas to determine at least one dynamically aggregated point cloud data; The fused point cloud data is determined by combining the at least one dynamically aggregated point cloud data with the statically spliced ​​point cloud data.

3. The method according to claim 2, characterized in that The multiple frames of static point cloud data include: current frame static point cloud data and at least one frame of non-current frame static point cloud data; the step of splicing the multiple frames of static point cloud data to determine static spliced ​​point cloud data includes: Determine at least one pose transformation matrix of the at least one frame of non-current frame static point cloud data corresponding to the current frame static point cloud data; According to the at least one pose transformation matrix, the at least one frame of non-current frame static point cloud data is pose transformed, and the at least one non-current frame static point cloud data after pose transformation is spliced ​​with the current frame static point cloud data to determine the static spliced ​​point cloud data.

4. The method according to claim 2, characterized in that: The multiple dynamic target areas include multiple areas corresponding to at least one dynamic target object in the multiple frames of point cloud data; the at least one dynamic target object corresponds to at least one target identifier; The step of performing target aggregation on the point cloud data corresponding to the plurality of dynamic target areas based on the semantic category of the points to determine at least one dynamically aggregated point cloud data comprises: Determine, according to the at least one target identifier, a target identifier of a point in the point cloud data corresponding to each of the multiple dynamic target areas; Aggregating points with the same target identifier in the point cloud data corresponding to the multiple dynamic target areas to determine at least one initial dynamic aggregated point cloud data corresponding to the at least one dynamic target object; For each dynamically aggregated point cloud data in the at least one initial dynamically aggregated point cloud data, semantic category normalization is performed on the points in the each initial dynamically aggregated point cloud data according to the semantic category of the point to determine the at least one dynamically aggregated point cloud data.

5. The method according to claim 2 or 4, characterized in that: Before combining the at least one dynamically aggregated point cloud data with the static spliced ​​point cloud data to determine the fused point cloud data, the method further includes: Symmetry completion and / or enhancement processing is performed on the at least one dynamically aggregated point cloud data.

6. The method according to any one of claims 2 to 4, characterized in that: The combining the at least one dynamically aggregated point cloud data with the statically spliced ​​point cloud data to determine the fused point cloud data includes: Determine the position of at least one target area in the static spliced ​​point cloud data according to the position of at least one dynamic target area in the current frame point cloud data in the multiple frames of point cloud data; According to the at least one target area position, the at least one dynamically aggregated point cloud data is fused into the static spliced ​​point cloud data to determine the fused point cloud data.

7. The method according to any one of claims 1 to 4, characterized in that: The voxelization processing of the fused point cloud data to determine the voxel information corresponding to the fused point cloud data includes: Performing voxel division on the fused point cloud data to determine a plurality of voxels; synchronously calculating a unique voxel coordinate corresponding to each voxel in the plurality of voxels; Based on the voxel unique coordinates corresponding to each voxel, the voxel information corresponding to each voxel is determined, thereby determining the voxel information corresponding to the fused point cloud data.

8. The method according to claim 7, characterized in that The determining the voxel information corresponding to each voxel based on the unique voxel coordinates corresponding to each voxel includes: Synchronously processing each voxel in the plurality of voxels, calculating a mean value corresponding to each non-integer data according to at least one non-integer data corresponding to a point contained in each voxel, and determining at least one non-integer voxel information corresponding to each voxel; The voxel information corresponding to each voxel is determined according to the voxel unique coordinates corresponding to each voxel and at least one non-integer voxel information.

9. The method according to claim 7, characterized in that: The determining the voxel information corresponding to each voxel based on the unique voxel coordinates corresponding to each voxel includes: Synchronously processing each voxel in the plurality of voxels, and determining the integer data with the largest number of occurrences in each type of integer data according to at least one type of integer data corresponding to the points contained in each voxel, thereby determining at least one type of integer voxel information corresponding to each voxel; The voxel information corresponding to each voxel is determined according to the voxel unique coordinates corresponding to each voxel and at least one integer voxel information.

10. The method according to claim 8 or 9, characterized in that: The at least one non-integer data includes: at least one of speed and / or reflection intensity; the at least one integer data includes: at least one of semantic category and / or target identification.

11. The method according to any one of claims 1 to 4, or claim 8 or 9, characterized in that: The step of splicing the static point cloud data in the multiple frames of point cloud data and aggregating the point cloud data corresponding to the multiple dynamic target areas based on the semantic categories of the points to determine the fused point cloud data includes: Aggregating point cloud data corresponding to the plurality of dynamic target areas through a first computing process, and writing the obtained at least one dynamically aggregated point cloud data into a shared storage space; Through the main process, the static point cloud data in the multiple frames of point cloud data are spliced, and the at least one dynamically aggregated point cloud data is obtained from the shared storage space, and is merged with the spliced ​​static spliced ​​point cloud data to determine the merged point cloud data; The voxelization processing of the fused point cloud data to determine the voxel information corresponding to the fused point cloud data includes: By means of the main process, the fused point cloud data is divided into voxels, and a plurality of voxels obtained by the division are written into the shared storage space; Through the second calculation process, the multiple voxels are obtained from the shared storage space, and the voxel information corresponding to each voxel in the multiple voxels is determined, so as to determine the voxel information corresponding to the fused point cloud data.

12. The method according to claim 11, characterized in that The method further comprises: Sending a first processing progress to a first monitoring process through the first computing process; Sending a second processing progress to a second monitoring process through the second computing process; managing the first computing process according to the first processing schedule through the first monitoring process; The second computing process is managed according to the second processing schedule through the second supervisory process.

13. A point cloud processing device, characterized in that: include: A determination module, used to determine multi-frame point cloud data of a target scene; The multi-frame point cloud data includes semantic categories of points and multiple dynamic target areas; A fusion module, configured to splice the static point cloud data in the multiple frames of point cloud data and aggregate the point cloud data corresponding to the multiple dynamic target areas based on the semantic categories of the points, to determine fused point cloud data; The voxelization module is used to perform voxelization processing on the fused point cloud data to determine the voxel information corresponding to the fused point cloud data.

14. An electronic device, characterized in that: include: A memory for storing computer executable instructions; A processor, configured to implement the method according to any one of claims 1 to 12 when executing the computer executable instructions stored in the memory.

15. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images