A Point Cloud Data Processing Method and Device

Abstract

Embodiments of the present invention provide a method and apparatus for processing point cloud data, relating to the technical field of data processing. The method includes: acquiring multiple frames of point cloud data to be processed; determining, from each frame of the point cloud data to be processed, the target point cloud data corresponding to the same target object; and performing fusion processing on the target point cloud data to obtain a target recognition result. By applying the point cloud data processing solution provided by the embodiments of the present invention, the accuracy of the object information reflected by the laser points in the point cloud data can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a method and device for processing point cloud data. Background Art

[0002] A lidar can collect point cloud data of areas to be collected, such as roads and parks. After a data processing device obtains the above point cloud data, it can process the point cloud data to obtain information such as the positions and types of objects such as people and vehicles in the area to be collected. Furthermore, based on the information of the above objects, the traffic conditions of the area to be collected can be determined. For example, the above traffic conditions can be the pedestrian traffic conditions, vehicle traffic conditions, etc.

[0003] In the prior art, a lidar can emit laser beams to an area to be collected and collect the laser beams reflected from the surfaces of objects in the area to be collected. Based on the information reflected by the collected laser beams, laser points reflecting the information of the above objects are generated, and the laser points can reflect data such as the positions of the objects. The set of laser points generated by the lidar collecting laser beams in one collection period is called point cloud data, and the laser points generated from the laser beams reflected from the surface of the above object in the point cloud data can be called the laser points of the object.

[0004] In the actual application process, after the lidar is installed and fixed, it usually can only emit laser beams to the area to be collected at a fixed angle, which results in that the lidar can only emit laser beams to a partial area of the surface of the object, and the collected laser beams are only the laser beams reflected from a partial area of the surface of the object. As a result, the laser points of the object in the generated point cloud data are only the laser points of a partial area of the surface of the object, and the information reflecting the object is inaccurate. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a method and device for processing point cloud data to improve the accuracy of the object information reflected by the laser points in the point cloud data. The specific technical solutions are as follows:

[0006] In a first aspect, the embodiments of the present invention provide a method for processing point cloud data, the method including:

[0007] Obtaining multiple frames of point cloud data to be processed, where each frame of point cloud data to be processed is collected by lidars installed in different areas, and the time difference between the earliest collection time and the latest collection time in their collection times is less than a preset first duration threshold;

[0008] Determining the target point cloud data corresponding to the same target object from each frame of point cloud data to be processed;

[0009] Performing fusion processing on the target point cloud data to obtain a target recognition result.

[0010] In one embodiment of the present invention, the fusion processing of the target point cloud data includes:

[0011] Synchronize each laser point in each target point cloud data to the target fusion time;

[0012] Perform fusion processing on all laser points synchronized to the target fusion time.

[0013] In one embodiment of the present invention, the synchronization of each laser point in each target point cloud data to the target fusion time includes:

[0014] Intra-frame synchronization: For each frame where the target point cloud data is located, synchronize each laser point in the target point cloud data to the intra-frame equalization time to obtain predicted point cloud data; wherein, the intra-frame equalization time is determined according to the timestamps of each laser point in the target point cloud data within the frame;

[0015] Inter-frame synchronization: Synchronize each predicted point cloud data to the target fusion time according to the first time difference between the intra-frame equalization time of each frame and the target fusion time.

[0016] In one embodiment of the present invention, the synchronization of each laser point in the target point cloud data to the intra-frame equalization time includes:

[0017] Determine the motion information of the target object according to the target point cloud data, and the motion information includes the motion speed;

[0018] Based on the motion information, and the second time difference between the timestamp of each laser point and the intra-frame equalization time, synchronize each laser point to the intra-frame equalization time.

[0019] In one embodiment of the present invention, the synchronization of each laser point to the intra-frame equalization time based on the motion information and the second time difference between the timestamp of each laser point and the intra-frame equalization time includes:

[0020] Based on the motion information of the target object, the second time difference, and the correction matrix, synchronize each laser point to the intra-frame equalization time, wherein the correction matrix is used to compensate for the coordinate error of the laser point caused by the phase difference of the lidar within the second time difference.

[0021] In one embodiment of the present invention, the intra-frame equalization time is the median or average value of the timestamps of each laser point in the target point cloud data.

[0022] In one embodiment of the present invention, the determination of the target point cloud data corresponding to the same target object from the point cloud data to be processed in each frame includes:

[0023] Cluster the point cloud data to be processed for each frame to obtain a clustering result;

[0024] Based on the transformation relationship of each lidar relative to the reference coordinate system, determine the target point cloud data corresponding to the same target object according to the position of the target object in the reference coordinate system and the clustering result.

[0025] In a second aspect, an embodiment of the present invention further provides a point cloud data processing device, and the device includes:

[0026] A point cloud acquisition module, configured to acquire multiple frames of point cloud data to be processed, where each frame of point cloud data to be processed is collected by lidars installed in different regions, and the time difference between the earliest acquisition time and the latest acquisition time in their acquisition times is less than a preset first duration threshold;

[0027] A target determination module, configured to determine the target point cloud data corresponding to the same target object from each frame of point cloud data to be processed;

[0028] A point cloud fusion module, configured to perform fusion processing on the target point cloud data to obtain a target recognition result.

[0029] In an embodiment of the present invention, the point cloud fusion module includes:

[0030] A point cloud synchronization sub-module, configured to synchronize each laser point in each target point cloud data to the target fusion time;

[0031] A point cloud fusion sub-module, configured to perform fusion processing on all laser points synchronized to the target fusion time to obtain a target recognition result.

[0032] In an embodiment of the present invention, the point cloud synchronization sub-module includes:

[0033] An intra-frame synchronization unit, configured to, for each frame where the target point cloud data is located, synchronize each laser point in the target point cloud data to the intra-frame equalization time to obtain predicted point cloud data; where the intra-frame equalization time is determined according to the timestamps of each laser point in the target point cloud data in this frame;

[0034] An inter-frame synchronization unit, configured to synchronize each predicted point cloud data to the target fusion time according to the first time difference between each intra-frame equalization time and the target fusion time.

[0035] In an embodiment of the present invention, the intra-frame synchronization unit includes:

[0036] An information acquisition sub-unit, configured to determine the motion information of the target object, where the motion information includes the motion speed;

[0037] A point cloud synchronization subunit, configured to synchronize each laser point to the in-frame equalization moment based on the motion information and the second time difference between the timestamp of each laser point and the in-frame equalization moment.

[0038] In one embodiment of the present invention, the point cloud synchronization subunit is specifically configured to:

[0039] Based on the motion information of the target object, the second time difference, and a correction matrix, synchronize each laser point to the in-frame equalization moment, where the correction matrix is used to compensate for the coordinate error of the laser point caused by the phase difference of the lidar within the second time difference.

[0040] In one embodiment of the present invention, the in-frame equalization moment is the median or average value of the timestamps of each laser point in the target point cloud data.

[0041] In one embodiment of the present invention, the target determination module is specifically configured to:

[0042] Cluster each frame of point cloud data to be processed to obtain a clustering result;

[0043] Based on the conversion relationship of each lidar relative to the reference coordinate system, determine the target point cloud data corresponding to the same target object according to the position of the target object in the reference coordinate system and the clustering result.

[0044] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0045] The memory is used to store a computer program;

[0046] The processor is configured to implement the steps of the point cloud data processing method described in any one of the first aspects when executing the program stored in the memory.

[0047] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program implements the steps of the point cloud data processing method described in any one of the first aspects when executed by a processor.

[0048] Advantageous effects of the embodiments of the present invention:

[0049] It can be seen that in the point cloud data processing solution provided by the embodiments of the present invention, since lidars are installed in different installation areas, the collection angles of each lidar for collecting point cloud data are different. A point cloud data to be processed contains laser points of a target object collected by a lidar from one collection angle, and multiple point cloud data to be processed contain laser points of the target object collected by multiple lidars from multiple collection angles. By performing point cloud fusion processing on multiple target point cloud data corresponding to the same target object in each frame of point cloud data to be processed, the laser points of the target object scattered in multiple point cloud data to be processed can be concentrated into the fused point cloud data, so as to obtain point cloud data containing relatively complete laser points of the target object, and further improve the accuracy of the object information reflected by the laser points in the point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other embodiments according to these drawings.

[0051] Figure 1 It is a flowchart of the first point cloud data processing method provided by the embodiments of the present invention;

[0052] Figure 2 It is a flowchart of the second point cloud data processing method provided by the embodiments of the present invention;

[0053] Figure 3 It is a flowchart of the third point cloud data processing method provided by the embodiments of the present invention;

[0054] Figure 4 It is a flowchart of the fourth point cloud data processing method provided by the embodiments of the present invention;

[0055] Figure 5 It is a flowchart of the fifth point cloud data processing method provided by the embodiments of the present invention;

[0056] Figure 6 It is a flowchart of the sixth point cloud data processing method provided by the embodiments of the present invention;

[0057] Figure 7 It is a structural diagram of the first point cloud data processing device provided by the embodiments of the present invention;

[0058] Figure 8 It is a structural diagram of the second point cloud data processing device provided by the embodiments of the present invention;

[0059] Figure 9Schematic diagram of the structure of the third point cloud data processing device provided by an embodiment of the present invention;

[0060] Figure 10 Schematic diagram of the structure of the fourth point cloud data processing device provided by an embodiment of the present invention;

[0061] Figure 11 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0062] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art based on the present invention belong to the scope of protection of the present invention.

[0063] See Figure 1 , Figure 1 which is a flowchart of the first point cloud data processing method provided by an embodiment of the present invention. The above method includes the following steps S101 - S103.

[0064] Step S101: Obtain multiple frames of point cloud data to be processed.

[0065] Among them, each frame of point cloud data to be processed is collected by lidars installed in different regions, and the time difference between the earliest collection time and the latest collection time in its collection times is less than a preset first time duration threshold.

[0066] Among them, there are the following two situations for the collection times of the above point cloud data to be processed:

[0067] In the first situation, the above collection time can be: the times when each lidar collects the point cloud data to be processed recorded by the same clock device. The above clock device can be a GPS (Global Positioning System) clock device.

[0068] In the second situation, each lidar can be time - synchronized by the same clock device to ensure that the clocks of each lidar are synchronized. In this case, the above collection time can be: the times when the lidar itself records the collection of the point cloud data to be processed.

[0069] Specifically, since the lidar installed in an installation area can collect multiple frames of point cloud data within a period of time, among the point cloud data collected by the lidars in different installation areas, there is point cloud data with similar acquisition times. Therefore, multiple frames of point cloud data can be determined from the point cloud data collected by the lidars in different installation areas. The multiple frames of point cloud data are collected by different radars, and the time difference between their acquisition times is less than the above-mentioned first time threshold. The multiple frames of point cloud data are used as the point cloud data to be processed.

[0070] In an embodiment of the present invention, when selecting the point cloud data to be processed, the point cloud data collected by the lidar in any installation area can be first selected as the reference point cloud data, and then among the point cloud data collected by the lidars in other installation areas, the point cloud frames whose acquisition times are less than / greater than the acquisition time of the reference point cloud data and the time difference between their acquisition times and the acquisition time of the reference point cloud data is less than the above-mentioned first time threshold are selected.

[0071] In an embodiment of the present invention, the above-mentioned first time threshold can be a manually set threshold. For example, it can be 20ms, 50ms, or determined according to the sampling frequency of the lidar.

[0072] Step S102: Determine the target point cloud data corresponding to the same target object from each frame of the point cloud data to be processed.

[0073] Among them, the above-mentioned object can be a person, a vehicle, etc., and the target object is determined as at least one of them according to specific needs.

[0074] When there is an overlapping area in the field of view of multiple lidars and the target object appears in the overlapping area, the lidars at different positions simultaneously collect the point cloud data of the target object. Therefore, the point cloud data corresponding to the same target object may exist in different point cloud data to be processed.

[0075] Specifically, the laser points of each object can be first determined in each frame of the point cloud data to be processed, and then the laser points of each object are extracted from the frame of the point cloud data to be processed to form multiple point cloud data to be matched. Among them, each point cloud data to be matched contains the laser points of one object. Since the laser points corresponding to the same object may exist in different point cloud data to be processed, the point cloud data to be matched corresponding to the same target object can be determined from the point cloud data to be matched derived from different point cloud data to be processed as the target point cloud data.

[0076] In an embodiment of the present invention, there are the following two implementation manners for determining the laser points of the target object in the point cloud data to be processed:

[0077] In the first implementation manner, the point cloud data to be processed can be subjected to clustering processing, and the laser points belonging to the same cluster obtained by clustering are determined as the laser points of the target object.

[0078] In the second implementation method, the laser points of the target object can also be determined in the point cloud data to be processed through the existing point cloud segmentation technology, which will not be elaborated here.

[0079] In another embodiment of the present invention, there are the following two implementation methods for determining the target point cloud data from multiple point cloud data to be matched:

[0080] In the first implementation method, feature extraction can be performed on the point cloud data to be matched, and based on the extracted features, the point cloud data to be matched that matches the object features of the target object is determined, and the determined point cloud data to be matched is used as the target point cloud data corresponding to the target object.

[0081] Feature extraction of the point cloud data to be matched can be implemented based on a feature extraction algorithm, or can also be implemented based on a deep learning network for feature extraction. Feature extraction of the point cloud data to be matched and determining the point cloud data in the point cloud data to be matched that matches the object features of the target object as the target point cloud data based on the extracted features can be achieved through the prior art and will not be elaborated here.

[0082] In the second implementation method, the similarity between the laser points of the target object in different point cloud data to be matched can also be calculated, and the laser points with a similarity greater than a preset threshold are determined as the laser points of the same object, and then the target point cloud data containing the same target object is determined.

[0083] Step S103: Perform fusion processing on the target point cloud data to obtain a target recognition result.

[0084] Specifically, the above-mentioned fusion processing can be understood as combining the target point cloud data information of the same target object from at least two radars for fusion to obtain the fused point cloud data. Then, target recognition is performed on the fused point cloud data to obtain the target recognition result.

[0085] The fused point cloud data can be a set of all laser points in each point cloud data before fusion, that is, all the laser points included in the determined target point cloud data can be superimposed, and the superimposed point cloud data is the fused point cloud data.

[0086] For example, if there are the determined point cloud data a and point cloud data b, point cloud data a contains 100 laser points, and point cloud data b contains 120 laser points. Point cloud fusion processing is performed on point cloud data a and point cloud data b, and the obtained fused point cloud data is a point cloud data containing 100 + 120 = 220 laser points.

[0087] It can be seen that in the point cloud data processing solution provided by the embodiments of the present invention, since lidars are installed in different installation areas, the acquisition angles of the lidars for acquiring point cloud data are different. A frame of point cloud data to be processed contains the laser points of the target object acquired by a lidar from one acquisition angle, and multiple frames of point cloud data to be processed contain the laser points of the target object acquired by multiple lidars from multiple acquisition angles. By performing point cloud fusion processing on multiple target point cloud data corresponding to the same target object in each frame of point cloud data to be processed, the laser points of the target object scattered in multiple point cloud data to be processed can be concentrated into the fused point cloud data, so as to obtain point cloud data containing relatively complete laser points of the target object, and further improve the accuracy of the object information reflected by the laser points in the point cloud data.

[0088] The following describes the determination of the target point cloud data corresponding to the same target object in combination with the specific implementation manner of determining the laser points of the target object by using point cloud clustering mentioned in the above step S102.

[0089] In an embodiment of the present invention, referring to Figure 2 , a flowchart of a second point cloud data processing method is provided. Compared with the embodiment shown in the foregoing Figure 1 , in this embodiment, the above step S102 can be implemented through the following steps S102A - S102B.

[0090] Step S102A: Cluster each frame of point cloud data to be processed to obtain a clustering result.

[0091] Among them, there are various ways of point cloud clustering. For example, Euclidean clustering, density clustering, hyperplane clustering, etc. in the prior art.

[0092] Specifically, clustering the point cloud data to be processed can determine the laser points of the same target object as the same cluster, and the laser points of different target objects belong to different clusters. Extracting the laser points in the same cluster from the point cloud data to be processed can form a point cloud data to be matched, obtaining a clustering result; extracting the laser points in multiple clusters from the point cloud data to be processed respectively can form multiple point cloud data to be matched, obtaining multiple clustering results.

[0093] Clustering each frame of point cloud data to be processed can be implemented based on the above existing clustering technologies, which will not be elaborated here.

[0094] Step S102B: Based on the conversion relationship of each lidar relative to the reference coordinate system, determine the target point cloud data corresponding to the same target object according to the position of the target object in the reference coordinate system and the clustering result.

[0095] Among them, the above reference coordinate system can be a pre-set coordinate system by humans. The above reference coordinate system is used to calibrate each lidar, and unify the positions of the acquisition objects reflected by each laser point in the point cloud data collected by the lidar to the same coordinate system.

[0096] Specifically, after obtaining the above clustering results, the position of the target object corresponding to the laser points in the clustering results in the reference coordinate system can be determined according to the position information of each laser point in the clustering results and the transformation relationship between the lidar and the reference coordinate system. The positions of multiple target objects correspond to the clustering results in each point cloud data to be processed. If the positions of the target objects corresponding to the clustering results in different point cloud data to be processed are the same or the distance difference is less than a preset distance threshold, it can be considered that these two target objects are the same target object, and the point cloud data to be processed where the laser points of the same target object are located is the target point cloud data of the target object.

[0097] Among them, the above preset distance threshold can be 0.1 meter, 0.2 meter or other lengths. The setting of this distance threshold can take into account the displacement of the target object during the time period between the earliest acquisition time and the latest acquisition time.

[0098] It can be seen that in the point cloud data processing solution provided by the embodiments of the present invention, the clustering results obtained by clustering the point cloud data to be processed can be considered to contain the laser points of the same target object. Based on the transformation relationship between each lidar and the reference coordinate system, the position information of the laser points in the clustering results can be transformed to the reference coordinate system. In this way, the position information reflected by each clustering result in multiple point cloud data to be processed is the position information in the same reference coordinate system. In this way, the laser points with the same or similar positions can be determined as the laser points of the same target object in the reference coordinate system, and then the target point cloud data corresponding to the same target object can be accurately determined.

[0099] It is usually difficult for multiple lidars to synchronously collect point cloud data. Therefore, the point cloud data collected by different lidars are usually point cloud data collected at different times, and the above target object may be in a moving state, and the position of the target object may change at different times. Therefore, the positions of the target object reflected by the target point cloud data of the same target object may deviate.

[0100] In view of the above situation, in one embodiment of the present invention, see Figure 3 , a schematic flowchart of the third point cloud data processing method is provided. In this embodiment, the above step S103 can be implemented through the following steps S103A - S103B.

[0101] Step S103A: Synchronize each laser point in each target point cloud data to the target fusion time.

[0102] Among them, the above-mentioned target fusion moment can be a manually set moment, and the above-mentioned target fusion moment can also be a moment determined according to the acquisition moments of each target point cloud data.

[0103] For example, the above-mentioned target fusion moment can be the earliest or latest acquisition moment among the acquisition moments of each target point cloud data, or it can be other acquisition moments among the acquisition moments of each target point cloud data, or it can also be the average value of the acquisition moments of each target point cloud data, etc.

[0104] According to the above description, it can be known that the acquisition moments of different target point cloud data may be different, and the acquisition moments of different laser points in the same frame of target point cloud data may also be different. For example, for a mechanically scanned lidar, the point cloud data collected by the lidar includes the laser points collected by the lidar within a scanning cycle. By synchronizing each laser point in each target point cloud data to the target fusion moment, the synchronized target point cloud data can be regarded as the point cloud data collected at the same acquisition moment, and the acquisition moments of the laser points included in each point cloud data after synchronization can also be regarded as this same acquisition moment.

[0105] Specifically, the motion information and acquisition moment of each laser point in the target point cloud data can be obtained first, and the above-mentioned target fusion moment can be determined. Then, based on the obtained motion information, acquisition moment, and target fusion moment, each laser point is adjusted to achieve synchronizing the laser point to the target fusion moment. The synchronized target point cloud data can be regarded as the point cloud data collected by the lidar at the target fusion moment.

[0106] In an embodiment of the present invention, there are the following two implementation manners for synchronizing each laser point in the target point cloud data to the target fusion moment.

[0107] In the first implementation manner, each laser point in each target point cloud data from different radars can be synchronized to the same moment first. At this time, each laser point in each target point cloud data can be regarded as the laser points collected at the same acquisition moment. After synchronizing each target point cloud data, each laser point in each target point cloud data is then synchronized to the target fusion moment. For details, please refer to steps S103A1 - S103A2 in the subsequent Figure 4 illustrated embodiment, which will not be elaborated here for the time being.

[0108] In the second implementation manner, since the above-mentioned target fusion moment can be preset, therefore, for each laser point in each target point cloud data, this laser point can be synchronized to the target fusion moment according to the acquisition moment of this laser point and the target fusion moment.

[0109] In addition, for the implementation manner of adjusting each laser point based on the obtained motion information, acquisition moment, and target fusion moment, please refer to the subsequentFigure 5 In the illustrated embodiment, steps S103A1a - S013A1b are not elaborated here for the time being.

[0110] Step S103B: Perform fusion processing on all laser points synchronized to the target fusion moment to obtain a target recognition result.

[0111] Specifically, all synchronized laser points can be superimposed, and the point cloud data obtained after superimposition is the target recognition result.

[0112] As can be seen from the above, in the point cloud data processing solution provided by the embodiment of the present invention, synchronizing each laser point in each target point cloud data to the target fusion moment can make the positions of the laser points of the target object reflected in each synchronized target point cloud data be at the same moment. Therefore, after performing fusion processing on all laser points synchronized to the target fusion moment, the accuracy of the obtained target recognition result can be improved.

[0113] Next, a further description is made for the first implementation manner mentioned in the above step S103A.

[0114] In an embodiment of the present invention, referring to Figure 4 , a flowchart of a fourth point cloud data processing method is provided. In this embodiment, the above step S103A can be implemented through the following steps S103A1 - S103A2.

[0115] Step S103A1: (Intra - frame synchronization) For each frame where the target point cloud data is located, synchronize each laser point in the target point cloud data to the intra - frame equilibrium moment to obtain predicted point cloud data.

[0116] Among them, the intra - frame equilibrium moment is determined according to the timestamps of each laser point in the target point cloud data in this frame.

[0117] In an embodiment of the present invention, the above intra - frame equilibrium moment is the median of the timestamps of each laser point in the target point cloud data.

[0118] In this solution, since it is necessary to synchronize each laser point in the target point cloud data to the intra - frame equilibrium moment, setting the intra - frame equilibrium moment as the median of the timestamps of each laser point can minimize the number of laser points that need to be synchronized, thereby improving the efficiency of data processing.

[0119] In an embodiment of the present invention, the above intra - frame equilibrium moment is the average value of the timestamps of each laser point in the target point cloud data.

[0120] In this solution, setting the intra - frame equilibrium moment as the average value of the timestamps of each laser point can accurately synchronize each laser point in the target point cloud data to the same moment, improving the accuracy of the predicted point cloud data.

[0121] Synchronizing each laser point in the target point cloud data to the in-frame equalization moment can be referred to the steps S103A1a - S013A1b in the subsequent Figure 5 illustrated embodiments, which will not be elaborated here for the time being.

[0122] Step S103A2: (Inter-frame synchronization) Synchronize each predicted point cloud data to the target fusion moment according to the first time difference between the in-frame equalization moment and the target fusion moment of each frame.

[0123] Specifically, the in-frame equalization moments of different predicted point cloud data may be the same or different. Therefore, for each predicted point cloud data, each laser point in the predicted point cloud data can be synchronized to the target fusion moment according to the time difference between the in-frame equalization moment of the predicted point cloud data and the target fusion moment.

[0124] The method of synchronizing each laser point in the predicted point cloud data to the target fusion moment is similar to the method of synchronizing each laser point in the target point cloud data to the in-frame equalization moment in the above steps S103A1a - S103A1b, which will not be elaborated here.

[0125] As can be seen from the above, in the point cloud data processing solution provided by the embodiments of the present invention, first, each laser point in each frame of target point cloud data is synchronized to the in-frame equalization moment. After such synchronization, the timestamps of each laser point in each target point cloud data are the same in-frame equalization moment. Then, according to the first time difference between the in-frame equalization moment and the target fusion moment of each frame, each predicted point cloud data can be accurately synchronized to the same target fusion moment. In this way, when performing fusion processing on the point cloud data at the same target fusion moment, a more accurate target recognition result can be obtained.

[0126] Next, the implementation manner of adjusting each laser point mentioned in the above step S103A will be described.

[0127] In an embodiment of the present invention, referring to Figure 5 , a flowchart of a fifth point cloud data processing method is provided. In this embodiment, the above step S103A can be implemented through the following steps S103A1a - S013A1b.

[0128] Step S103A1a: Determine the motion information of the target object.

[0129] Among them, the above motion information includes the motion speed. Based on multiple frames of point cloud data collected by the same lidar and adjacent to the frame where the target point cloud data is located, target tracking can be performed on the target object to determine the motion information of the target object corresponding to the frame timestamp.

[0130] Step S103A1b: Synchronize each laser point to the in-frame equalization moment based on the motion information and the second time difference between the timestamp of each laser point and the in-frame equalization moment.

[0131] Specifically, assign the motion information of the target object to each laser point corresponding to the target object, that is, the motion information of the target object is equal to the motion information of each laser point corresponding to the target object. Then, based on the motion information of the laser point and the second time difference between the timestamp of the laser point and the in-frame equalization moment, synchronize the laser point to the in-frame equalization moment.

[0132] The motion information of the laser point includes the motion speed of the laser point. Multiplying the motion speed by the above second time difference can obtain the position offset of the laser point within the second time difference. Based on this position offset, the laser point can be corrected, thereby realizing synchronizing the laser point to the in-frame equalization moment.

[0133] That is, assume the frame timestamp is t1, the motion speed of the target object corresponding to the frame timestamp t1 is v1, and the timestamps of all laser points corresponding to the target object within the frame are t2, t3.....t100 respectively. Then the determined in-frame equalization moment (taking the average value) is tx = (t2 + t3 +.... + t100) / 99. For a certain laser point of the target object (assuming the timestamp is t2), the displacement compensation amount for synchronizing the laser point to the in-frame equalization moment is v1*(t2 - tx). And so on, until all laser points corresponding to the target object within the frame have completed synchronization to the in-frame equalization moment.

[0134] As can be seen from the above, in the point cloud data processing solution provided by the embodiments of the present invention, based on the determined motion information of the target object and the second time difference between the timestamp of each laser point and the in-frame equalization moment, the position offset of the target object corresponding to each laser point within the second time difference can be accurately calculated. In this way, based on the above motion information and the second time difference, each laser point can be corrected, so that each corrected laser point can be regarded as a laser point at the in-frame equalization moment, thereby improving the accuracy of the predicted point cloud data and further improving the accuracy of point cloud data processing.

[0135] For a mechanical lidar, the error of the point cloud coordinates not only comes from the displacement of the target object itself within the second time difference, but also from the error caused by the phase difference of the radar rotating component within the second time difference.

[0136] To eliminate this deviation, in one embodiment of the present invention, refer to Figure 6 , a flowchart of the sixth point cloud data processing method is provided. In this embodiment, the above step S103A1b can be implemented through the following step S103A1b1.

[0137] Step S103A1b1: Synchronize each laser point to the in-frame equalization moment based on the motion information of the target object, the second time difference, and the correction matrix.

[0138] Among them, the above correction matrix is used to compensate for the coordinate error of the laser points caused by the phase difference of the lidar within the second time difference. That is, assuming that the timestamps of the laser points of the target object within a single frame are t1 to t99, and the in-frame equalization moment is t50, then for any laser point (assuming the timestamp is t30), the displacement of the point cloud that needs to be compensated for converting this laser point to the t50 moment can be determined through the correction matrix, that is, the coordinate error of the point cloud caused by the rotation of the laser emission module is completely eliminated. After passing through the correction matrix, all the laser points of the target object can be regarded as being collected at the same moment. The above correction matrix can be pre-calibrated.

[0139] Specifically, for each target point cloud data, the phase difference of the rotating part of the radar within the second time difference can be measured by the inertial measurement mechanism, and the first compensation amount of each laser point can be determined by querying the correction matrix; then, based on the motion information of the target object and the second time difference, the second compensation amount of each laser point can be determined. Combining the first compensation amount and the second compensation amount, each laser point in each target point cloud data is synchronized to the in-frame equalization moment.

[0140] As can be seen from the above, the point cloud data processing solution provided by the embodiments of the present invention can simultaneously eliminate the coordinate error of the laser points caused by the phase difference of the rotating part of the lidar within the second time difference and the motion of the target object within the second time difference, thereby improving the accuracy of the predicted point cloud data.

[0141] When the lidar collects the point cloud data of the target object, it also collects the point cloud data of the surrounding environment of the target object. In this case, there are more laser points in the point cloud data collected by the lidar, and the computational amount for processing this point cloud data is relatively large.

[0142] In view of this, in one embodiment of the present invention, before determining the target point cloud data, the environmental information denoising process can be performed on the point cloud data to be processed.

[0143] The above environmental information denoising process can be understood as removing the laser points belonging to the environmental object from the point cloud data.

[0144] The above environmental object can be other objects in the area to be collected by the lidar except the target object. For example, the above environmental object can be a tree, a building, or a traffic sign, etc. The denoising process can reduce the number of laser points in the point cloud data to be processed. In this way, during the subsequent process of processing the point cloud data to be processed, the amount of data to be processed is less, thereby improving the efficiency of point cloud data processing.

[0145] Corresponding to the foregoing point cloud data processing method, an embodiment of the present invention further provides a point cloud data processing device.

[0146] Referring to Figure 7 , a structural schematic diagram of a first point cloud data processing device is provided. The device includes:

[0147] A point cloud acquisition module 701, configured to acquire multiple frames of point cloud data to be processed. Among them, each frame of point cloud data to be processed is collected by lidars installed in different areas, and the time difference between the earliest acquisition time and the latest acquisition time in the acquisition times is less than a preset first duration threshold;

[0148] A target determination module 702, configured to determine target point cloud data corresponding to the same target object from each frame of point cloud data to be processed;

[0149] A point cloud fusion module 703, configured to perform fusion processing on the target point cloud data to obtain a target recognition result.

[0150] It can be seen that in the point cloud data processing solution provided by the embodiment of the present invention, since the lidars are installed in different installation areas, the acquisition angles of the lidars for collecting point cloud data are different. A frame of point cloud data to be processed contains the laser points of the target object collected by a lidar from one acquisition angle, and multiple frames of point cloud data to be processed contain the laser points of the target object collected by multiple lidars from multiple acquisition angles. By performing point cloud fusion processing on the multiple target point cloud data corresponding to the same target object in each frame of point cloud data to be processed, the laser points of the target object scattered in multiple point cloud data to be processed can be concentrated in the fused point cloud data, so as to obtain point cloud data containing relatively complete laser points of the target object, and further improve the accuracy of the object information reflected by the laser points in the point cloud data.

[0151] In an embodiment of the present invention, referring to Figure 8 , a structural schematic diagram of a second point cloud data processing device is provided. In this embodiment, the point cloud fusion module 703 includes:

[0152] A point cloud synchronization sub-module 703A, configured to synchronize each laser point in each target point cloud data to the target fusion time;

[0153] A point cloud fusion sub-module 703B, configured to perform fusion processing on all the laser points synchronized to the target fusion time to obtain a target recognition result.

[0154] As can be seen from the above, in the point cloud data processing solution provided by the embodiments of the present invention, synchronizing each laser point in each target point cloud data to the target fusion moment can make the positions of the target objects reflected by the laser points in each synchronized target point cloud data be the positions at the same moment. Therefore, after fusing all the laser points synchronized to the target fusion moment, the accuracy of the obtained target recognition result can be improved.

[0155] In one embodiment of the present invention, referring to Figure 9 , a schematic structural diagram of a third point cloud data processing device is provided. In this embodiment, the point cloud synchronization sub-module 703A includes:

[0156] An intra-frame synchronization unit 703A1, configured to synchronize each laser point in the target point cloud data to the intra-frame equilibrium moment for each frame where the target point cloud data is located, so as to obtain predicted point cloud data; wherein, the intra-frame equilibrium moment is determined according to the timestamps of each laser point in the target point cloud data within the frame;

[0157] An inter-frame synchronization unit 703A2, configured to synchronize each predicted point cloud data to the target fusion moment according to a first time difference between each intra-frame equilibrium moment and the target fusion moment.

[0158] As can be seen from the above, in the point cloud data processing solution provided by the embodiments of the present invention, first, each laser point in each frame of target point cloud data is synchronized to the intra-frame equilibrium moment. After such synchronization, the timestamps of each laser point in each target point cloud data are the same intra-frame equilibrium moment. Then, according to the first time difference between each intra-frame equilibrium moment and the target fusion moment, each predicted point cloud data can be accurately synchronized to the same target fusion moment. In this way, when fusing the point cloud data at the same target fusion moment, a relatively accurate target recognition result can be obtained.

[0159] In one embodiment of the present invention, referring to Figure 10 , a schematic structural diagram of a fourth point cloud data processing device is provided. In this embodiment, the intra-frame synchronization unit 703A1 includes:

[0160] An information acquisition sub-unit 703A1a, configured to determine the motion information of the target object, where the motion information includes the motion speed;

[0161] A point cloud synchronization sub-unit 703A1b, configured to synchronize each laser point to the intra-frame equilibrium moment based on the motion information and a second time difference between the timestamp of each laser point and the intra-frame equilibrium moment.

[0162] As can be seen from the above, in the point cloud data processing solution provided by the embodiments of the present invention, based on the determined motion information of the target object and the second time difference between the timestamp of each laser point and the in-frame equilibrium moment, the position offset of the target object corresponding to each laser point within the second time difference can be accurately calculated. In this way, based on the above motion information and the second time difference, each laser point can be corrected so that each corrected laser point can be regarded as a laser point at the in-frame equilibrium moment, thereby improving the accuracy of the predicted point cloud data and further improving the accuracy of point cloud data processing.

[0163] In one embodiment of the present invention, the point cloud synchronization subunit 703A1b is specifically configured to:

[0164] Synchronize each laser point to the in-frame equilibrium moment based on the motion information of the target object, the second time difference, and the correction matrix, where the correction matrix is used to compensate for the coordinate error of the laser point caused by the phase difference of the lidar within the second time difference.

[0165] As can be seen from the above, in the point cloud data processing solution provided by the embodiments of the present invention, the correction matrix is used to compensate for the coordinate error of the laser point caused by the phase difference of the lidar within the second time difference. Based on the motion information of the target object, the second time difference, and the correction matrix, the coordinate error of the laser point caused by the phase difference of the lidar within the second time difference can be eliminated, thereby improving the accuracy of the predicted point cloud data and further improving the accuracy of point cloud data processing.

[0166] In one embodiment of the present invention, the in-frame equilibrium moment is the median of the timestamps of each laser point in the target point cloud data.

[0167] In this solution, since each laser point in the target point cloud data needs to be synchronized to the in-frame equilibrium moment, setting the in-frame equilibrium moment as the median of the timestamps of each laser point can minimize the number of laser points that need to be synchronized, thereby improving the efficiency of data processing.

[0168] In one embodiment of the present invention, the in-frame equilibrium moment is the average value of the timestamps of each laser point in the target point cloud data.

[0169] In this solution, setting the in-frame equilibrium moment as the average value of the timestamps of each laser point can accurately synchronize each laser point in the target point cloud data to the same moment, improving the accuracy of the predicted point cloud data.

[0170] In one embodiment of the present invention, the target determination module 703 is specifically configured to:

[0171] Cluster each frame of point cloud data to be processed to obtain a clustering result;

[0172] Based on the conversion relationships of each lidar relative to the reference coordinate system, and according to the position and clustering result of the target object in the reference coordinate system, determine the target point cloud data corresponding to the same target object.

[0173] It can be seen that in the point cloud data processing solution provided by the embodiments of the present invention, the clustering result obtained by clustering the point cloud data to be processed can be considered to contain the laser points of the same target object. Based on the conversion relationships of each lidar relative to the reference coordinate system, the position information of the laser points in the clustering result can be converted to the reference coordinate system. In this way, the position information reflected by each clustering result in multiple point cloud data to be processed is the position information in the same reference coordinate system. In this way, the laser points with the same or similar positions in the reference coordinate system can be determined as the laser points of the same target object, and then the target point cloud data corresponding to the same target object can be accurately determined.

[0174] The embodiments of the present invention also provide an electronic device, as Figure 11 shown, including a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104. Among them, the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.

[0175] The memory 1103 is used to store a computer program.

[0176] When the processor 1101 is used to execute the program stored in the memory 1103, the following steps are implemented:

[0177] Obtain multiple frames of point cloud data to be processed, where each frame of point cloud data to be processed is collected by lidars installed in different regions, and the time difference between the earliest collection time and the latest collection time in their collection times is less than a preset first duration threshold.

[0178] Determine the target point cloud data corresponding to the same target object from each frame of point cloud data to be processed.

[0179] Perform fusion processing on the target point cloud data to obtain a target recognition result.

[0180] Other solutions for the processor 1101 to execute the program stored in the memory 1103 to implement point cloud data processing are the same as those mentioned in the foregoing method embodiments and will not be elaborated here.

[0181] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0182] The communication interface is used for communication between the above electronic device and other devices.

[0183] The memory may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0184] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0185] In another embodiment provided by the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above point cloud data processing methods are implemented.

[0186] In another embodiment provided by the present invention, there is also provided a computer program product containing instructions, which when running on a computer, causes the computer to execute any of the point cloud data processing methods in the above embodiments.

[0187] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0188] It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0189] Each embodiment in this specification is described in a related manner. The same or similar parts between the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the embodiments of the device, electronic device, computer-readable storage medium, and computer program product, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0190] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A method for processing point cloud data, characterized in that, the method includes: obtaining multiple frames of point cloud data to be processed, wherein each frame of point cloud data to be processed is collected by lidars installed in different regions, and the time difference between the earliest collection time and the latest collection time among their collection times is less than a preset first duration threshold; determining, from each frame of point cloud data to be processed, the target point cloud data corresponding to the same target object; performing fusion processing on the target point cloud data to obtain a target recognition result; the performing fusion processing on the target point cloud data includes: synchronizing each laser point in each target point cloud data to a target fusion time; performing fusion processing on all the laser points synchronized to the target fusion time; the synchronizing each laser point in each target point cloud data to a target fusion time includes: intra-frame synchronization: for each frame where the target point cloud data is located, synchronizing each laser point in the target point cloud data to an intra-frame equilibrium time to obtain predicted point cloud data; wherein the intra-frame equilibrium time is determined according to the timestamps of each laser point in the target point cloud data in this frame; inter-frame synchronization: synchronizing each predicted point cloud data to the target fusion time according to the first time difference between the intra-frame equilibrium time of each frame and the target fusion time; the synchronizing each laser point in the target point cloud data to an intra-frame equilibrium time includes: determining the motion information of the target object, where the motion information includes a motion speed; based on the motion information of the target object, a second time difference, and a correction matrix, synchronizing each laser point to the intra-frame equilibrium time, where the correction matrix is used to compensate for the coordinate error of the laser point caused by the phase difference of the lidar within the second time difference; the second time difference is the time difference between the timestamp of each laser point and the intra-frame equilibrium time; the process of synchronizing each laser point to the intra-frame equilibrium time based on the motion information of the target object, a second time difference, and a correction matrix includes: determining the first compensation amount of each laser point by querying the correction matrix; determining the second compensation amount of each laser point through the motion information of the target object and the second time difference; combining the first compensation amount and the second compensation amount to synchronize each laser point to the intra-frame equilibrium time.

2. The method according to claim 1, characterized in that, the intra-frame equilibrium time is the median or average value of the timestamps of each laser point in the target point cloud data.

3. The method according to claim 1 or 2, characterized in that, the determining, from each frame of point cloud data to be processed, the target point cloud data corresponding to the same target object includes: clustering each frame of point cloud data to be processed to obtain a clustering result; based on the conversion relationship of each lidar relative to a reference coordinate system, and according to the position of the target object in the reference coordinate system and the clustering result, determining the target point cloud data corresponding to the same target object.

4. A point cloud data processing device, characterized in that, the device includes: A point cloud acquisition module, configured to acquire multiple frames of point cloud data to be processed, where each frame of point cloud data to be processed is collected by lidar installed in different areas, and the time difference between the earliest collection time and the latest collection time among the collection times is less than a preset first duration threshold; A target determination module, configured to determine target point cloud data corresponding to the same target object from each frame of point cloud data to be processed; A point cloud fusion module, configured to perform fusion processing on the target point cloud data to obtain a target recognition result; The point cloud fusion module includes: A point cloud synchronization sub-module, configured to synchronize each laser point in each target point cloud data to a target fusion time; A point cloud fusion sub-module, configured to perform fusion processing on all laser points synchronized to the target fusion time to obtain a target recognition result; The point cloud synchronization sub-module includes: An intra-frame synchronization unit, configured to, for each frame where the target point cloud data is located, synchronize each laser point in the target point cloud data to an intra-frame equalization time to obtain predicted point cloud data; where the intra-frame equalization time is determined according to the timestamps of each laser point in the target point cloud data in the frame; An inter-frame synchronization unit, configured to synchronize each predicted point cloud data to the target fusion time according to a first time difference between each intra-frame equalization time and the target fusion time; The intra-frame synchronization unit includes: an information acquisition sub-unit, configured to determine the motion information of the target object, where the motion information includes a motion speed; A point cloud synchronization sub-unit, specifically configured to synchronize each laser point to the intra-frame equalization time based on the motion information of the target object, a second time difference, and a correction matrix, where the correction matrix is used to compensate for the coordinate error of the laser point caused by the phase difference of the lidar within the second time difference; the second time difference is the time difference between the timestamp of each laser point and the intra-frame equalization time; The process of synchronizing each laser point to the intra-frame equalization time based on the motion information of the target object, the second time difference, and the correction matrix includes: determining a first compensation amount for each laser point by querying the correction matrix; determining a second compensation amount for each laser point through the motion information of the target object and the second time difference; combining the first compensation amount and the second compensation amount to synchronize each laser point to the intra-frame equalization time.

5. An electronic device, characterized in that, it includes a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used to store a computer program; The processor, when executing the program stored on the memory, implements the steps of the point cloud data processing method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the point cloud data processing method according to any one of claims 1-3.

Images