Point cloud data enhancement method, device, computer equipment, system and storage medium

Abstract

The application is suitable for the technical field of data processing, and provides a point cloud data enhancement method, device, computer equipment, system and storage medium. The point cloud data enhancement method comprises: acquiring point cloud data of a laser radar and millimeter wave signals of a millimeter wave radar; acquiring three-dimensional cubic data according to the millimeter wave signals; the three-dimensional cubic data contains information of distance, speed and angle dimensions; acquiring a heatmap distribution map according to the three-dimensional cubic data; performing target extraction on the heatmap distribution map to determine targets in the heatmap distribution map; based on the targets, determining corresponding target point cloud sets in the point cloud data that are spatially synchronized with the heatmap distribution map; and performing data enhancement on the target point cloud sets. The application can solve the problem that the point cloud data of the laser radar is relatively sparse and affects the detection accuracy of targets.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a point cloud data enhancement method, apparatus, computer equipment, system and storage medium. Background Technology

[0002] LiDAR (Light Detection and Ranging) is a radar system that uses laser beams to detect the position, velocity, and other characteristics of targets. Currently, LiDAR primarily relies on point cloud sensing to detect targets such as vehicles and pedestrians on roads, and to determine their position and velocity within the radar's own coordinate system. However, the point cloud data generated by LiDAR is relatively sparse, which affects the accuracy of target detection. Summary of the Invention

[0003] This application provides a point cloud data enhancement method, apparatus, computer equipment, system, and storage medium to address the problem that the sparse point cloud data of lidar affects the accuracy of target detection.

[0004] In a first aspect, embodiments of this application provide a point cloud data augmentation method, the point cloud data augmentation method comprising:

[0005] The point cloud data of the lidar and the millimeter-wave signal of the millimeter-wave radar are acquired, wherein the point cloud data and the millimeter-wave signal are obtained by sampling the same area by the lidar and the millimeter-wave radar, respectively.

[0006] Three-dimensional cubic data is obtained based on the millimeter-wave signal; the three-dimensional cubic data includes information on distance, velocity, and angular dimensions;

[0007] A heatmap distribution map is obtained based on the three-dimensional cubic data; the heatmap distribution map is an information distribution map of distance and angle dimensions, and the information map contains the energy of the points corresponding to the distance and angle;

[0008] Target extraction is performed on the heatmap distribution map to determine the targets in the heatmap distribution map;

[0009] Based on the target, determine the target point cloud set corresponding to the point cloud data that is spatially synchronized with the heatmap distribution map;

[0010] Data augmentation is performed on the target point cloud set.

[0011] Secondly, embodiments of this application provide a point cloud data enhancement device, the point cloud data enhancement device comprising:

[0012] The signal acquisition module is used to acquire point cloud data from the lidar and millimeter-wave signals from the millimeter-wave radar. The point cloud data and the millimeter-wave signals are obtained by sampling the same area by the lidar and the millimeter-wave radar, respectively.

[0013] The data acquisition module is used to acquire three-dimensional data based on the millimeter-wave signal; the three-dimensional data includes information on distance, velocity, and angle dimensions.

[0014] The distribution map acquisition module is used to acquire a heatmap distribution map based on the three-dimensional data; the heatmap distribution map is an information distribution map of distance and angle dimensions, and the information map contains the energy of the points corresponding to the distance and angle;

[0015] The target extraction module is used to extract targets from the heatmap distribution map and determine the targets in the heatmap distribution map;

[0016] The set determination module is used to determine, based on the target, the target point cloud set corresponding to the point cloud data that is spatially synchronized with the heatmap distribution map;

[0017] The data augmentation module is used to augment the target point cloud set.

[0018] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0019] Fourthly, embodiments of this application provide a multi-sensor system, which includes a lidar, a millimeter-wave radar, and a computer device as described in the third aspect above;

[0020] The lidar and the millimeter-wave radar collect data from the same area;

[0021] The lidar is used to output the collected point cloud data to the computer device;

[0022] The millimeter-wave radar is used to output the collected millimeter-wave signals to the computer device.

[0023] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0024] In a sixth aspect, embodiments of this application provide a computer program product that, when run on a computer device, causes the computer device to perform the steps of the method described in the first aspect above.

[0025] As can be seen from the above, this application obtains point cloud data and millimeter-wave signals from data sampling of the same area by lidar and millimeter-wave radar. It can first obtain three-dimensional data based on the millimeter-wave signals, then obtain a heatmap distribution map based on the three-dimensional data, and extract targets from the heatmap distribution map to determine the targets in the heatmap distribution map. Based on the targets in the heatmap distribution map, the corresponding target point cloud set in the point cloud data that is spatially synchronized with the heatmap distribution map can be determined. By performing data augmentation on the target point cloud set, the density of lidar point cloud data (especially the point cloud data at the far end of lidar) can be increased, thereby improving the lidar's accuracy in detecting targets. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the architecture of a multi-sensor system provided in an embodiment of this application;

[0028] Figure 2 This is a schematic diagram illustrating the implementation process of a point cloud data augmentation method provided in an embodiment of this application;

[0029] Figure 3 This is an example diagram of the millimeter-wave signal processing flow;

[0030] Figure 4 This is an example image of a vehicle driving on a road;

[0031] Figure 5 This is an example image of a heatmap distribution.

[0032] Figure 6 This is a schematic diagram illustrating the implementation process of a point cloud data augmentation method provided in another embodiment of this application;

[0033] Figure 7 This is a schematic diagram of the structure of a point cloud data enhancement device provided in an embodiment of this application;

[0034] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0035] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0036] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0037] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0038] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0039] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0040] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0041] It should be understood that the sequence number of each step in this embodiment does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0042] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0043] See Figure 1 This is a schematic diagram of the architecture of a multi-sensor system provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0044] The aforementioned multi-sensor system includes a lidar 11, a millimeter-wave radar 12, and a computer device 13.

[0045] This application does not limit the type of lidar 11. For example, lidar 11 can be an 8-line, 16-line, 24-line, 32-line, 64-line, or 128-line lidar, etc.

[0046] This application does not limit the type of millimeter-wave radar 12. For example, millimeter-wave radar 12 can be a 77G or 24G baseband radar, etc.

[0047] In multi-sensor system applications, to ensure that the sensing ranges of the LiDAR 11 and the millimeter-wave radar 12 at least partially overlap, the installation of the LiDAR 11 and the millimeter-wave radar 12 needs to consider the matching of their fields of view. For example, the millimeter-wave radar 12 can use a planar antenna array, covering a forward 180° space or less. The LiDAR 11 has a limited field of view and a 360° panoramic field of view. During forward scanning, the LiDAR 11 with a limited field of view can be paired with a single millimeter-wave radar 12 using a planar array to complete the scanning and imaging of the forward field of view. The LiDAR 11 with a 360° panoramic field of view can be paired with 3 to 4 millimeter-wave radars 12 using planar arrays, and the fields of view of the multiple millimeter-wave radars 12 can partially overlap. Of course, the LiDAR 11 with a 360° panoramic field of view can also be paired with a single millimeter-wave radar 12 with the same 360° panoramic field of view; this is not a limitation.

[0048] LiDAR 11 and millimeter-wave radar 12 sample data from the same area within their sensing range, obtaining point cloud data from LiDAR 11 and millimeter-wave signals from millimeter-wave radar 12, respectively. LiDAR 11 can send the point cloud data to computer device 13, and millimeter-wave radar 12 can send the millimeter-wave signals to computer device 13. After receiving the point cloud data and millimeter-wave signals, computer device 13 can first acquire three-dimensional data based on the millimeter-wave signals, then acquire a heatmap distribution map based on the three-dimensional data, and extract targets from the heatmap distribution map to determine the targets in the heatmap distribution map. Based on the targets in the heatmap distribution map, the corresponding target point cloud set in the point cloud data spatially synchronized with the heatmap distribution map can be determined. By performing data augmentation on the target point cloud set, the density of the point cloud data of LiDAR 11 (especially the point cloud data at the far end of LiDAR) can be increased, improving the target detection accuracy of LiDAR 11.

[0049] Millimeter-wave radar 12 uses electromagnetic waves for environmental perception, has a long detection range, and is suitable for moving targets. LiDAR 11 primarily relies on point cloud perception, providing better stereoscopic perception and easier distinction between target types. However, the point cloud data of LiDAR 11 is sparse at long distances, affecting the accuracy of target detection. Since the detection range of millimeter-wave radar 12 is greater than that of LiDAR 11, utilizing the detection capability of millimeter-wave radar 12 to assist LiDAR 11 at long distances can increase the density of point cloud data at the far end of LiDAR 11, thereby improving the accuracy of target detection.

[0050] It should be noted that this application does not limit the application scenarios of the aforementioned multi-sensor system. For example, the aforementioned application scenario can be an autonomous driving scenario for vehicles. The aforementioned LiDAR 11 is an onboard LiDAR 11, the aforementioned millimeter-wave radar 12 is an onboard millimeter-wave radar, and the aforementioned computer device 13 can be an onboard terminal. Through this application, vehicles can accurately detect targets on the road during autonomous driving, thereby improving the safety of autonomous driving. The aforementioned application scenario can also be a vehicle assisted driving scenario. After detecting a target through this application, the size, position, speed, heading angle, and other information of the detected target can be displayed on a high-precision map to better assist the driver in driving the vehicle safely.

[0051] See Figure 2 This is a schematic diagram illustrating the implementation flow of a point cloud data augmentation method provided in an embodiment of this application. This point cloud data augmentation method is applied to... Figure 1 The computer equipment in the multi-sensor system shown. Figure 2 As shown, the point cloud data augmentation method may include the following steps:

[0052] Step 201: Acquire point cloud data from the lidar and millimeter-wave signals from the millimeter-wave radar.

[0053] Point cloud data and millimeter-wave signals were obtained by sampling the same area using lidar and millimeter-wave radar, respectively.

[0054] The aforementioned "same area" can refer to any area within the sensing range of both the lidar and the millimeter-wave radar, without limitation. For example, when the lidar is an automotive lidar and the millimeter-wave radar is an automotive millimeter-wave radar, the aforementioned "same area" could be the area in front of the vehicle.

[0055] Taking a 24G baseband radar as an example, the millimeter-wave radar uses frequency modulated continuous wave (FMCW) to perform spatial measurements on the same area. It transmits and receives FMCW signals with a base frequency of 24GHz from the radio frequency antenna (including transmitting and receiving antennas). The echo received by the receiving antenna (i.e., the received continuously modulated electromagnetic wave) is sampled by AD, and the AD sampled echo is stored in a register. After receiving one frame of millimeter-wave signal, it is output from the network interface to the computer device.

[0056] After the lidar scans the same area to obtain point cloud data, noise can be filtered out to reduce noise interference and improve the accuracy of target detection.

[0057] Step 202: Obtain three-dimensional cubic data based on millimeter-wave signals.

[0058] Among them, the three-dimensional cubic data contains information in the dimensions of distance, angle and velocity, that is, the three-dimensional solid data contains information in three dimensions such as distance, angle and velocity.

[0059] In one embodiment, the millimeter-wave signal can be subjected to Fourier transforms in the distance, velocity, and angle dimensions to obtain three-dimensional cubic data. This is achieved by processing the millimeter-wave signal using a Three Dimensional Fast Fourier Transform (3DFFT). Alternatively, the millimeter-wave signal can be processed using FFT and algorithms such as BeamForming (BF), Capon, and MUSIC to obtain three-dimensional data; no specific limitation is made here. The 3DFFT includes a range FFT for the distance dimension, a doppler FFT for the velocity dimension, and an angle FFT for the angle dimension.

[0060] like Figure 3The diagram illustrates an example of millimeter-wave signal processing. Millimeter-wave radar detects targets within its sensing range by transmitting and receiving FMCW signals. The FMCW signal transmitted by the millimeter-wave radar's transmitting antenna is divided into multiple chirps (…). Figure 3 Only four chirs are shown. The intermediate frequency (IF) signal from each returned chirp is sampled using an A / D algorithm and then subjected to an FFT. The target's position can be calculated based on the resulting peak frequency; this process is called range FFT. Based on the doppler frequency shift caused by the target's motion velocity, an inter-chirp FFT is performed on the results of the range FFT for all chirs in a single antenna; this is called doppler FFT. The target's velocity can be calculated based on the resulting peak frequency (doppler FFT result). Based on the phase difference between different receiving antennas caused by the target's angle, an inter-antenna FFT is performed; this is called angle FFT. The target's angle can be calculated based on the resulting peak frequency (angle FFT result). Due to the separability of the Fourier transform, the above process can be considered as a 3DFFT. The target's angle can also be calculated using algorithms such as BF, Capon, and MUSIC.

[0061] Step 203: Obtain the heatmap distribution map based on the three-dimensional cubic data.

[0062] The heatmap is an information distribution map in the distance and angle dimensions, containing the energy of points corresponding to distance and angle. The energy of points corresponding to distance and angle can be represented by the signal-to-noise ratio (SNR).

[0063] Three-dimensional data contains information in three dimensions: distance, velocity, and angle. The distance and angle information can be extracted from these three dimensions to obtain a heatmap distribution map.

[0064] like Figure 4 The image shows an example of a vehicle traveling on a road. Vehicle A is equipped with a lidar (i.e., vehicle-mounted lidar) and a millimeter-wave radar (i.e., vehicle-mounted millimeter-wave radar). Targets 1, 2, and 3 are located in the area ahead of vehicle A. The lidar and millimeter-wave radar on vehicle A collect data from this area, obtaining point cloud data and millimeter-wave signals. 3D DFFT processing of the millimeter-wave signals yields three-dimensional data. Based on this three-dimensional data, the following can be obtained: Figure 5 The example image shown is a heatmap distribution diagram. Figure 5It can be seen that a heatmap contains the signal-to-noise ratio (SNR) of points corresponding to distance and angle. Points with higher SNR in a heatmap are usually the target. Among them, Figure 4 X and Y are two number axes of a Cartesian coordinate system established with vehicle A as the origin. 46.7dB is the signal-to-noise ratio of target 2, 51.6dB is the signal-to-noise ratio of target 1, and 37.7dB is the signal-to-noise ratio of target 3.

[0065] Step 204: Extract targets from the heatmap distribution map to determine the targets in the heatmap distribution map.

[0066] When a millimeter-wave radar transmits an FMCW signal towards the same area, targets within that area will reflect the signal, resulting in higher energy. Therefore, targets within the same area can be extracted from the heatmap based on the energy distribution.

[0067] As an alternative embodiment, the energy peak can be used as the target in the heatmap distribution map.

[0068] Because the target has high energy, all energy peaks in the heatmap distribution can be considered as the target. For example, in Figure 5 The heatmap distribution map has four energy peaks, and all four energy peaks can be used as targets, that is, there are four targets in the heatmap distribution map.

[0069] Step 205: Based on the target, determine the target point cloud set corresponding to the point cloud data that is spatially synchronized with the heatmap distribution map.

[0070] The target point cloud set can be the set of point clouds that constitute the target in the heatmap distribution map.

[0071] With the heatmap distribution map and point cloud data spatially synchronized, the target point cloud set corresponding to the target in the point cloud data can be obtained based on the target in the heatmap distribution map.

[0072] As an optional embodiment, based on the target, the set of target point clouds corresponding to the point cloud data spatially synchronized with the heatmap distribution map is determined, including:

[0073] Given that the point cloud data and the heatmap distribution are spatially synchronized, determine whether there are at least two target point clouds in the point cloud data within the 3 sigma distribution of the target.

[0074] If at least two target point clouds exist in the point cloud data within the 3sigma distribution of the target, then at least two target point clouds constitute the target point cloud set.

[0075] Depend on Figure 5 As can be seen, the target in the heatmap distribution can be described by a 3 sigma distribution. Therefore, when the point cloud data and the heatmap distribution are spatially synchronized, the corresponding target point cloud set can be obtained by determining whether there are at least two target point clouds in the point cloud data within the 3 sigma distribution of the target.

[0076] Since a target is typically composed of a large number of target point clouds, a quantity threshold can be set to detect whether the target actually exists. Specifically, when a target point cloud exists within the point cloud data within the target's 3 sigma distribution, the number of target point clouds can be counted. If the number of target point clouds is greater than the quantity threshold, the target can be determined to exist, and subsequent steps can be performed to augment the target's point cloud set. If the number of target point clouds is less than or equal to the quantity threshold, the target can be determined to not exist (i.e., the target is a false target), and no further steps need to be performed, thus reducing the target's consumption of computing resources.

[0077] As an optional embodiment, markers are placed within the sensing range of the lidar and millimeter-wave radar; it also includes:

[0078] Acquire the position information of the marker in the lidar coordinate system and the millimeter-wave radar coordinate system, respectively;

[0079] Based on the position information of the marker in the lidar coordinate system and the millimeter-wave radar coordinate system, the relative pose between the lidar coordinate system and the millimeter-wave radar coordinate system is obtained.

[0080] Based on the relative pose between the lidar coordinate system and the millimeter-wave radar coordinate system, the point cloud data and the millimeter-wave signal are spatially synchronized to achieve spatial synchronization between the point cloud data and the heatmap distribution map.

[0081] Among them, the aforementioned markers can be metal markers with strong reflective properties, such as corner reflectors.

[0082] To minimize the impact of other targets with strong echo characteristics on the spatial calibration of millimeter-wave radar and lidar, in the spatial calibration environment of millimeter-wave radar and lidar, FMCW signals from other locations in the environment, except for the aforementioned markers, will not generate echoes once absorbed. For lidar, apart from the aforementioned markers being white, all other locations are made of light-absorbing black material.

[0083] A lidar coordinate system can refer to a coordinate system established with the lidar as its origin. A millimeter-wave radar coordinate system can refer to a coordinate system established with the millimeter-wave radar as its origin.

[0084] The position information of the marker in the lidar coordinate system includes the distance between the marker and the lidar, as well as the angle of the marker in the lidar's field of view.

[0085] The location information of the marker in the millimeter-wave radar coordinate system includes the distance between the marker and the millimeter-wave radar, as well as the angle of the marker in the view of the millimeter-wave radar.

[0086] This embodiment uses a marker as a reference point. Based on the marker's position information in both the millimeter-wave radar coordinate system and the lidar coordinate system, the marker can be rotated, translated, and aligned in space to obtain a rotation matrix and a translation vector. These rotation matrix and translation vector are the parameters in the coordinate transformation relationship between the lidar and millimeter-wave radar. This coordinate transformation relationship represents the relative pose between the lidar and millimeter-wave radar coordinate systems. This process requires no manual intervention, improving the efficiency and accuracy of spatial synchronization between the millimeter-wave radar and the lidar. After obtaining the coordinate transformation relationship between the lidar and millimeter-wave radar, all point clouds in the point cloud data can be transformed into a heatmap distribution map based on this relationship. The target point cloud set corresponding to the target can then be obtained based on the target's position in the heatmap distribution map (for example, the point cloud located within the target's position in the heatmap distribution map is the target point cloud corresponding to that target).

[0087] It should be noted that after acquiring point cloud data and millimeter wave signals, the computer equipment can first spatially synchronize the point cloud data and millimeter wave signals, and then acquire three-dimensional data based on the millimeter wave signals after spatial synchronization is completed. Alternatively, it can first acquire three-dimensional data based on the millimeter wave signals and then spatially synchronize the point cloud data and millimeter wave signals. No limitation is made here.

[0088] As an optional embodiment, after spatially synchronizing the point cloud data and the millimeter-wave signal, the method further includes:

[0089] Regularization processing is performed on point cloud data and millimeter wave signals respectively to unify the amplitude range of point cloud data and millimeter wave signals.

[0090] By performing regularization processing on point cloud data and millimeter wave signals, the point cloud data and millimeter wave signals can be unified within the same amplitude range, so that computer equipment can quickly find the target point cloud from the point cloud data.

[0091] As an optional embodiment, before the point cloud data and heatmap distribution map are spatially synchronized, the following is also included:

[0092] Point cloud data and millimeter-wave signals are aligned with frame numbers on the same time axis to synchronize their time. Point cloud data and millimeter-wave signals are simultaneously output by lidar and millimeter-wave radar, respectively. The first radar transmits and receives electromagnetic waves after receiving trigger pulses sent by the second radar at fixed intervals. The first radar is either lidar or millimeter-wave radar, and the second radar is any radar other than the first radar.

[0093] To achieve time synchronization between point cloud data and millimeter-wave signals, a combination of software and hardware synchronization can be used to improve the accuracy of time synchronization. Hardware synchronization involves the second radar sending a trigger pulse to the first radar at fixed intervals. Upon receiving the trigger pulse, the first radar transmits and receives electromagnetic waves, and both radars simultaneously output sampled signals (i.e., point cloud data and millimeter-wave signals). Software synchronization, building upon hardware synchronization, aligns the frame numbers of the point cloud data output by the lidar and the millimeter-wave signal output by the millimeter-wave radar on the same time axis.

[0094] In a hardware synchronization application scenario, the first radar is a millimeter-wave radar, and the second radar is a lidar. When the lidar rotates and scans, it sends a trigger pulse to the millimeter-wave radar each time its motor passes its zero point. Upon receiving this trigger pulse, the millimeter-wave radar begins transmitting FMCW signals at a base frequency of 24GHz or 77GHz. Simultaneously, the millimeter-wave radar's receiving antenna begins receiving electromagnetic echoes. After transmission ends, reception also essentially ends, and the lidar has also rotated and scanned the same field of view. At this point, the sampled signals from both radars are output as the same frame.

[0095] Step 206: Perform data augmentation on the target point cloud set.

[0096] In this context, data augmentation of the target point cloud set can be understood as increasing the density of the point cloud data.

[0097] Data augmentation of target point cloud datasets primarily leverages the long-range detection capabilities of millimeter-wave radar to address the limitations of lidar, such as sparse point clouds and limited detection range. Furthermore, point cloud data augmentation schemes based on heatmap distribution maps can resolve issues of missed detections and false detections in point cloud-level or target-level fusion.

[0098] In an example diagram, Figure 4 Target 2 in the image is a distant target detected by the lidar of vehicle A. Figure 5 The energy peak at the longest distance is target 2, which is... Figure 5It can be seen that the probability distribution of the millimeter-wave signal location of target 2 covers a range from -20° to 20°, with a distance of approximately 80 meters. Compared to the probability distribution of angle, the probability distribution of distance is more concentrated. When the lidar detects target 2 simultaneously, due to the sparsity of the lidar at a distance, with a scanning angle of 0.2°, the point distance at 80 meters is 28cm. A car's width is approximately 1.8m, so for this car (i.e., target 2), there are only 6 to 7 target point clouds. Since both the millimeter-wave radar and lidar detect the presence of target 2, the target point clouds of target 2 will appear in the peaks of the millimeter-wave signal and also in the energy peaks of the heatmap distribution. By performing data augmentation on the 6 to 7 target point clouds of target 2, denser point cloud data can be obtained.

[0099] This application embodiment acquires point cloud data and millimeter-wave signals obtained by sampling the same area using lidar and millimeter-wave radar. Three-dimensional data is first obtained based on the millimeter-wave signals, then a heatmap distribution is obtained based on the three-dimensional data, and targets are extracted from the heatmap distribution to determine the targets within it. Based on the targets in the heatmap distribution, the corresponding target point cloud set in the point cloud data spatially synchronized with the heatmap distribution can be determined. By performing data augmentation on the target point cloud set, the density of the lidar point cloud data (especially the point cloud data at the far end of the lidar) can be increased, improving the lidar's accuracy in detecting targets.

[0100] See Figure 6 This is a schematic diagram illustrating the implementation process of a point cloud data augmentation method provided in another embodiment of this application. This point cloud data augmentation method is applied to... Figure 1 The computer equipment in the multi-sensor system shown. Figure 6 As shown, the point cloud data augmentation method may include the following steps:

[0101] Step 601: Acquire point cloud data from the lidar and millimeter-wave signals from the millimeter-wave radar.

[0102] This step is the same as step 201. For details, please refer to the relevant description of step 201. It will not be repeated here.

[0103] Step 602: Obtain three-dimensional cubic data based on millimeter-wave signals.

[0104] This step is the same as step 202, and you can refer to the relevant description of step 202 for details, which will not be repeated here.

[0105] Step 603: Obtain the heatmap distribution map based on the three-dimensional cubic data.

[0106] This step is the same as step 203, and you can refer to the relevant description of step 203 for details, which will not be repeated here.

[0107] Step 604: Extract targets from the heatmap distribution map to determine the targets in the heatmap distribution map.

[0108] This step is the same as step 204, and you can refer to the relevant description of step 204 for details, which will not be repeated here.

[0109] Step 605: Based on the target, determine the target point cloud set corresponding to the point cloud data that is spatially synchronized with the heatmap distribution map.

[0110] This step is the same as step 205, and you can refer to the relevant description of step 205 for details, which will not be repeated here.

[0111] Step 606: Fit the target point cloud in the target point cloud set at equal intervals.

[0112] By performing equal-interval fitting on the target point cloud, a reasonable enhancement of the point cloud density can be ensured. This equal-interval fitting, also known as equal-interval interpolation, is a linear interpolation method. Of course, other interpolation methods can also be used to enhance point cloud density, and this application does not limit the specific methods used.

[0113] As an optional embodiment, the target point cloud in the target point cloud set can be fitted at equal intervals using a power function. That is, equal interval fitting is achieved using a power function.

[0114] This application embodiment acquires point cloud data and millimeter-wave signals obtained by sampling the same area using lidar and millimeter-wave radar. Three-dimensional data is first obtained based on the millimeter-wave signals, then a heatmap distribution is obtained based on the three-dimensional data, and targets are extracted from the heatmap distribution to determine the targets within it. Based on the targets in the heatmap distribution, the corresponding target point cloud set in the point cloud data spatially synchronized with the heatmap distribution can be determined. By fitting the target point cloud at equal intervals, a reasonable enhancement of the point cloud density can be ensured, improving the detection accuracy of the lidar.

[0115] See Figure 7 This is a schematic diagram of the structure of a point cloud data enhancement device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0116] The aforementioned point cloud data augmentation device includes:

[0117] The signal acquisition module 71 is used to acquire point cloud data of the lidar and millimeter wave signal of the millimeter wave radar. The point cloud data and the millimeter wave signal are obtained by sampling the same area by the lidar and the millimeter wave radar, respectively.

[0118] Data acquisition module 72 is used to acquire three-dimensional data based on the millimeter-wave signal; the three-dimensional data includes information on distance, velocity, and angular dimensions;

[0119] The distribution map acquisition module 73 is used to acquire a heatmap distribution map based on the three-dimensional data; the heatmap distribution map is an information distribution map of distance and angle dimensions, and the information map contains the energy of points corresponding to distance and angle;

[0120] Target extraction module 74 is used to extract targets from the heatmap distribution map and determine the targets in the heatmap distribution map;

[0121] The set determination module 75 is used to determine, based on the target, the target point cloud set corresponding to the point cloud data that is spatially synchronized with the heatmap distribution map;

[0122] The data augmentation module 76 is used to perform data augmentation on the target point cloud set.

[0123] Optionally, the target extraction module 74 described above is specifically used for:

[0124] The energy peak is used as the target in the heatmap distribution map.

[0125] Optionally, the above set determination module 75 is specifically used for:

[0126] When the point cloud data and the heatmap distribution map are spatially synchronized, determine whether there are at least two target point clouds in the point cloud data within the 3 sigma distribution of the target.

[0127] If at least two target point clouds exist in the point cloud data within the 3sigma distribution of the target, then the at least two target point clouds constitute the target point cloud set.

[0128] Optionally, markers are placed within the sensing range of the lidar and the millimeter-wave radar; the point cloud data enhancement device further includes:

[0129] The information acquisition module is used to acquire the position information of the marker in the lidar coordinate system and the position information in the millimeter-wave radar coordinate system, respectively;

[0130] The pose acquisition module is used to acquire the relative pose between the lidar coordinate system and the millimeter-wave radar coordinate system based on the position information of the marker in the lidar coordinate system and the position information of the millimeter-wave radar coordinate system, respectively.

[0131] The spatial synchronization module is used to spatially synchronize the point cloud data and the millimeter-wave signal according to the relative pose between the lidar coordinate system and the millimeter-wave radar coordinate system, so as to complete the spatial synchronization of the point cloud data and the heatmap distribution map.

[0132] Optionally, the point cloud data augmentation device mentioned above further includes:

[0133] An amplitude processing module is used to perform regularization processing on the point cloud data and the millimeter wave signal respectively, so as to unify the amplitude range of the point cloud data and the millimeter wave signal.

[0134] Optionally, the point cloud data augmentation device mentioned above further includes:

[0135] A time synchronization module is used to align the point cloud data and the millimeter-wave signal with frame numbers on the same time coordinate axis to synchronize the point cloud data and the millimeter-wave signal in time. The point cloud data and the millimeter-wave signal are output simultaneously by the lidar and the millimeter-wave radar, respectively. The first radar transmits and receives electromagnetic waves after receiving trigger pulses sent by the second radar at fixed intervals. The first radar is either the lidar or the millimeter-wave radar, and the second radar is any radar other than the first radar among the lidar and the millimeter-wave radar.

[0136] Optionally, the data acquisition module 72 described above is specifically used for:

[0137] The millimeter-wave signal is subjected to Fourier transforms in terms of distance, velocity, and angle dimensions to obtain the three-dimensional cubic data.

[0138] Optionally, the data augmentation module 76 described above is specifically used for:

[0139] The target point cloud in the target point cloud set is fitted at equal intervals.

[0140] Optionally, the data augmentation module 76 described above is specifically used for:

[0141] The target point cloud in the target point cloud set is fitted at equal intervals using a power function.

[0142] The point cloud data enhancement device provided in this application embodiment can be applied in the foregoing method embodiments. For details, please refer to the description of the above method embodiments, which will not be repeated here.

[0143] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 8 As shown, the computer device in this embodiment includes: one or more processors 80 (only one is shown in the figure), a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, it implements the steps in the above-described point cloud data augmentation method embodiment.

[0144] The computer device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc. Optionally, the computer device may also include an antenna array.

[0145] The processor 80 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0146] The memory 81 can be an internal storage unit of the computer device, such as a hard drive or RAM. The memory 81 can also be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 81 can include both internal and external storage units. The memory 81 is used to store the computer program and other programs and data required by the computer device. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0148] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0149] This application also provides a computer program product that, when run on a computer device, enables the computer device to perform the steps described in the above-described method embodiments.

[0150] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0151] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0152] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0153] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0155] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A point cloud data augmentation method, characterized in that, The point cloud data augmentation method includes: The point cloud data of the lidar and the millimeter-wave signal of the millimeter-wave radar are acquired, wherein the point cloud data and the millimeter-wave signal are obtained by sampling the same area by the lidar and the millimeter-wave radar, respectively. Three-dimensional cubic data is obtained based on the millimeter-wave signal; the three-dimensional cubic data includes information on distance, velocity, and angular dimensions; A heatmap distribution map is obtained based on the three-dimensional cubic data; the heatmap distribution map is an information distribution map of distance and angle dimensions, and the information map contains the energy of the points corresponding to the distance and angle; Target extraction is performed on the heatmap distribution map to determine the targets in the heatmap distribution map; Based on the target, determine the target point cloud set corresponding to the point cloud data that is spatially synchronized with the heatmap distribution map; Perform data augmentation on the target point cloud set; The determination of the target point cloud set corresponding to the point cloud data spatially synchronized with the heatmap distribution map includes: When the point cloud data and the heatmap distribution map are spatially synchronized, it is determined whether there are at least two target point clouds in the point cloud data within the 3sigma distribution of the target, where the 3sigma distribution is used to describe the target in the heatmap distribution map. If at least two target point clouds exist in the point cloud data within the 3sigma distribution of the target, then the at least two target point clouds constitute the target point cloud set.

2. The method according to claim 1, characterized in that, Target extraction is performed on the heatmap distribution map to determine the targets in the heatmap distribution map, including: The energy peak is used as the target in the heatmap distribution map.

3. The method according to claim 1, characterized in that, The lidar and the millimeter-wave radar have markers positioned within their sensing range; it also includes: Obtain the position information of the marker in the lidar coordinate system and the position information in the millimeter-wave radar coordinate system, respectively; Based on the position information of the markers in the lidar coordinate system and the millimeter-wave radar coordinate system, the relative pose between the lidar coordinate system and the millimeter-wave radar coordinate system is obtained; Based on the relative pose between the lidar coordinate system and the millimeter-wave radar coordinate system, the point cloud data and the millimeter-wave signal are spatially synchronized to achieve spatial synchronization between the point cloud data and the heatmap distribution map.

4. The method according to claim 3, characterized in that, After spatially synchronizing the point cloud data and the millimeter-wave signal, the method further includes: The point cloud data and the millimeter wave signal are respectively regularized to unify the amplitude range of the point cloud data and the millimeter wave signal.

5. The method according to claim 1, characterized in that, Before the point cloud data and the heatmap distribution map complete spatial synchronization, the following steps are also included: The point cloud data and the millimeter-wave signal are aligned with their frame numbers on the same time axis to synchronize their time. The point cloud data and the millimeter-wave signal are simultaneously output by the lidar and the millimeter-wave radar, respectively. The first radar transmits and receives electromagnetic waves after receiving trigger pulses sent by the second radar at fixed intervals. The first radar is either the lidar or the millimeter-wave radar, and the second radar is any radar other than the first radar.

6. The method according to any one of claims 1 to 5, characterized in that, Obtaining three-dimensional cubic data based on the millimeter-wave signal includes: The millimeter-wave signal is subjected to Fourier transforms in terms of distance, velocity, and angle dimensions to obtain the three-dimensional cubic data.

7. The method according to any one of claims 1 to 5, characterized in that, Data augmentation of the target point cloud set includes: The target point cloud in the target point cloud set is fitted at equal intervals.

8. The method according to claim 7, characterized in that, Equal-interval fitting of the target point cloud in the target point cloud set includes: The target point cloud in the target point cloud set is fitted at equal intervals using a power function.

9. A point cloud data augmentation device, characterized in that, The point cloud data enhancement device includes: The signal acquisition module is used to acquire point cloud data from the lidar and millimeter-wave signals from the millimeter-wave radar. The point cloud data and the millimeter-wave signals are obtained by sampling the same area by the lidar and the millimeter-wave radar, respectively. The data acquisition module is used to acquire three-dimensional data based on the millimeter-wave signal; the three-dimensional data includes information on distance, velocity, and angle dimensions. The distribution map acquisition module is used to acquire a heatmap distribution map based on the three-dimensional data; the heatmap distribution map is an information distribution map of distance and angle dimensions, and the information map contains the energy of the points corresponding to the distance and angle; The target extraction module is used to extract targets from the heatmap distribution map and determine the targets in the heatmap distribution map; The set determination module is used to determine, based on the target, the target point cloud set corresponding to the point cloud data that is spatially synchronized with the heatmap distribution map; The data augmentation module is used to augment the target point cloud set. The set determination module is specifically used for: When the point cloud data and the heatmap distribution map are spatially synchronized, it is determined whether there are at least two target point clouds in the point cloud data within the 3sigma distribution of the target, where the 3sigma distribution is used to describe the target in the heatmap distribution map. If at least two target point clouds exist in the point cloud data within the 3sigma distribution of the target, then the at least two target point clouds constitute the target point cloud set.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 8.

11. A multi-sensor system, characterized in that, The multi-sensor system includes lidar, millimeter-wave radar, and the computer device as described in claim 10; The lidar and the millimeter-wave radar collect data from the same area; The lidar is used to output the collected point cloud data to the computer device; The millimeter-wave radar is used to output the collected millimeter-wave signals to the computer device.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images