Super-resolution point cloud acquisition method suitable for laser imaging radar

Abstract

The present invention relates to a method for acquiring super-resolution point clouds suitable for laser imaging radar (LIDR) and belongs to the technical field of LDR. By combining point cloud and image data, the method successfully achieves super-resolution of the original point cloud. This method can effectively improve the resolution of point cloud data collected by the imaging LDR, obtaining more uniform point cloud data that exceeds hardware resolution.

Description

Technical Field

[0001] The present invention belongs to the technical field of laser imaging radar and relates to a super-resolution point cloud acquisition method suitable for laser imaging radar. Background Art

[0002] LiDAR uses lasers as its emission source and calculates the time of flight (ToF) to obtain target distance information. The obtained target scene information is output as a point cloud. Due to the limitations of imaging principles, device costs, and acquisition scenarios, the point cloud obtained by LiDAR is generally of limited density. In addition, point cloud data collected in real-world scenarios often has a certain degree of defects and noise, and is inevitably distributed along the trajectory of the radar scan, that is, it is sparse and uneven. LiDAR super-resolution refers to the use of algorithms to obtain a denser and more uniform point cloud based on the original sparse point cloud obtained by 3D LiDAR.

[0003] Existing deep learning super-resolution algorithms for point clouds fall into two main categories. The first involves reducing the point cloud's dimensionality into regularly arranged data formats, such as depth maps, bird's-eye views, and voxels, and extracting the point cloud's geometric structure from the relative positions of adjacent pixels or voxels. The second involves directly processing the point cloud data, using a kNN search algorithm to identify neighboring points and obtain their geometric data. Both approaches effectively improve point cloud resolution based on the original data, but they place certain demands on the density of the original data. For less dense point clouds, existing algorithms are unable to effectively increase the data's information entropy, as they lack sufficient detail in the target structure, making it difficult to achieve optimal results. Summary of the Invention

[0004] (1) Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is: how to design a super-resolution point cloud acquisition method suitable for laser imaging radar.

[0006] (2) Technical solution

[0007] To solve the above technical problems, the present invention provides a super-resolution point cloud acquisition method suitable for laser imaging radar. When a vehicle is moving, the laser imaging radar scans the environment around the vehicle, and the visible light camera captures images in the direction of travel. The radar point cloud is cropped using a pre-calibrated field of view, retaining only the point cloud content in the direction of travel.

[0008] The point cloud set obtained by laser imaging radar scanning is represented as Camera images are represented as a collection of pixels Where P is an N×3 matrix, representing the three-dimensional coordinate set of N points; Q is an m×n matrix, representing an image with a resolution of m×n. The mapping relationship between the two is expressed as (ui,vi)=P0·Tr·pi, where P0 is the projection matrix, representing the conversion relationship between three-dimensional points and two-dimensional projections, and Tr is the Euclidean transformation matrix, representing the transformation relationship from the radar point cloud to the image coordinate system;

[0009] Set the target super-resolution ratio to R, the image branch of the neural network projects the point cloud P into the image coordinate system, connects the coordinate information of each point with the corresponding pixel (ui, vi), and initially forms a higher-dimensional (m×n×(c+3) feature, which is then input into the encoder and decoder in sequence to obtain the generated high-density point cloud (r1m×r1n×1) and the confidence estimate corresponding to each point (r1m×r1n×1); in m×n×(c+3), 3 is the RGB color channel, and c is the number of feature dimensions other than RGB; in the high-density point cloud r1m×r1n×1, r1 is the super-resolution multiple. This formula converts the image into an image with a super-resolution r1 times, 1 is the depth channel, corresponding to 3 in m×n×(c+3); the confidence estimate r1m×

[0010] In r1n×1, 1 represents the confidence, and the value in the confidence estimate r1m×r1n×1 corresponds one-to-one to each value in the high-density point cloud r1m×r1n×1;

[0011] The point cloud branch of the neural network obtains the coordinate deviation (r2-1)N×3 of the new generated point and the confidence (r2-1)N×1 of the corresponding point through feature extraction, upsampling and coordinate reconstruction of the point cloud P, and obtains the new upsampled point cloud (r2-1)N×3 by adding it to the original point cloud P, and then combines it with the original point cloud P to obtain the output result r2N×3 of the point cloud branch of the neural network; the output result r2N×3 of the point cloud branch of the neural network and the output r1m×r1n×3 of the image branch of the neural network are weighted and summed according to the confidence, and finally a uniformly distributed super-resolution point cloud RN×3 is obtained by resampling.

[0012] Preferably, the laser imaging radar is a panoramic radar.

[0013] Preferably, residual correction is also performed during the upsampling process.

[0014] Preferably, the relative positions of the visible light camera and the laser imaging radar are fixed.

[0015] Preferably, the forward direction image is a visible light image in the target direction.

[0016] Preferably, the mapping relationship between the point cloud and the camera image is expressed in the form of projection, rotation, and translation matrices.

[0017] The present invention also provides a method for verifying the effectiveness of the method.

[0018] Preferably, in this method, experiments are conducted using simulated data and measured data. The simulated data downsamples the 64-line laser imaging radar data and compares it with the original data through super-resolution point cloud. The measured data is collected on the same route using 16-line laser imaging radar and 64-line laser imaging radar. The processed 16-line point cloud is aligned and compared with the 64-line point cloud. It is found that the difference between the super-resolution point cloud obtained by the two methods and the control group is within the allowable error range.

[0019] The present invention also provides a system for implementing the method.

[0020] The present invention also provides a laser imaging radar using the system.

[0021] (3) Beneficial effects

[0022] The present invention adopts a method of combining point cloud and image data to successfully achieve the super-resolution effect of the original point cloud, which can effectively improve the resolution of the point cloud data collected by the imaging lidar and obtain point cloud data that is higher than the hardware resolution and more uniform. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the radar and camera installation and coordinate definition involved in the present invention.

[0024] Figure 2 This is a schematic diagram of the mapping relationship between the radar and camera involved in the present invention;

[0025] Figure 3 It is a schematic diagram of the method principle of the present invention.

[0026] Among them: 1 radar coordinate system, 2 laser imaging radar, 3 camera coordinate system, 4 visible light camera, 5 target object, 6 two-dimensional projection, 7 point cloud branch of neural network, 8 image branch of neural network. DETAILED DESCRIPTION

[0027] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0028] The present invention provides a laser imaging radar super-resolution method, which is a method for super-resolution reconstruction of point clouds assisted by visible light images, and mainly includes the following steps:

[0029] During scene scanning, the laser imaging radar scans the target normally, while a camera fixed relative to the radar captures a visible light image in the target direction. The target scene data is registered, and the mapping relationship between the point cloud and the visible light image is output, expressed in the form of projection, rotation, and translation matrices. All of the above data is read and processed in two branches through a pre-trained neural network, including: 1) feature extraction, upsampling, and coordinate reconstruction; 2) encoding, decoding, and confidence-weighted merging; and finally, the output results of the two branches are integrated through farthest point sampling to obtain a uniform and dense super-resolution point cloud.

[0030] See Figure 1 Specifically, according to the present invention, when the vehicle is moving, the laser imaging radar (surrounding radar) 2 scans the environment around the vehicle, and the visible light camera 4 captures the image in the forward direction. The radar point cloud is cropped by the pre-calibrated field of view angle, and only the point cloud content in the forward direction is retained.

[0031] exist Figure 2 In the example, the point cloud set obtained by radar scanning is represented as Camera images are represented as a collection of pixels Where P is an N×3 matrix, representing a set of three-dimensional coordinates of N points; Q is an m×n matrix, representing an image with a resolution of m×n. The mapping relationship between the two is expressed as (ui,vi)=P0·Tr·pi, where P0 is the projection matrix, representing the conversion relationship between three-dimensional points and two-dimensional projections, and Tr is the Euclidean transformation matrix, representing the transformation relationship from the radar point cloud to the image coordinate system. Set the target super-resolution magnification to R. According to the above data, the image branch 8 of the neural network projects the point cloud P into the image coordinate system, connects the coordinate information of each point with the corresponding pixel (ui,vi), and preliminarily forms a higher-dimensional (m×n×(c+3) feature, which is then input into the encoder and decoder in turn to obtain the generated high-density point cloud (r1m×r1n×1) and the confidence estimate corresponding to each point (r1m×r1n×1). The point cloud branch 7 of the neural network obtains the point cloud P through feature extraction, upsampling and coordinate reconstruction. The coordinate deviation (r2-1)N×3 of the newly generated point and the confidence level (r2-1)N×1 of the corresponding point are added to the original point cloud P to obtain a new upsampled point cloud (r2-1)N×3, which is then combined with the original point cloud P to obtain the output r2N×3 of the point cloud branch 7 of the neural network. The output r2N×3 of the point cloud branch 7 of the neural network and the output r1m×r1n×3 of the image branch 8 of the neural network are weighted and summed according to the confidence level, and finally a uniformly distributed super-resolution point cloud RN×3 is obtained through resampling.

[0032] In m×n×(c+3), 3 represents the RGB color channels, and c represents the number of feature dimensions other than RGB. In the high-density point cloud r1m×r1n×1, r1 represents the super-resolution factor, converting the image into an image with a super-resolution factor of r1. 1 represents the depth channel, corresponding to the 3 in m×n×(c+3). In the confidence estimate r1m×r1n×1, 1 represents the confidence level, and the values ​​in the confidence estimate r1m×r1n×1 correspond one-to-one to each value in the high-density point cloud r1m×r1n×1. r2N×3 upsamples the original point cloud through a neural network and adds the original point cloud to the output, for a total of r2N xyz data.

[0033] The results of the present invention were tested using both simulated and measured data. The simulated data consisted of down-sampled 64-line lidar data, processed using the present invention, and compared with the original data. The measured data was collected using 16-line and 64-line lidar along the same route. The processed 16-line point clouds were registered and compared with the 64-line point clouds. The differences between the super-resolution point clouds obtained by both methods and the control group were found to be within the allowable error range, demonstrating the reliability and effectiveness of the present invention.

[0034] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for obtaining super-resolution point clouds suitable for laser imaging radar, characterized in that: As the vehicle moves, the laser imaging radar scans the environment around it, and the visible light camera captures images in the direction of travel. The radar point cloud is cropped using a pre-calibrated field of view, retaining only the point cloud content in the direction of travel. The point cloud set obtained by laser imaging radar scanning is represented as Camera images are represented as a collection of pixels Where P is an N×3 matrix, representing the three-dimensional coordinate set of N points; Q is an m×n matrix, which represents an image with a resolution of m×n. The mapping relationship between the two is expressed as (u i ,v i )=P0·T r ·p i , where P0 is the projection matrix, which represents the conversion relationship between three-dimensional points and two-dimensional projections, T r is the Euclidean transformation matrix, which represents the transformation relationship from the radar point cloud to the image coordinate system; Set the target super-resolution magnification to R, and the image branch of the neural network projects the point cloud P into the image coordinate system, and compares the coordinate information of each point with the corresponding pixel (u i ,v i ) are connected to initially form a higher-dimensional m×n×(c+3) feature, which is then input into the encoder and decoder in sequence to obtain the generated high-density point cloud r1m×r1n×1 and the confidence estimate r1m×r1n×1 corresponding to each point; in m×n×(c+3), 3 is the three RGB color channels, and c is the number of feature dimensions other than RGB; in the high-density point cloud r1m×r1n×1, r1 is the multiple of super resolution. This formula converts the image into an image with super resolution r1 times, and 1 is the depth channel, corresponding to 3 in m×n×(c+3); in the confidence estimate r1m×r1n×1, 1 represents the confidence, and the value in the confidence estimate r1m×r1n×1 corresponds one-to-one to each value in the high-density point cloud r1m×r1n×1; The point cloud branch of the neural network obtains the coordinate deviation (r2-1)N×3 of the new generated point and the confidence (r2-1)N×1 of the corresponding point through feature extraction, upsampling and coordinate reconstruction of the point cloud P, and obtains the new upsampled point cloud (r2-1)N×3 by adding it to the original point cloud P, and then combines it with the original point cloud P to obtain the output result r2N×3 of the point cloud branch of the neural network; the output result r2N×3 of the point cloud branch of the neural network and the output r1m×r1n×3 of the image branch of the neural network are weighted and summed according to the confidence, and finally a uniformly distributed super-resolution point cloud RN×3 is obtained by resampling.

2. The method according to claim 1, wherein The laser imaging radar is a panoramic radar.

3. The method according to claim 1, wherein During the upsampling process, residual correction is also performed.

4. The method according to claim 1, wherein The relative positions of the visible light camera and the laser imaging radar are fixed.

5. The method according to claim 1, wherein The forward direction image is a visible light image in the target direction.

6. The method according to claim 1, wherein The mapping relationship between the point cloud and the camera image is expressed in the form of projection, rotation, and translation matrices.

7. A method for verifying the effectiveness of the method as described in any one of claims 1 to 6, in which experiments are conducted using simulated data and measured data. The simulated data downsamples the 64-line laser imaging radar data and compares the super-resolution point cloud with the original data. The measured data is collected using a 16-line laser imaging radar and a 64-line laser imaging radar on the same route. The processed 16-line point cloud is aligned and compared with the 64-line point cloud. It is found that the difference between the super-resolution point clouds obtained by the two methods and the control group is within the allowable error range.

8. A system for implementing the method according to any one of claims 1 to 6.

9. A laser imaging radar using the system as claimed in claim 8.

Images