Pixel-level relative-absolute depth conversion method based on radar point cloud

Abstract

The invention discloses a pixel-level relative-absolute depth conversion method based on radar point cloud. Radar point cloud sampling based on pseudo depth is carried out according to the structural relationship between a relative depth map and a radar point cloud map, and weighted estimation is carried out on a mapping parameter matrix through a small number of high-correlation radar point cloud points, so that relative-absolute depth conversion is realized; however, due to the fact that pixel-by-pixel estimation and global sampling can cause great reduction of calculation efficiency, multi-scale parameter estimation and a dynamic neighborhood sampling optimization method based on a DBSCAN algorithm are further provided, and great improvement of efficiency is achieved with extremely small performance loss. According to the invention, the depth conversion algorithm is combined with the optimization method, the high-performance zero-sample radar point cloud completion effect is realized, and particularly, good performance is achieved in the aspects of structural consistency and depth accuracy. According to the method, the relative depth estimation error is corrected, relatively good relative depth structure information is reserved as much as possible, and a high-precision depth completion result graph with a clear structure is obtained.

Description

Technical Field

[0001] The present invention relates to a pixel-level relative-to-absolute depth conversion method based on radar point cloud. The relative depth is converted into absolute depth by the method of the present invention to obtain a dense absolute depth map. The absolute depth map has been greatly improved in terms of structural consistency and depth accuracy. Background Art

[0002] With the rapid development of the automobile manufacturing industry, the demand for intelligent driving has gradually become prominent. Depth estimation technology is the core demand of intelligent driving. The current mainstream depth estimation technology is the depth completion method based on radar point cloud signals. The depth completion method based on radar point cloud signals is divided into supervised radar point cloud signal depth completion algorithm and unsupervised radar point cloud signal depth completion algorithm.

[0003] Among them, the supervised radar point cloud signal depth completion algorithm is based on neural network supervised learning. Through the radar signal data set containing the real depth map, the neural network learns the ability of depth completion from it. For example, Jinsun Park et al. proposed a depth completion method based on lidar in the literature ["Non-Local Spatial Propagation Network for Depth Completion,"ComputerVision–ECCV 2020:16th European Conference,Glasgow,UK,August 23–28,2020,Proceedings,Part XIII 16.Springer International Publishing,2020.], which is supervised and trained on KITTI and NYUv2 data sets. According to the diffusion of the real depth points of the lidar, dense depth images are iteratively obtained. However, due to the limited coverage of the data set itself, the supervised radar point cloud signal depth completion algorithm often has the problem of low generalization, resulting in poor depth completion effect for scenes that have not appeared in the data set.

[0004] The unsupervised radar point cloud signal depth completion algorithm uses the depth and spatial relationship of the radar point cloud signal itself and the smoothness prior assumption of the depth map to efficiently estimate the depth corresponding to the missing radar depth point. For example, Y. Zhao et al. proposed an unsupervised depth completion method based on the distribution characteristics of the radar point cloud itself in the literature ["A Surface Geometry Model for LiDAR Depth Completion, IEEE Robotics and Automation Letters, 6(3), 4457-4464.], which effectively achieves radar point cloud depth completion without the need for training samples by pre-filtering the radar point cloud and constraining the depth estimation based on the surface normal. However, due to the lack of information on the real depth image, the depth estimation accuracy and consistency are relatively poor. Summary of the invention

[0005] The purpose of the present invention is to split radar point cloud completion into two steps: relative depth estimation and relative-to-absolute depth conversion. The relative depth estimation is not within the scope of the present invention. The present invention focuses on the implementation of pixel-level relative-to-absolute depth conversion based on radar point cloud. The present invention provides a new paradigm for radar point cloud depth completion, which can realize pixel-level relative-to-absolute depth conversion. While correcting the relative depth estimation error, it retains the better relative depth structure information as much as possible, and obtains a high-precision depth completion result image with a clear structure.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A pixel-level relative-to-absolute depth conversion method based on radar point cloud, the method is: based on the relative depth map and the radar point cloud map, a linear mapping parameter matrix is ​​estimated, and relative-to-absolute depth conversion is achieved by this method, while retaining the structural features of the relative depth, an accurate absolute depth scale is obtained;

[0008] The specific method for estimating the mapping parameter matrix is ​​as follows: the linear mapping parameters corresponding to the pixel points in the relative depth map are calculated to form a linear mapping parameter matrix; for any pixel point in the relative depth map, the method for calculating its linear mapping parameters is as follows: the radar point cloud points adjacent to the pixel points corresponding to the pixel point in the radar point cloud map are sampled by a neighborhood sampling algorithm based on pseudo-depth, and the linear mapping parameters of the pixel point are estimated based on the relative depth value of the pixel point and the absolute depth value of the radar point cloud points adjacent to the pixel point obtained by sampling; the neighborhood sampling algorithm based on pseudo-depth is as follows: the traditional two-dimensional Euclidean distance is converted into a pseudo-three-dimensional distance through relative depth, and neighborhood sampling is performed based on the pseudo-three-dimensional distance.

[0009] The specific steps of the above method are as follows:

[0010] First, relative depth is obtained from the visible light image (specifically, the existing monocular depth method or other methods may be used). The relative depth only contains accurate depth structure information but lacks depth scale information.

[0011] Ideally, there should be a global linear mapping relationship between relative depth and absolute depth, which can be expressed as:

[0012] d metric =s*d rel +t

[0013] Among them, d metric represents the absolute depth, (s, t) represents the global linear mapping parameters, s and t represent the slope and intercept respectively, d rel Represents relative depth. That is, the ideal relative-to-absolute depth conversion method only needs to estimate these two parameters, but the relative depth obtained by the current method (such as monocular depth estimation) cannot be completely ideal. In order to give the relative-to-absolute depth conversion step a certain degree of freedom to eliminate the difference between non-ideal relative depth and absolute depth, the relationship between non-ideal relative depth and absolute depth is defined as a local linear mapping relationship, which can be expressed as:

[0014] d metric =S⊙d rel +T (1)

[0015] Among them, S∈R w×h and T∈R w×h Represents the linear mapping parameter matrix of each pixel, corresponding to the slope and intercept matrices respectively, and ⊙ represents the matrix dot product. By estimating the linear mapping matrix at the pixel level, the depth conversion of non-ideal relative depth can be effectively adapted, but the depth conversion at the pixel level will lead to a square increase in the amount of calculation. To solve this problem, the present invention proposes a multi-scale parameter estimation method, which adds a grid to the relative depth map, with a distance between N pixels, and only calculates the linear parameters on the grid points with a step size of N, and then upsamples to the original scale through the bilinear interpolation method. The bilinear interpolation method can be expressed as:

[0016] S = BilInterp(S ds ) (2)

[0017] T=BilInterp(T ds ) (3)

[0018] in, and The matrix represents the linear mapping parameters corresponding to the grid points with a step size of N. w and h are the width and height of the relative depth map. This method improves the computational efficiency by N. 2Multi-scale parameter estimation is based on the observation of parameter estimation continuity. It converts the full-image mapping parameter estimation into fewer parameter estimation calculations at the pixel positions corresponding to the downsampled grid, and uses a bilinear interpolation algorithm for upsampling, which greatly improves the computational efficiency of the algorithm with less accuracy loss.

[0019] In order to estimate the linear mapping parameters of the corresponding pixel points, the present invention performs neighborhood sampling on each pixel point of the parameter to be estimated. The neighborhood sampling is limited to a smaller range, K closer radar point cloud points are extracted, and the K radar point cloud points closest to the pixel point are used as depth reference points. However, since the two-dimensional Euclidean distance cannot be completely equivalent to the corresponding depth correlation, the neighborhood sampling directly based on the distance implies that the similarity between the actual depths is strongly correlated with the corresponding Euclidean distance. However, in the edge area, there is no correlation between some actual depth similarities and the Euclidean distance, which will cause the final linear mapping matrix estimation to produce deviations, resulting in a deterioration of the final effect. Therefore, the present invention further proposes a neighborhood sampling algorithm based on pseudo-depth, which expands the two-dimensional Euclidean distance to a pseudo-three-dimensional distance through the existing relative depth map, and performs neighborhood sampling based on the pseudo-three-dimensional distance, thereby effectively excluding misleading radar depth points with close Euclidean distances but low actual depth similarity, and weightedly estimates the mapping parameter matrix through a small number of highly correlated radar point cloud points. For a pixel point (x, y) and a radar point cloud point (x l ,y l ), the pseudo three-dimensional distance dist between the two is defined as follows:

[0020]

[0021] Among them, d rel (x, y) represents the relative depth value of the corresponding position. Based on the pseudo 3D distance, the KNN algorithm is used to obtain the K radar points closest to each pixel as the depth reference point. The process can be expressed as:

[0022]

[0023] in, represents the set of K radar point cloud points with the closest pseudo-three-dimensional distance to the pixel point (x, y), Represents the relative depth value set of pixel point (x, y), D r Represents the relative depth map. Furthermore, in order to eliminate the problem of insufficient generalization of the hyperparameter K in different situations, the DBSCAN algorithm is used to detect outliers on the K radar point cloud points, and the most concentrated cluster in the radar point cloud point set is used as the reference set of high-similarity radar point cloud points, thereby effectively removing the interference of low-correlation radar point cloud points. The process can be expressed as:

[0024]

[0025] in, Represents a set of high-similarity radar point cloud reference points.

[0026] Based on the obtained high-similarity radar point cloud reference set, the linear mapping parameters corresponding to the pixel point can be estimated by weighted least squares regression. The weights in the weighted least squares regression of the present invention are defined with reference to the pseudo-three-dimensional distance, so that the weight definition conforms to the depth correlation, and thereby obtains a depth estimation result that is more in line with the actual situation. The specific process can be expressed as:

[0027]

[0028] Among them, formula (8) is the weight definition based on pseudo three-dimensional distance, formula (9) is the least squares optimization problem definition, and formula (10) is the solution corresponding to the optimization problem. i,u,v is the weight defined based on the pseudo 3D distance, where i represents the i-th lidar point in the set of high-similarity radar point cloud reference points corresponding to the lidar point whose linear mapping parameters are to be estimated, (u, v) is the pixel coordinate of the linear mapping parameter to be estimated, b represents the variance control parameter, and D r (u, v, i) represent the absolute depth and relative depth value of the i-th adjacent lidar point when estimating the linear mapping parameters of the (u, v) position, respectively. [S ds (u,v),T ds (u,v)] represents the linear mapping parameter of the (u,v) position, S ds (u,v) and T ds (u,v) represents the slope component and the intercept component, and W represents the weight matrix corresponding to the reference set of high-similarity radar point cloud points in the current linear mapping parameter estimation process; and D r (u,v) respectively represent the absolute depth and relative depth value matrices of the high similarity radar point cloud point reference set corresponding to the lidar point at the position (u,v).

[0029] Substituting the corresponding values ​​(i.e., the relative depth value of the pixel in the relative depth map and the absolute depth value of the radar point cloud point adjacent to the pixel obtained by sampling) into formula (10), the result of the final linear mapping parameter matrix can be obtained, and the relative-to-absolute depth conversion at the pixel level can be obtained.

[0030] The beneficial effects of the present invention are:

[0031] The pixel-level relative-to-absolute depth conversion algorithm based on radar point cloud of the present invention can be combined with the monocular depth estimation method with good effect at present to obtain the corresponding absolute depth map, improve the depth completion effect and generalization based on radar point cloud, and make the more difficult absolute depth estimation degenerate into a simpler single-image relative depth estimation, greatly reducing the difficulty of the problem, and thereby greatly improving the effect of the method. Based on the multi-scale parameter estimation method, the DBSCAN algorithm is used to dynamically sample the neighborhood radar point cloud points on the downsampling grid; the neighborhood sampling algorithm based on pseudo-depth is given a pseudo-three-dimensional distance weight based on relative depth, and the local linear mapping parameter matrix is ​​estimated, and the final result is converted into a full-image linear mapping parameter matrix through bilinear interpolation, and the relative depth to absolute depth conversion is finally realized through linear mapping. The present invention proposes two efficiency optimization techniques of multi-scale parameter estimation and dynamic neighborhood sampling based on DBSCAN algorithm, which greatly improve the efficiency with minimal performance loss; the depth conversion algorithm is combined with the optimization technique to achieve high-performance zero-sample radar point cloud completion effect, especially in terms of structural consistency and depth accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Flowchart of pixel-level relative-to-absolute depth conversion method based on radar point cloud;

[0033] Figure 2 (a) Visible light image, (b) LiDAR visualization image, (c) relative depth map, (d) absolute depth conversion result;

[0034] Figure 3 (a) Visible light image, (b) LiDAR visualization image, (c) Unsupervised LiDAR depth completion result image proposed by Y. Zhao et al., (d) Absolute depth conversion result image of the present invention. DETAILED DESCRIPTION

[0035] The specific implementation modes of the present invention will be further described below in conjunction with the accompanying drawings.

[0036] like Figure 1 As shown, the pixel-level relative-to-absolute depth conversion method based on radar point cloud of the present invention comprises the following steps:

[0037] 1. Through monocular depth estimation from Figure 2(a) The relative depth map is obtained from the visible light image I shown in FIG. The present invention refers to the method used by L. Yang et al. in the document ["Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 10371-10381, 2024.], and obtains a monocular depth estimation large model LM by training on multiple data sets. Based on the model, the relative depth map d is obtained. r ,like Figure 2 (c). The process can be expressed as:

[0038] d r =LM(I) (11)

[0039] Among them, d r Represents a relative depth map.

[0040] 2. The radar point cloud is collected by the radar instrument and projected on a two-dimensional plane to form a radar point cloud map. m On the downsampling grid with a step size of 4, the neighboring radar point cloud points are sampled based on the pseudo three-dimensional distance to form a radar point cloud point set And the corresponding relative depth value set According to the weight based on the pseudo 3D distance, the relative depth value of the pixel in the relative depth map and the absolute depth value of the radar point cloud point adjacent to the pixel obtained by sampling are substituted into the formula corresponding to the solution of the weighted linear estimation to obtain the pixel-by-pixel linear mapping parameter [S ds ,T ds ]. Finally, the full-scale linear mapping parameter matrix [S, T] is obtained according to bilinear interpolation. In this way, the relative-absolute depth conversion result is achieved, such as Figure 2 (d) is shown. Figure 2 Compared with (a), the method of the present invention has achieved good results in depth accuracy and consistency.

[0041] Example 1

[0042] The following is a comparison of the absolute depth map effect obtained by the method of the present invention and the absolute depth map results obtained based on other radar point cloud depth completion models from a visual perspective. Figure 2It can be seen that (a) is a visible light image, (b) is a laser radar visualization image, (c) is a relative depth map obtained from a monocular depth estimation model, and (d) is an absolute depth map after conversion by the present invention. It can be seen that the present invention can effectively achieve accurate relative-to-absolute depth conversion based on radar point clouds, and restore relatively accurate scale information while ensuring structural consistency. Figure 3 It can be further seen that (a) is a visible light image, (b) is a lidar image, (c) is the depth completion result of the radar point cloud proposed by Y. Zhao et al., and (d) is the absolute depth map result after conversion by the method of the present invention. It can be seen that the pixel-level relative-to-absolute depth conversion method based on radar point cloud of the present invention has significantly better performance than other methods in terms of depth accuracy and consistency. Compared with (c), the results generated by the method of the present invention have more excellent performance in terms of object depth consistency and depth accuracy.

[0043] The above are only specific implementation methods of the present invention, which cannot be used to limit the scope of the present invention. Equivalent changes made by ordinary technicians in this technical field based on this creation, as well as changes known to technicians in this field, should still fall within the scope of the present invention.

Claims

1. A pixel-level relative-to-absolute depth conversion method based on radar point cloud, characterized by: The steps are as follows: Based on the relative depth map and radar point cloud map, the linear mapping parameter matrix is ​​estimated, and relative-to-absolute depth conversion is achieved to obtain an accurate absolute depth scale while retaining the structural features with good relative depth; The specific method for estimating the mapping parameter matrix is ​​as follows: the linear mapping parameters corresponding to the pixel points in the relative depth map are calculated to form a linear mapping parameter matrix; for any pixel point in the relative depth map, the method for calculating its linear mapping parameters is as follows: the radar point cloud points adjacent to the pixel points corresponding to the pixel point in the radar point cloud map are sampled by a neighborhood sampling algorithm based on pseudo-depth, and the linear mapping parameters of the pixel point are estimated based on the relative depth value of the pixel point and the absolute depth value of the radar point cloud points adjacent to the pixel point obtained by sampling; the neighborhood sampling algorithm based on pseudo-depth is as follows: the traditional two-dimensional Euclidean distance is converted into a pseudo-three-dimensional distance through relative depth, and neighborhood sampling is performed based on the pseudo-three-dimensional distance.

2. The pixel-level relative-to-absolute depth conversion method based on radar point cloud according to claim 1, characterized in that: In the pseudo-depth-based neighborhood sampling algorithm, the traditional two-dimensional Euclidean distance is converted into a pseudo-three-dimensional distance. The specific method is as follows: For pixel points (x, y) and radar point cloud points (x l ,y l ), the pseudo three-dimensional distance dist between the two is defined as follows: Among them, d rel (x, y) represents the relative depth value of the corresponding position.

3. The pixel-level relative-to-absolute depth conversion method based on radar point cloud according to claim 1, characterized in that: In the neighborhood sampling based on pseudo three-dimensional distance, the specific process of neighborhood sampling is as follows: for the sampled K radar point cloud points, the DBSCAN algorithm is used to take the most concentrated cluster in the set composed of the K radar point cloud points as the reference set of high similarity radar point cloud points, which is specifically expressed as follows: in, Represents a reference set of high-similarity radar point cloud points.

4. The pixel-level relative-to-absolute depth conversion algorithm based on radar point cloud according to claim 3 is characterized in that: The linear mapping parameter of the pixel is estimated based on the relative depth value of the pixel and the absolute depth value of the radar point cloud point adjacent to the pixel obtained by sampling. The specific method is: Based on the obtained high-similarity radar point cloud point reference set, the linear mapping parameters corresponding to the pixel points are estimated by weighted least squares regression; the weights in the weighted least squares regression are defined with reference to the pseudo three-dimensional distance, so that the weight definition conforms to the depth correlation, and the estimation process of the linear mapping parameters is expressed as: Among them, w i，u，v is the weight defined based on the pseudo 3D distance dist, i represents the i-th lidar point in the high similarity radar point cloud reference point set corresponding to the lidar point whose linear mapping parameter is to be estimated, (u, v) is the pixel coordinate of the linear mapping parameter to be estimated; b represents the variance control parameter, and formula (4) is the least squares optimization problem definition, and D r (u, v, i) represent the absolute depth and relative depth value of the i-th adjacent LiDAR point when estimating the linear mapping parameters of the (u, v) position, respectively. [S ds (u,v),T ds (u, v)] represents the linear mapping parameters of the (u, v) position, S ds (u, v) and T ds (u, v) represent the slope component and intercept component respectively, W(u, v) represents the weight matrix of the reference set of high similarity radar point cloud points corresponding to the lidar point at the position (u, v); and D r (u, v) respectively represent the absolute depth and relative depth value matrices of the high-similarity radar point cloud point reference set corresponding to the lidar point at the position (u, v).

5. The pixel-level relative-to-absolute depth conversion method based on radar point cloud according to claim 1, characterized in that: In the process of calculating the linear mapping parameters corresponding to the pixel points in the relative depth map, a multi-scale parameter estimation method is used to add a grid to the relative depth map, the distance between the grids is N pixels, the linear mapping parameters on the grid points with a step size of N are calculated, and then the bilinear interpolation method is used to upsample to the original scale.

Images