Millimeter wave radar absolute depth estimation method based on monocular depth estimation large model and laser radar

Abstract

The invention discloses a millimeter wave radar absolute depth estimation method based on a monocular depth estimation large model and a laser radar. According to the method, the reasoning capability contained in a current existing monocular depth estimation large model is injected into a millimeter wave radar depth estimation model, but because the relative depth generated by monocular depth estimation lacks scale information, the relative depth is mapped by using a relatively dense laser radar, and the relative depth is estimated by using the laser radar. The relative depth is recovered to the absolute depth by estimating global linear mapping parameters, training of a millimeter wave radar depth estimation model is supervised, and improvement of depth precision and depth consistency is realized at the same time by introducing dense supervision and sparse supervision. According to the invention, world knowledge in a monocular depth estimation large model is efficiently introduced into a millimeter wave radar depth estimation model by using laser radar information through a supervised training framework, and the accuracy and consistency of a depth result are greatly improved on the premise of the same supervised data.

Description

Technical Field

[0001] The present invention relates to the supervision framework optimization of millimeter-wave radar depth completion. Through a unique framework design, the excellent performance of a large monocular depth estimation model is transferred to the millimeter-wave radar absolute depth estimation task, greatly improving the accuracy and consistency of the output results. Background Art

[0002] With the rapid development of intelligent driving technology, the perception of the surrounding environment, especially the demand for depth information acquisition, is very strong. Accurate depth information plays a decisive role in key technologies such as target detection and object segmentation. The current depth estimation technology has many different frameworks, including monocular depth estimation, binocular depth estimation, optical flow depth estimation, lidar depth completion, and millimeter wave radar depth completion. Among them, monocular depth estimation cannot obtain accurate absolute depth due to its own pathological nature, and can only obtain reasonable relative depth; binocular depth estimation cannot obtain good depth images due to the existence of occlusion problems; optical flow estimation requires sufficient lighting information and obtains continuous images with high frame rate to achieve good results; lidar depth can obtain reasonable and accurate depth images, but the cost is high and is greatly affected by weather; millimeter wave radar depth completion is a popular research topic at present. The high penetration and low cost of millimeter wave radar itself make it have high value at the application level, but the extreme sparsity and low signal-to-noise ratio of its depth information make millimeter wave radar depth completion difficult.

[0003] For the millimeter-wave radar depth completion algorithm, AD Singh et al. proposed a completion method based on millimeter-wave radar depth point filtering network and fusion network in the literature ["Depth estimation from camera image and mmWave radar point cloud," in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pp. 9275–9285, 2023.]. Compared with traditional methods, this method has achieved relatively good results, but its accuracy and consistency performance are far from practical applications. Summary of the invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the present invention introduces a dense depth supervision branch obtained by using a large monocular depth estimation model on the basis of traditional sparse lidar supervision. In order to eliminate the scale missing problem existing in monocular depth estimation, the global linear mapping parameters are estimated using the lidar image, and the relative depth is mapped to the absolute depth. A relatively accurate dense absolute depth map is obtained while ensuring depth consistency. Furthermore, in supervised training, through the simultaneous supervision of sparse depth and absolute depth, the accuracy and consistency of the depth completion results are improved under the same data and the same model conditions, so that depth estimation achieves better results in practical applications.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A millimeter-wave radar absolute depth estimation method based on a monocular depth estimation large model and a laser radar relies on a millimeter-wave radar absolute depth estimation model. The input of the millimeter-wave radar absolute depth estimation model is a visible light image and a sparse millimeter-wave radar image, and the output is a dense absolute depth map. The training process of the millimeter-wave radar absolute depth estimation model is as follows: first, a dense relative depth map is obtained by using a monocular depth estimation large model (the relative depth map only contains accurate depth structure information and lacks depth scale information), and a global linear mapping parameter from relative depth to absolute depth is estimated by using a sparse laser radar image, thereby realizing relative to absolute depth conversion and obtaining a dense absolute depth map (recovering accurate absolute depth while retaining structural information); then, the dense absolute depth map and the sparse laser radar image are used to supervise the training of the millimeter-wave radar depth estimation model, thereby obtaining the millimeter-wave radar depth estimation model.

[0007] In the above technical solution, the scale information of relative depth is further restored through a relatively dense LiDAR image. Since the ideal relationship between relative depth and absolute depth is a linear mapping relationship, the present invention estimates the global linear mapping parameters through sparse LiDAR depth information, and converts relative depth into absolute depth through precise linear parameter mapping.

[0008] Furthermore, the global linear mapping parameters of relative depth to absolute depth are estimated through the sparse LiDAR image, thereby realizing relative to absolute depth conversion and obtaining a dense absolute depth map. The specific method is to obtain the relative depth information of each LiDAR point according to the dense relative depth map, select the relative depth information and absolute depth information corresponding to the location of the LiDAR point, and form a set of relative-absolute depth information pairs. There is a linear mapping relationship between the relative-absolute depth information pairs, so the present invention performs iterative linear regression estimation on all relative-absolute depth information pairs to obtain global linear mapping parameters and realize relative-absolute depth conversion, which can ensure that the structural information of the relative depth remains consistent and the recovery of the scale information as much as possible. It is specifically implemented based on the following formula:

[0009]

[0010] d metric =s*d rel +t (5)

[0011] Among them, d rel represents the relative depth map estimated by the large monocular depth estimation model, d lidar represents the absolute depth map obtained from the LiDAR, ⊙ represents the two-dimensional matrix dot product, Indicates that only the relative depth map corresponding to the laser radar point is retained, flatten(·) means flattening the non-zero pixel values ​​in the image in row order, and They represent the relative depth information vector and the absolute depth information vector respectively, ILR(·) represents iterative linear regression estimation (Iterative LinearRegression), (s, t) represents the global linear mapping parameters, and d metric Represents the dense absolute depth map recovered by the global linear mapping parameters and the relative depth map.

[0012] Furthermore, the iterative linear regression estimation is performed on all relative-absolute depth information pairs to obtain global linear mapping parameters. The specific method is: multiple rounds of linear regression are performed on the selected relative-absolute depth information pairs. The first round of linear regression estimation is based on the original relative-absolute information pairs to obtain the initial linear regression parameters s i ,t i ; Perform relative-to-absolute depth conversion based on the linear regression parameters of the current round, remove the lidar points whose difference between the conversion result and the true absolute depth information is greater than the threshold v, and estimate the linear regression parameters s of the next round based on the relative-to-absolute information after removal i+1 ,t i+1; Perform the above estimation steps iteratively until the difference between the estimated absolute depth information corresponding to all lidar points and the true absolute depth information is less than the threshold v, stop the iteration, and obtain the global linear regression parameters (s, t).

[0013] Furthermore, the use of dense absolute depth maps and sparse lidar maps to supervise the training of the millimeter-wave radar depth estimation model can assist in the supervised training of any millimeter-wave radar depth completion network. Specifically, for the output of the millimeter-wave radar depth estimation model, the loss function result is calculated by L1 Loss with the sparse lidar map and the dense absolute depth map, and the training of the millimeter-wave radar depth estimation model is supervised. At the same time, dense supervision and sparse supervision are used to improve the depth accuracy and depth consistency of the model; this is specifically implemented based on the following formula:

[0014]

[0015] L total =L sparse +αL dense (9)

[0016] Where M represents any millimeter-wave radar depth estimation model that takes an image and a millimeter-wave radar image as input, I is a visible light image, and d radar The absolute depth map obtained by the millimeter wave radar, represents the absolute depth map estimated by the model, L sparse represents the loss function result obtained by using the absolute depth map estimated by the model and the dense absolute depth map, L dense represents the loss function result obtained by using the absolute depth map estimated by the model and the sparse lidar map, α represents the scale hyperparameter, L total Represents the overall loss function result used for gradient backpropagation.

[0017] The beneficial effects of the present invention are:

[0018] The method of the present invention utilizes the zero-sample reasoning capability contained in the existing monocular depth estimation large model and injects it into the millimeter-wave radar depth estimation model. However, since the relative depth generated by the monocular depth estimation lacks scale information, it is proposed to use a relatively dense laser radar to map the relative depth, and restore the relative depth to the absolute depth by estimating the global linear mapping parameters, and supervise the training of the millimeter-wave radar depth estimation model. By introducing dense supervision and sparse supervision, the depth accuracy and depth consistency are improved at the same time. The present invention uses the laser radar information to efficiently introduce the world knowledge in the monocular depth estimation large model into the millimeter-wave radar depth estimation model through the designed supervised training framework, and greatly improves the accuracy and consistency of the depth results under the premise of the same supervised data. In addition, the iterative linear regression estimation method proposed in this application improves the robustness to laser radar noise on the basis of traditional linear regression estimation, effectively eliminates the misleading of laser radar noise to the linear regression estimation process, and improves the accuracy of the linear mapping parameters obtained by linear regression estimation.

[0019] The supervised training framework of the millimeter-wave radar absolute depth estimation model of the present invention can effectively improve the accuracy and consistency of all existing millimeter-wave radar depth completion models, enhance the application value of millimeter-wave radar in the field of intelligent driving, and with the performance improvement of large monocular depth estimation models, the application value of this method will be further enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a supervised training framework for the millimeter wave radar absolute depth estimation model of the present invention;

[0021] Figure 2 (a) Visible light image, (b) millimeter wave radar visualization image, (c) lidar visualization image, (d) relative depth result image of the monocular depth estimation large model, (e) relative-to-absolute depth conversion result image based on lidar, (f) millimeter wave radar depth completion image obtained based on the supervised training framework;

[0022] Figure 3 (a) Visible light image, (b) LiDAR depth map, (c) millimeter-wave radar depth completion result map proposed by AD Singh et al., (d) millimeter-wave radar depth completion result map after processing by the present invention. DETAILED DESCRIPTION

[0023] The specific implementation modes of the present invention will be further described below in conjunction with the accompanying drawings.

[0024] A millimeter-wave radar absolute depth estimation method based on a monocular depth estimation large model and a laser radar is implemented by relying on a millimeter-wave radar absolute depth estimation model. Figure 1As shown in FIG. 1 , a supervised training framework of the millimeter wave radar absolute depth estimation model proposed in the present invention is provided. The training process of the model specifically includes the following steps:

[0025] 1. Through the zero-shot reasoning capability of the monocular depth estimation large model, Figure 2 (a) A dense relative depth map is obtained from the visible light image I shown in FIG. The present invention refers to the method used by L. Yang et al. in the document ["Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pp. 10371-10381, 2024.], and obtains a relatively good relative depth map d on the target dataset through a large monocular depth estimation model LM trained on multiple datasets. rel ,like Figure 2 (d). The process can be expressed as:

[0026] d rel =LM(I) (10)

[0027] 2. Estimate the global linear mapping parameters from relative depth to absolute depth through the sparse LiDAR image, thereby realizing the relative to absolute depth conversion and obtaining a dense absolute depth map. The scale information of relative depth can be restored through relatively dense LiDAR scanning. Since there is a linear mapping relationship between relative depth and absolute depth in an ideal situation, the present invention uses Figure 2 (c) is used to infer the global linear mapping parameters and convert the relative depth into absolute depth. Specifically, the relative depth information of each lidar point is obtained according to the dense relative depth map, and the relative depth information and absolute depth information corresponding to the position of the lidar point are selected to form a set of relative-absolute depth information pairs. This process can be expressed as:

[0028]

[0029]

[0030] Among them, d rel represents the relative depth map estimated by the large monocular depth estimation model, d lidar represents the absolute depth map obtained from the LiDAR, ⊙ represents the two-dimensional matrix dot product, Indicates that only the relative depth information of the corresponding position of the lidar point is retained, and flatten(·) means flattening the non-zero pixel values ​​in the image in row order. and Represent the relative depth information vector and the absolute depth information vector respectively.

[0031] Iterative linear regression is performed on all relative-absolute depth information pairs to obtain global linear mapping parameters to achieve relative-absolute depth conversion. Since the ideal depth information pairs satisfy the linear mapping relationship, the present invention estimates the global mapping parameters through linear regression to achieve relative-absolute depth conversion. The obtained results are as follows: Figure 2 (e) As shown in Figure 5. And while restoring the scale information as much as possible, such a framework can ensure that the structural information of relative depth remains consistent, which is specifically based on the following formula:

[0032]

[0033] d metric =s*d rel +t (15)

[0034] Among them, d rel represents the relative depth map estimated from the monocular depth estimation model, d lidar represents the absolute depth map obtained from the LiDAR, ⊙ represents the two-dimensional matrix dot product, Indicates that only the relative depth information of the corresponding position of the lidar point is retained, and flatten(·) means flattening the non-zero pixel values ​​in the image in row order. and They represent the relative depth information vector and the absolute depth information vector respectively, ILR(·) represents iterative linear regression estimation, (s, t) represents the global linear mapping parameters, s and t represent the slope and intercept respectively, d metric Represents the dense absolute depth map recovered by the global linear mapping parameters and the relative depth map.

[0035] 3. The absolute depth map obtained through the above steps can be used to assist in the supervised training of any mmWave radar depth completion model. Specifically, for an input such as Figure 2 The millimeter wave radar image shown in (b) and Figure 2 The output of the millimeter-wave radar depth estimation model for the visible light image I shown in (a) introduces a more detailed loss function calculation method. The L1 Loss considers both the sparse lidar image and the dense absolute depth map at the same time to enable the model to have a stronger depth completion capability. The results are shown in Figure 2 (f) shows that the results are similar to Figure 2(e) is close, indicating that the model has achieved consistency performance similar to that of the monocular depth estimation model and accuracy performance similar to that of the lidar. The specific implementation is based on the following formula:

[0036]

[0037] L total =L sparse +αL dense (19)

[0038] Where M represents any millimeter-wave radar depth estimation model that takes images and millimeter-wave radar images as input, and d radar The absolute depth map obtained by the millimeter wave radar, represents the absolute depth map estimated by the model, L sparse represents the loss function result obtained by using the absolute depth map estimated by the model and the dense absolute depth map, L dense represents the loss function result obtained by using the absolute depth map estimated by the model and the sparse lidar map, α represents the scale hyperparameter, L total Represents the overall loss function result used for gradient backpropagation.

[0039] Example 1

[0040] The following is a visual comparison of the effect of the millimeter wave radar depth estimation model obtained by the method of the present invention and the results of other millimeter wave radar depth completion models. Figure 2 It can be seen that (a) is a visible light image, (b) is a millimeter-wave radar visualization image with a high degree of sparsity, (c) is a lidar visualization image used to achieve relative-to-absolute depth conversion, (d) is a relative depth map obtained from a large monocular depth estimation model, (e) is an absolute depth map after conversion, and (f) is an inference map of millimeter-wave radar depth completion after training with the training framework corresponding to the present invention. It can be seen that the present invention can effectively convert the reasoning ability of the large monocular depth estimation model from relative to absolute depth through global linear mapping parameters, and restore more accurate scale information while ensuring structural consistency, which is used to assist in the training of the millimeter-wave radar depth estimation model. The results obtained also well retain the reasoning ability of the large monocular depth estimation model. Figure 3 It can be further seen that (a) is a visible light image, (b) is a lidar image, (c) is the millimeter-wave radar depth completion result proposed by ADSingh et al., and (d) is the millimeter-wave radar depth completion result of the present invention. It can be seen that the model obtained by using the supervised training framework proposed by the present invention has significantly better performance than other methods in terms of depth accuracy and consistency. Compared with (c), the results generated by the method of the present invention have better performance in terms of object depth consistency and sky segmentation.

[0041] The above are only specific implementation methods of the present invention, which cannot be used to limit the scope of the present invention. Equivalent changes made by ordinary technicians in this technical field based on this creation, as well as changes known to technicians in this field, should still fall within the scope of the present invention.

Claims

1. A millimeter wave radar absolute depth estimation method based on a monocular depth estimation large model and a laser radar, characterized in that: The invention relies on a millimeter-wave radar absolute depth estimation model, the input of which is a visible light image and a sparse millimeter-wave radar image, and the output is a dense absolute depth map; the training process of the millimeter-wave radar absolute depth estimation model is as follows: first, a dense relative depth map is obtained by using a monocular depth estimation large model, and a global linear mapping parameter from relative depth to absolute depth is estimated by using a sparse lidar image, thereby realizing relative to absolute depth conversion and obtaining a dense absolute depth map; then, the dense absolute depth map and the sparse lidar image are used to supervise the training of the millimeter-wave radar depth estimation model, thereby obtaining the millimeter-wave radar depth estimation model.

2. The method for absolute depth estimation of millimeter wave radar based on a monocular depth estimation large model and a laser radar according to claim 1, characterized in that: The global linear mapping parameters from relative depth to absolute depth are estimated through the sparse laser radar map, thereby realizing relative to absolute depth conversion and obtaining a dense absolute depth map. The specific method is: according to the dense relative depth map, the relative depth information of each laser radar point is obtained, and the relative depth information and absolute depth information corresponding to the position of the laser radar point are selected to form a set of relative-absolute depth information pairs; all relative-absolute depth information pairs are estimated by iterative linear regression to obtain global linear mapping parameters to realize relative-absolute depth conversion; it is specifically realized based on the following formula: d metric =s*d rel +t (5) Among them, d rel represents the relative depth map estimated by the large monocular depth estimation model, d lidar represents the absolute depth map obtained from the LiDAR, ⊙ represents the two-dimensional matrix dot product, Indicates that only the relative depth information of the corresponding position of the lidar point is retained, and flatten(·) means flattening the non-zero pixel values ​​in the image in row order. and denote the relative depth information vector and the absolute depth information vector, respectively. ILR(·) denotes iterative linear regression estimation. (s, t) denotes the global linear mapping parameters. s and t denote the slope and intercept, respectively. d metric Represents the dense absolute depth map recovered by the global linear mapping parameters and the relative depth map.

3. The method for absolute depth estimation of millimeter wave radar based on a monocular depth estimation large model and a laser radar according to claim 2, characterized in that: The iterative linear regression estimation is performed on all relative-absolute depth information pairs to obtain global linear mapping parameters. The specific method is: multiple rounds of linear regression are performed on the selected relative-absolute depth information pairs. The first round of linear regression estimation is based on the original relative-absolute information pairs to obtain the initial linear regression parameters s i , t i ; Perform relative-to-absolute depth conversion based on the linear regression parameters of the current round, remove the lidar points whose difference between the conversion result and the true absolute depth information is greater than the threshold v, and estimate the linear regression parameters s of the next round based on the relative-to-absolute information after removal i+1 , t i+1 ; Perform the above estimation steps iteratively until the difference between the estimated absolute depth information corresponding to all lidar points and the true absolute depth information is less than the threshold v, stop the iteration, and obtain the global linear regression parameters (s, t).

4. The method for absolute depth estimation of millimeter wave radar based on a monocular depth estimation large model and a laser radar according to claim 2, characterized in that: The method of using a dense absolute depth map and a sparse laser radar map to supervise the training of the millimeter wave radar depth estimation model is as follows: for the output of the millimeter wave radar depth estimation model, the loss function result is calculated by L1Loss with the dense absolute depth map and the sparse laser radar map respectively, and the training of the millimeter wave radar depth estimation model is supervised, which is specifically implemented based on the following formula: L total =L sparse +αL dense (9) Where M represents any millimeter-wave radar depth estimation model that takes an image and a millimeter-wave radar image as input, I is a visible light image, and d radar The absolute depth map obtained by the millimeter wave radar, represents the absolute depth map estimated by the model, L sparse represents the loss function result obtained by using the absolute depth map estimated by the model and the dense absolute depth map, L dense represents the loss function result obtained by using the absolute depth map estimated by the model and the sparse lidar map, α represents the scale hyperparameter, L total Represents the overall loss function result used for gradient backpropagation.

Images