A depth estimation method based on millimeter wave radar-camera multi-stage fusion

Abstract

The present application relates to the technical field of depth estimation and multi-sensor fusion, and proposes a depth estimation method based on millimeter wave radar-camera multi-stage fusion, which uses sparse point cloud received by a millimeter wave radar and a single frame image. First, the monocular depth estimation result of the image is output by using an independent monocular depth estimator, and a quasi-dense depth map is output by using a radar pixel correlation network; then, a global alignment monocular depth estimation result and a quasi-dense scale map are obtained by using a scale estimator; finally, a dense depth estimation is completed by using a scale learner. The present application solves the problems of blur and detail loss in the direct fusion of millimeter wave radar data and images in the prior art, and introduces the robustness of millimeter wave point cloud to smoke, rain and fog into the depth estimation task, so that high-precision and high-robust dense depth estimation can be realized under various weather conditions.

Description

Technical Field

[0001] This invention relates to the field of depth estimation and multi-sensor fusion technology, specifically to a depth estimation method based on millimeter-wave radar-camera multi-stage fusion. Background Technology

[0002] Environmental perception is crucial in the field of autonomous driving, and accurate depth estimation forms the basis for key tasks such as planning, reconstruction, and detection. Optical sensors and ranging sensors are widely used in this field.

[0003] Optical sensor applications can be categorized into stereo and monocular methods. Compared to hardware-intensive stereo methods, monocular depth estimation is more practical and easier to implement. However, current monocular methods face the challenge of balancing absolute scale generalization and accuracy, thus requiring additional scale constraints. In contrast, ranging sensors such as lidar and millimeter-wave radar can directly provide the measurement scale. LiDAR generates dense and accurate 3D point clouds, but it has some shortcomings in terms of cost, power consumption, and data bandwidth. In comparison, millimeter-wave radar has significant advantages in cost and power consumption, and compared to other sensors such as cameras or lidar, it maintains excellent performance in harsh weather conditions (such as rain, snow, fog, etc.) due to its strong penetration and detection capabilities for targets in the environment. Traditional radar provides ranging, azimuth, and Doppler velocity measurements, hence the term 3D radar. Newly developed 4D radar can provide elevation dimension, generating even denser point clouds, thus showing great potential in the field of perception.

[0004] Cameras excel in autonomous driving perception by providing high-precision visual data and the ability to identify and reconstruct the environment. However, their performance can be limited in certain situations, such as low light or complex weather conditions. On the other hand, millimeter-wave radar offers direct ranging and excellent robustness, enabling it to provide reliable ranging data even in adverse weather, low light, or low visibility conditions. However, due to the sparsity and accuracy limitations of millimeter-wave radar data, using them alone for depth estimation may be insufficient to meet the requirements of high-precision perception. Therefore, fusing cameras and millimeter-wave radar is a promising approach. This multi-sensor fusion strategy fully leverages the high-precision visual information of cameras and the advantages of direct ranging and robustness of millimeter-wave radar. By combining their data, more accurate and robust depth estimation results can be obtained. This fusion method not only helps overcome the limitations of individual sensors but also improves the perception capabilities of autonomous driving systems under various complex environmental conditions, thereby enhancing safety and reliability.

[0005] To address the challenge of depth estimation through the fusion of millimeter-wave radar and camera data, several advanced methods and techniques have emerged. These include traditional sensor fusion techniques such as the Extended Kalman Filter (EKF) and Particle Filter, as well as the increasingly prominent deep learning techniques. Traditional sensor fusion methods aim to fuse data from different sensors to provide more accurate depth estimation results. However, these methods still face problems such as data inconsistency and noise, especially in complex driving scenarios.

[0006] In recent years, deep learning technology has made significant progress in the field of depth estimation. Deep learning models such as convolutional neural networks (CNNs) have been used to predict depth information from images, and through training on large amounts of labeled data, the accuracy of depth estimation has been improved.

[0007] Due to the sparsity and noise of millimeter-wave radar data, fusing it with camera data for depth estimation presents certain challenges. Past methods have primarily focused on the direct fusion of radar points and pixels, but this direct fusion often compromises high-quality details in the image and significantly impacts the accuracy of depth estimation. Summary of the Invention

[0008] To address the aforementioned problems, this invention proposes an innovative multi-stage fusion method based on depth estimation, distance sensing technology, and deep learning. This method combines the detail richness of monocular depth estimation with the robust distance scale of radar, thereby avoiding the image quality degradation and reduced measurement accuracy issues caused by direct fusion. The framework of this invention employs a multi-stage fusion strategy to fully leverage the respective advantages of millimeter-wave radar and cameras, improving depth estimation performance. On one hand, a scale predictor is used to globally align the monocular depth estimation results with the sparse radar depth. On the other hand, a radar pixel correlation network is used to enhance the sparse radar scale, obtaining a quasi-dense radar scale map. Subsequently, a scale learner is used to adjust the local metric scale, ultimately generating a dense depth map.

[0009] To achieve the above objectives, this invention provides a depth estimation method based on multi-stage fusion of millimeter-wave radar and camera, comprising the following steps:

[0010] S1. Output monocular depth estimation results through an independent monocular depth estimator;

[0011] S2. Output a quasi-dense depth map through a radar pixel correlation network;

[0012] S3. Obtain globally aligned monocular depth estimation results and quasi-dense scale maps through the scale predictor;

[0013] S4. Perform dense depth estimation using a scale learner.

[0014] Preferably, step S1 specifically includes the following steps:

[0015] S11. Use any monocular depth estimation network and load the corresponding pre-trained weights;

[0016] S12. Input a monocular RGB image, and obtain the depth information through the forward propagation of the network, without any particular limitation on the absolute scale. Or inverse depth estimation results

[0017] Preferably, step S2 specifically includes the following steps:

[0018] S21. Construct the architecture of the radar pixel correlation network;

[0019] S22. Traverse all radar points and obtain the radar point p projected onto the image. i Image slices of size H×W around the location C i , as input;

[0020] S23. For each set of inputs, calculate image slice C using the radar pixel correlation network. i Each pixel and p in i The confidence level is used to obtain the confidence graph. For k radar points, k confidence maps can be obtained;

[0021] S24. For each pixel x in the original image uv For each pixel, n∈[0,k] confidence levels can be found. The maximum value greater than the threshold τ can be selected from these to obtain the radar point depth corresponding to the pixel.

[0022] S25. Traverse each pixel to obtain the corresponding depth value, thus obtaining a quasi-dense depth map.

[0023] Preferably, the radar pixel correlation network consists of an image encoder, a radar encoder, a transformer module, and a multi-scale decoder. The image encoder is constructed using a standard ResNet18 backbone network, and the radar encoder is constructed using a multilayer perceptron with fully connected layers. The outputs of the image encoder and the radar encoder are connected to the transformer module, which consists of four layers of self-attention and cross-attention layers. The output of the transformer module, together with the multi-resolution outputs of each layer of the image encoder, is fed to the multi-scale decoder for multi-resolution decoding with a threshold control mechanism.

[0024] Preferably, step S3 specifically includes the following steps:

[0025] S31, Use and The ratio value is obtained at the quasi-dense scale.

[0026] S32, Using sparse radar point cloud p and Perform global scale alignment.

[0027] Preferably, step S32 specifically involves: for the input radar point depth and The depth of the corresponding pixel is used to calculate the global scale using the least squares method. and Applied to The pixels in the array have the smallest residual.

[0028] Preferably, step S4 specifically includes the following steps:

[0029] S41. Construct the scale learner network based on the MiDaS-Small network;

[0030] S42, The output of steps S31 and S32 is... and Input the scale learner network to obtain dense scale.

[0031] S43. The dense scale output by the scale learner network. and Multiplication yields dense depth

[0032] Preferably, step S42 specifically involves: taking The empty space in the middle is filled with a unit scale of 1; and The scale residual r is obtained by connecting the input to the scale learner network; the scale map s is calculated using ReLU(1+r), and the scale is applied to the network. Multiplication yields dense depth

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] This invention provides a depth estimation method based on multi-stage fusion of millimeter-wave radar and camera, which combines depth estimation with multi-sensor fusion technology. It uses a novel multi-stage fusion technique, which combines heterogeneous sensors of millimeter-wave radar and camera in the depth estimation process. It combines the high-fidelity details provided by vision with the robust distance scale provided by millimeter-wave radar, avoiding the quality loss and accuracy reduction caused by direct fusion. Attached Figure Description

[0035] Figure 1 This is a structural diagram of the method of the present invention;

[0036] Figure 2 This is a schematic diagram of the input method for the present invention;

[0037] Figure 3 This is a structural diagram of the converter module of the present invention;

[0038] Figure 4 This is a diagram showing the effect of the depth estimation output of this invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] This invention aims to solve the following problems in the prior art:

[0041] The trade-off between accuracy and robustness: In existing technologies, depth estimation using a single sensor typically requires a trade-off between accuracy and robustness. Optical sensors (cameras) provide high-precision visual information, but are easily limited under harsh conditions, such as low light or inclement weather. In contrast, millimeter-wave radar provides robust distance measurement, but its data sparsity and limited resolution can affect the accuracy of depth estimation.

[0042] The problem with direct fusion: Simply fusing camera and millimeter-wave radar data directly can lead to a decrease in image quality and a reduction in the accuracy of depth estimation. This is because the inconsistencies in the data characteristics and scales of different sensors can cause mutual interference.

[0043] By combining a camera and millimeter-wave radar and employing a multi-stage fusion strategy, this invention solves the above-mentioned problems and brings the following effects and advantages of multi-stage fusion:

[0044] The combined advantages of high accuracy and robustness: By combining camera and millimeter-wave radar data, this invention achieves high-precision visual information and robust radar range scale in depth estimation. This combined advantage enables accurate depth estimation under various environmental conditions, whether in good lighting or in adverse weather conditions (such as rain, snow, fog, etc.).

[0045] Advantages in avoiding loss of quality and accuracy: By employing a multi-stage fusion strategy, this invention avoids the image quality degradation and reduced measurement accuracy that can occur with direct fusion. This method fully leverages the strengths of both the camera and millimeter-wave radar while minimizing their weaknesses, thereby improving the performance of depth estimation.

[0046] In summary, this invention aims to solve the problem of balancing accuracy and robustness in existing technologies by combining camera and millimeter-wave radar data through a multi-stage fusion strategy, and avoids the quality and accuracy loss caused by direct fusion. This provides a more efficient and reliable depth estimation method, which is particularly suitable for autonomous driving and other perception applications.

[0047] This invention proposes a depth estimation method based on multi-stage fusion of millimeter-wave radar and camera, comprising the following steps:

[0048] S1. Output monocular depth estimation results through an independent monocular depth estimator;

[0049] S2. Output a quasi-dense depth map through a radar pixel correlation network;

[0050] S3. Obtain globally aligned monocular depth estimation results and quasi-dense scale maps through the scale predictor;

[0051] S4. Perform dense depth estimation using a scale learner.

[0052] The following is a detailed explanation of each step.

[0053] Step S1: Output the monocular depth estimation result through an independent monocular depth estimator.

[0054] Specifically, it mainly includes the following steps:

[0055] S11. Use any monocular depth estimation network and load the corresponding pre-trained weights. Optional monocular depth estimation networks include mainstream monocular depth estimation methods such as DPT and MiDaS. Although these methods do not directly output monocular depth estimates at the absolute scale, they have strong generalization ability.

[0056] S12. Input a monocular RGB image, and obtain the depth information through the forward propagation of the network, without any particular limitation on the absolute scale. Or inverse depth estimation results

[0057] Step S2: Output a quasi-dense depth map through the radar pixel correlation network.

[0058] Specifically, it mainly includes the following steps:

[0059] S21. Construct the architecture of the radar pixel correlation network. The radar pixel correlation network consists of an image encoder, a radar encoder, a transformer module, and a multi-scale decoder. The image encoder is constructed using a standard ResNet18 backbone network, and the radar encoder is constructed using a multilayer perceptron with fully connected layers. The outputs of the image encoder and the radar encoder are connected to the transformer module, which consists of four layers of self-attention and cross-attention layers. The output of the transformer module, together with the multi-resolution output of each layer of the image encoder, is fed to the multi-scale decoder for multi-resolution decoding with a threshold control mechanism.

[0060] S22. Traverse all radar points and obtain the radar point p projected onto the image. i Image slices of size H×W around the location C i , as input;

[0061] S23. For each set of inputs, calculate the image slice C using an associative network. i Each pixel and p in i The confidence level is used to obtain the confidence graph. For k radar points, k confidence maps can be obtained;

[0062] S24. For each pixel x in the original image uv For each pixel, n∈[0,k] confidence levels can be found. The maximum value greater than the threshold τ can be selected from these to obtain the radar point depth corresponding to the pixel.

[0063] S25. Traverse each pixel to obtain the corresponding depth value, thus obtaining a quasi-dense depth map.

[0064] Step S3: Obtain globally aligned monocular depth estimation results and quasi-dense scale maps through the scale predictor.

[0065] Specifically, it mainly includes the following steps:

[0066] S31, Use and The ratio value is obtained at the quasi-dense scale.

[0067] S32, Using sparse radar point cloud p and Perform global scale alignment. For the input radar point depth and... The depth of the corresponding pixel is used to calculate the global scale using the least squares method. and Applied to The pixels in the array have the smallest residual.

[0068] Step S4: Perform dense depth estimation using a scale learner.

[0069] Specifically, it mainly includes the following steps:

[0070] S41. Based on the MiDaS-Small network, construct a scale learner network, in which the ImageNet backbone network is initialized with pre-trained weights, and other network layers are randomly initialized;

[0071] S42, The output of steps S31 and S32 is... and Input the scale learner to obtain dense scale Will The empty space in the middle is filled with a unit scale of 1; and The input is connected to the scale learner network to obtain the scale residual r; the scale map s is calculated using ReLU(1+r), and the scale is applied to the network. Multiplication yields dense depth

[0072] S43, Dense scale output by the scale learner and Multiplication yields dense depth

[0073] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A depth estimation method based on multi-stage fusion of millimeter-wave radar and camera, characterized in that, Includes the following steps: S1. Output monocular depth estimation results through an independent monocular depth estimator; S2. Output a quasi-dense depth map through a radar pixel correlation network; S3. Obtain globally aligned monocular depth estimation results and quasi-dense scale maps through the scale predictor; S4. Perform dense depth estimation using a scale learner; Step S1 specifically includes the following steps: S11. Use any monocular depth estimation network and load the corresponding pre-trained weights; S12. Input a monocular RGB image, and obtain the depth information through the forward propagation of the network, without any particular limitation on the absolute scale. Or inverse depth estimation results ; Step S2 specifically includes the following steps: S21. Construct the architecture of the radar pixel correlation network; S22. Traverse all radar points and obtain the radar points projected onto the image. Image slices of size H×W around the location , as input; S23. For each set of inputs, calculate image slices using the radar pixel correlation network. Each pixel in The confidence level is used to obtain the confidence graph. For k radar points, k confidence maps can be obtained; S24. For each pixel in the original image All can be found A confidence level is selected from those that are greater than the threshold. The maximum value of the value can be used to obtain the radar point depth corresponding to the pixel. S25. Traverse each pixel to obtain the corresponding depth value, thus obtaining a quasi-dense depth map. ; The radar pixel correlation network consists of an image encoder, a radar encoder, a transformer module, and a multi-scale decoder. The image encoder uses a standard ResNet18 backbone network, and the radar encoder uses a multilayer perceptron with fully connected layers. The outputs of the image encoder and the radar encoder are connected to the transformer module, which consists of four layers of self-attention and cross-attention layers. The output of the transformer module, along with the multi-resolution outputs of each layer of the image encoder, is fed into the multi-scale decoder for multi-resolution decoding with a threshold control mechanism. Step S3 specifically includes the following steps: S31, Use and The ratio value is obtained at the quasi-dense scale. ; S32, Using sparse radar point cloud p and Perform global scale alignment; Step S32 specifically involves: for the input radar point depth and The depth of the corresponding pixel is used to calculate the global scale using the least squares method. and , applied to The pixels in the array have the smallest residual. ; Step S4 specifically includes the following steps: S41. Construct the scale learner network based on the MiDaS-Small network; S42, The output of steps S31 and S32 is... and Input the scale learner network to obtain dense scale. ; S43. The dense scale output by the scale learner network. and Multiplication yields dense depth .

2. The depth estimation method based on millimeter-wave radar-camera multi-stage fusion according to claim 1, characterized in that, Step S42 specifically involves: [The text abruptly ends here, likely due to an incomplete sentence or a missing section.] The empty space in the middle is filled with a unit scale of 1; and The scale residual r is obtained by connecting the input to the scale learner network; the scale map s is calculated using ReLU(1+r), and the scale is applied to the network. Multiplying yields the dense depth. .

Images