4D millimeter wave radar-assisted end-to-end image depth estimation model
By using an end-to-end image depth estimation model and optimizing it with a shared encoder and joint loss function, the problems of depth estimation error and computational redundancy in multi-stage models are solved, thereby improving the accuracy and real-time performance of image depth estimation.
Patent Information
- Application Number
- CN202511277041.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-08
- Publication Date
- 2025-12-19
AI Technical Summary
Existing image depth estimation models based on 4D millimeter-wave radar typically employ a multi-stage processing flow. Independent training of each stage limits the depth estimation performance and incurs high computational resource costs, making it difficult to meet real-time requirements.
An end-to-end image depth estimation model based on 4D millimeter-wave radar is adopted. Through a multimodal feature extraction module, a sparse depth map generation module, and a dense depth map generation module, a shared encoder is used for unified training to generate sparse and dense depth maps. The model is then optimized by combining confidence loss, depth loss, and edge-aware smoothing loss.
It improves the accuracy and processing speed of image depth estimation, solves the problems of depth estimation error and computational redundancy in multi-stage models, and meets real-time requirements.
Smart Images
Figure CN121169984A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of image depth estimation and multi-sensor fusion, and particularly relates to an end-to-end image depth estimation model based on 4D millimeter wave radar assistance. BACKGROUND
[0002] Monocular depth estimation, as one of the basic tasks in computer vision, is of great significance for realizing automatic driving and robot navigation. Existing monocular image-based depth estimation models can generate dense depth maps by learning rich semantic and contextual information, but the inherent scale uncertainty still poses challenges to the accuracy of monocular depth estimation. To improve the accuracy and reliability of depth estimation, fusing external sensor information has become an effective means. Laser radar is used to assist image depth estimation due to its relatively accurate depth information, but it is costly and very sensitive to bad weather conditions such as rain, snow and fog. In contrast, millimeter wave radar has great potential as an auxiliary sensor for image depth estimation due to its low cost, strong penetration ability and robustness in harsh environments. Traditional 3D millimeter wave radar has limited effect on measuring the elevation angle of the target, and its data usually only contains distance, azimuth angle and Doppler velocity information. With the development of multiple-input multiple-output antenna technology, 4D millimeter wave radar has emerged, realizing the joint measurement of target distance, azimuth angle, elevation angle and velocity. 4D millimeter wave radar will play an increasingly important role in the field of autonomous driving, and in-depth research on image depth estimation models assisted by 4D millimeter wave radar has important application value.
[0003] Currently, there are some image depth estimation models assisted by 4D millimeter wave radar, and these models usually adopt a two-stage or multi-stage processing flow. In these models, since the millimeter wave radar data usually contains a large amount of noise, the first stage is generally responsible for predicting the correlation between radar points and image pixels. When this correlation exceeds a preset threshold, the depth value of the radar point is assigned to the corresponding pixel, thereby generating a sparse depth map. In the subsequent stage, different models use different strategies to fuse the sparse depth map and image features to generate the final dense depth map. However, such multi-stage models usually use independent encoder models at each stage, and the models of the two stages are trained separately. Independent training of each stage limits the feedback of dense depth prediction in the second stage to the correlation prediction in the first stage, thereby restricting the overall improvement of depth estimation performance. In addition, providing an independent encoder for each stage significantly increases the computational resource overhead, making it difficult for the model to meet the real-time requirements. SUMMARY
[0004] The present application aims at the deficiencies in the prior art, and proposes an end-to-end image depth estimation model based on 4D millimeter wave radar assistance, which can realize an end-to-end unified training strategy and solve the depth estimation error and calculation redundancy problems caused by independent training of each stage in the traditional multi-stage model.
[0005] To achieve the above technical effects, the present application adopts the following technical solutions:
[0006] An end-to-end image depth estimation model based on 4D millimeter wave radar assistance comprises the following three sub-modules:
[0007] A multi-modal feature extraction module is used to extract multi-level features of a visible light image and features of millimeter wave radar points;
[0008] A sparse depth map generation module is connected to the output end of the multi-modal feature extraction module and is used to predict the confidence of each radar point according to the radar point features and multi-level image features and filter low-confidence noise points to generate a sparse depth map;
[0009] A dense depth map generation module is connected to the output ends of the multi-modal feature extraction module and the sparse depth map generation module and is used to fuse the multi-level image features and the sparse depth map to generate a final dense depth map.
[0010] Further, the multi-modal feature extraction module specifically comprises:
[0011] A visible light image encoder adopts a pre-trained DINOv2-Small model as a visible light image encoder and is used to output four levels of image features of the 12th, 9th, 6th and 3rd layers;
[0012] A millimeter wave radar encoder adopts a multi-layer perceptron composed of 6 fully connected layers as a millimeter wave radar encoder and is used to process radar point clouds containing image pixel coordinates and depth information and output radar point cloud features.
[0013] Further, the sparse depth map generation module specifically comprises:
[0014] An image feature up-sampler is used to up-sample the 12th and 9th layer image features to the original image size;
[0015] A coordinate sampling unit is used to extract the image features of an N×N neighborhood around each radar point from the up-sampled image features according to the coordinates of the radar points in the image;
[0016] A confidence network is used to fuse the radar point features and the neighborhood image features and output the confidence of each radar point;
[0017] A noise filtering unit is used to filter out radar points with low confidence according to a pre-set confidence threshold.
[0018] Sparse depth map construction unit: for generating a sparse depth map using the depth information of the remaining radar points.
[0019] Further, the dense depth map generation module is a depth decoder of a DPT-like structure, specifically comprising:
[0020] Multi-scale image feature processing unit: for receiving and processing the multi-level image features output by the multi-modal feature extraction module;
[0021] Sparse depth map processing unit: for receiving and processing the sparse depth map output by the sparse depth map generation module;
[0022] Image-depth map fusion module: its input end is connected to the output ends of the multi-scale image feature processing unit and the sparse depth map processing unit, for fusing the multi-scale image features and the sparse depth map features;
[0023] Depth map generation unit: for further processing the fused features to obtain a normalized dense depth map;
[0024] Inverse normalization unit: for inversely normalizing the normalized dense depth map, i.e., multiplying it by the difference between the maximum depth and the minimum depth and adding the minimum depth, to output a final dense depth map.
[0025] Further, the loss function used by the 4D millimeter wave radar assisted end-to-end image depth estimation model in the training process is a joint loss function containing confidence loss, depth loss and edge perception smoothing loss, and the model is optimized through end-to-end training. The specific roles and formulas of the three losses are as follows:
[0026] 1) Confidence loss:
[0027] The confidence loss is used to constrain the difference between the predicted value and the true value of the radar point confidence. When there is a pixel point in the N×N image neighborhood centered on the radar point, the depth difference between the pixel point and the radar point is less than the depth threshold, the true confidence of the radar point is set to 1, otherwise it is set to 0; it adopts a cross-entropy loss function for calculation, the formula is:
[0028]
[0029] Where, Ω R represents a set containing all radar points, C(p) is the true confidence of radar point p, is the predicted confidence of radar point p;
[0030] 2) Depth loss:
[0031] The depth loss is used to constrain the difference between the predicted depth and the ground truth depth, and is calculated by using a smooth L1 loss function, and the formula is:
[0032]
[0033] wherein, d(x) is the ground truth depth of pixel x, is the predicted depth of pixel x, and d Ω represents the image pixel region with valid depth values;
[0034] 3) Edge-aware smoothing loss:
[0035] The edge-aware smoothing loss is used to encourage the depth prediction to remain smooth while allowing larger changes at image edges, and the formula is:
[0036]
[0037] wherein, and are the horizontal and vertical gradients of the image respectively, Ω represents the final dense depth map, and I represents the original image.
[0038] The beneficial effects of the present application are: the present application proposes an end-to-end image depth estimation model based on 4D millimeter wave radar assistance. The model uses a unified architecture, uses a shared image encoder and radar encoder to jointly complete sparse depth estimation and dense depth estimation. The entire model can use an end-to-end training strategy to optimize the synergistic effect between tasks, effectively improving the accuracy and processing speed of image depth estimation. BRIEF DESCRIPTION OF DRAWINGS
[0039] Figure 1 The figure is a model overall architecture diagram of the present application.
[0040] Figure 2 The figure is an image up-sampler structure diagram of the present application.
[0041] Figure 3 The figure is a confidence network structure diagram of the present application. DETAILED DESCRIPTION
[0042] The present application will now be further described in detail in conjunction with the accompanying drawings.
[0043] Figure 1The overall architecture of an end-to-end image depth estimation model based on 4D millimeter wave radar assistance is shown, which includes three sub-modules: a multi-modal feature extraction module for extracting multi-level features of a visible light image and features of millimeter wave radar points; a sparse depth map generation module connected to the output end of the multi-modal feature extraction module, for predicting the confidence of each radar point according to the radar point features and multi-level image features, filtering low-confidence noise points, and generating a sparse depth map; and a dense depth map generation module with input ends connected to the output ends of the multi-modal feature extraction module and the sparse depth map generation module, for fusing multi-level image features and a sparse depth map to generate a final dense depth map.
[0044] The multi-modal feature extraction module specifically includes:
[0045] 1) Visible light image encoder: a pre-trained DINOv2-Small model is used as the visible light image encoder to output four levels of image features of the 12th, 9th, 6th and 3rd layers H0 and W0 represent the original height and width of the image respectively, C I is the dimension of the image features;
[0046] 2) Millimeter wave radar encoder: a multi-layer perceptron composed of 6 fully connected layers is used as the millimeter wave radar encoder to process radar point clouds containing image pixel coordinates and depth information, and output radar point cloud features K is the number of radar points, C R is the feature dimension of the radar points.
[0047] The sparse depth map generation module specifically includes:
[0048] 1) Image feature up-sampler: for up-sampling the 12th and 9th layer image features to the original image size, Figure 2 The specific structure is shown, and the processing flow is as follows:
[0049] The 9th layer image features are sequentially subjected to convolution, transposed convolution, convolution operations, and input to a residual convolution unit for processing, which sequentially connects a rectified linear unit (ReLU), a convolution layer, a ReLU, and a convolution layer inside, and performs an element-wise addition operation on the features after the serial processing and the features before inputting into the unit;
[0050] The 12th layer image features are also sequentially subjected to convolution, transposed convolution, convolution and residual convolution unit processing, and then the spatial size is adjusted to be consistent with the processed by bilinear interpolation, and the size-adjusted performing convolution operation again;
[0051] The processed radar point feature and the processed image feature are added element by element to obtain a fusion feature, and the fusion feature is sent to the residual convolution unit for secondary processing.
[0052] The secondary processed feature is up-sampled by using a bilinear interpolation method to obtain a final image feature wherein C R represents the number of radar point feature channels.
[0053] 2) The coordinate sampling unit is used to extract an image feature block in an N*N range centered on the coordinates (u, v) of each millimeter wave radar point in the image pixel coordinate system. wherein the horizontal coordinate range covered by the feature block is and the vertical coordinate range is
[0054] 3) The confidence network is used to fuse the radar point feature and the neighborhood image feature to obtain the confidence of the radar point, Figure 3 The specific structure is shown, and the processing flow is as follows:
[0055] The radar point feature and the neighborhood image feature are input into a 2-layer bidirectional cross attention module to obtain updated features and
[0056] f′ I is summed in the spatial dimension, and f′ R is spliced in the channel dimension.
[0057] The spliced feature is input into a multi-layer perceptron composed of 3 fully connected layers, and the confidence c of the corresponding radar point is output after being processed by a Sigmoid function.
[0058] 4) The noise filtering unit sets a confidence threshold τ c = 0.5, and the radar points with a confidence lower than τ c are regarded as noise and filtered out.
[0059] 5) The sparse depth map construction unit is used to generate a sparse depth map by using the depth information of the remaining radar points.
[0060] The dense depth map generation module is a DPT structure depth decoder, and specifically includes:
[0061] 1) Multi-scale image feature processing unit: for receiving and processing the multi-level image features output by the multi-modal feature extraction module, gradually adjusting the channel number and spatial resolution of each level feature through convolution and transposed convolution operation, generating multi-scale image features;
[0062] 2) Sparse depth map processing unit: for receiving and processing the sparse depth map output by the sparse depth map generation module, which is sequentially subjected to bilinear interpolation, convolution layer and ReLU for feature enhancement;
[0063] 3) Image-depth map fusion module: its input end is connected to the output ends of the multi-scale image feature processing unit and the sparse depth map processing unit, for fusing multi-scale image features and sparse depth map features by feature addition method;
[0064] 4) Depth map generation unit: for further processing the fused features, sequentially subjected to convolution, bilinear interpolation up-sampling, ReLU activation function, convolution, etc. operations, and finally activated by Sigmoid function to generate a dense depth map normalized to the interval [0, 1];
[0065] 5) Denormalization unit: for denormalizing the normalized dense depth map, i.e. multiplying it by the difference between the maximum depth and the minimum depth and adding the minimum depth, outputting the final dense depth map.
[0066] Further, the loss function used by the 4D millimeter wave radar assisted end-to-end image depth estimation model in the training process is a joint loss function containing confidence loss, depth loss and edge perception smoothing loss, and the model is optimized through end-to-end training. The specific functions and formulas of the three losses are as follows:
[0067] 1) Confidence loss:
[0068] The confidence loss is used to constrain the difference between the predicted value and the true value of the radar point confidence. When there is a pixel point in the N x N image neighborhood centered on the radar point whose depth difference with the radar point depth is less than the depth threshold, the true confidence is set to 1, otherwise it is set to 0; it adopts cross-entropy loss function for calculation, the formula is:
[0069]
[0070] Where, Ω R represents a set containing all radar points, C(p) is the true confidence of radar point p, is the predicted confidence of radar point p;
[0071] 2) Depth loss:
[0072] The depth loss is used to constrain the difference between the predicted depth and the ground truth depth, and is calculated using a smooth L1 loss function, with the formula being:
[0073]
[0074] wherein, d(x) is the ground truth depth of pixel x, is the predicted depth of pixel x, Ω d represents the region of image pixels having valid depth values;
[0075] 3) Edge-aware smoothing loss:
[0076] The edge-aware smoothing loss is used to encourage the depth prediction to remain smooth while allowing large changes at image edges, with the formula being:
[0077]
[0078] wherein, and are the horizontal and vertical gradients of the image, respectively, represents the final dense depth map, and I represents the original image.
[0079] The above is only a preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments only, and any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall be deemed to fall within the protection scope of the present application.
Claims
1. A 4D millimeter wave radar assisted based end-to-end image depth estimation model, characterized in that, The method comprises the following sub-modules: A multi-modal feature extraction module is used to extract multi-level features of a visible light image and features of millimeter wave radar points; A sparse depth map generation module is connected to the output end of the multi-modal feature extraction module, and is used to predict the confidence of each radar point according to the radar point features and the multi-level image features, filter low-confidence noise points, and generate a sparse depth map; A dense depth map generation module is connected to the output ends of the multi-modal feature extraction module and the sparse depth map generation module, and is used to fuse the multi-level image features and the sparse depth map to generate a final dense depth map.
2. The 4D millimeter wave radar-assisted end-to-end image depth estimation model of claim 1, wherein, The multi-modal feature extraction module comprises: A visible light image encoder adopts a pre-trained DINOv2-Small model as a visible light image encoder, and is used to output four levels of image features of the 12th, 9th, 6th and 3rd layers; A millimeter wave radar encoder adopts a multi-layer perception as a millimeter wave radar encoder, and is used to process a radar point cloud containing image pixel coordinates and depth information, and output radar point cloud features.
3. The 4D millimeter wave radar-assisted end-to-end image depth estimation model of claim 1, wherein, The sparse depth map generation module comprises: An image feature up-sampler is used to up-sample the 12th and 9th layer image features to the original image size; A coordinate sampling unit is used to extract the image features of an N×N neighborhood around each radar point from the up-sampled image features according to the coordinates of the radar points in the image; A confidence network is used to fuse the radar point features and the neighborhood image features, and output the confidence of each radar point; A noise filtering unit is used to filter out radar points with low confidence according to a pre-set confidence threshold; A sparse depth map construction unit is used to generate a sparse depth map using the depth information of the remaining radar points.
4. The 4D millimeter wave radar-assisted end-to-end image depth estimation model of claim 1, wherein, The dense depth map generation module is a depth decoder of a DPT-like structure, which comprises: A multi-scale image feature processing unit is used to receive and process the multi-level image features output by the multi-modal feature extraction module; A sparse depth map processing unit is used to receive and process the sparse depth map output by the sparse depth map generation module; An image-depth map fusion module is connected to the output ends of the multi-scale image feature processing unit and the sparse depth map processing unit, and is used to fuse the multi-scale image features and the sparse depth map features; A depth map generation unit is used to further process the fused features to obtain a normalized dense depth map; An inverse normalization unit is used to inverse normalize the normalized dense depth map, that is, multiply it by the difference between the maximum depth and the minimum depth and add the minimum depth, and output a final dense depth map.
5. The 4D millimeter wave radar-assisted end-to-end image depth estimation model of claim 1, wherein, The multi-modal feature extraction module, the sparse depth map generation module and the dense depth map generation module can be jointly optimized by using an end-to-end unified training strategy, and the loss function used in the training process is a joint loss function comprising a confidence loss, a depth loss and an edge perception smoothing loss.
6. The joint loss function of claim 5, wherein: The confidence loss is used for constraining the difference between the predicted value and the true value of the radar point confidence, when there is a pixel point in the image neighborhood with a size of N×N centered on the radar point, and the depth difference between the pixel point and the radar point is less than a depth threshold, the true confidence of the radar point is set to 1, otherwise, the true confidence of the radar point is set to 0; the confidence loss is calculated by using a cross-entropy loss function.
7. The joint loss function of claim 5, wherein: The depth loss is used for constraining the difference between the predicted depth and the true depth, and is calculated by using a smooth L1 loss function.
8. The joint loss function of claim 5, wherein: The edge-aware smooth loss is used for encouraging the depth prediction to keep smooth while allowing larger changes at image edges, and the formula is: where and are the horizontal and vertical gradients of the image, respectively, denotes the final dense depth map and I denotes the original image.