Visible light image guided millimeter wave radar depth enhancement method based on structure perception
By introducing structure perception technology into millimeter-wave radar depth data, the visible light image is used to extract structural prior information and generate relevant areas, the sparsity and noise problems of radar depth data are solved, and high-quality depth enhancement effect is achieved.
Patent Information
- Application Number
- CN202411758808.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-02-28
AI Technical Summary
Due to the limitations of the imaging mechanism and the multi-path propagation effect, millimeter-wave radar depth data has problems such as sparseness, detection noise and poor semantic information, and the existing technology is difficult to effectively solve these problems.
Using a visible light image guidance method based on structure perception, structural prior information is extracted through the zero-sample monocular depth estimation model, radar depth point-related areas are generated, and depth diffusion and noise filtering are performed in these areas. Radar point-pixel correlation is learned through deep neural networks to enhance millimeter wave radar depth.
Dense and noise filtering of millimeter-wave radar depths in reasonable space areas significantly improve the quality and accuracy of radar depths, reduce the introduction of additional noise, and reduce the difficulty of neural network learning.
Smart Images
Figure CN119535447A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a visible light image-guided millimeter-wave radar depth enhancement method based on structure perception. Background Art
[0002] With the development of autonomous driving technology, the perception of depth has become increasingly important for understanding the three-dimensional environment. Common depth sensors on the market, such as lidar and radar, can obtain measured depth directly from the environment. Among them, lidar can generate higher accuracy and denser depth information than millimeter-wave radar, but it is more expensive, larger in size, and more sensitive to environmental factors such as rain, fog, and snow. In contrast, millimeter-wave radar is easy to deploy and has strong robustness to adverse light and weather conditions, and has received increasing attention in recent years. However, due to the limitations of the millimeter-wave radar imaging mechanism and the influence of multipath propagation effects, the depth data obtained by millimeter-wave radar faces inherent challenges such as sparsity, detection noise, and poor semantic information.
[0003] To overcome the above problems, many studies have focused on effectively enhancing low-quality millimeter-wave radar depth data. A common radar data enhancement strategy is to merge the radar point clouds of adjacent frames into the current frame to densify the depth information of the current frame. However, the multi-frame fusion operation will cause a large time delay, and future frames are often not available in real scenes. Some other methods diffuse the depth of radar points along the height dimension or within the detection box, and some methods learn the correlation between radar and camera pixels in a predefined area to model the uncertainty of radar points, and then filter out potential noisy radar points through the modeled results. However, all of these methods restrict the relevant area of the radar point to a predefined regular shape without taking into account the prior information in the scene. This may introduce additional error measurements and interfere with the learning of the correlation between radar and camera pixels.
[0004] With the widespread use of deep learning technology in monocular depth estimation tasks, many powerful monocular depth estimation models have been proposed. They have achieved high accuracy and rich structural details on test samples in the same domain as the training samples. However, these methods cannot be effectively generalized to test environments outside the training sample domain, which makes their excellent results unable to be directly transferred to new scenes. Recently, some methods have explored the task of zero-shot monocular depth estimation. They jointly train monocular depth estimation models on multiple sub-datasets. These sub-datasets contain different scenes and depth distributions. In order to eliminate the influence of depth scale and offset differences between different sub-datasets, these methods use affine invariant loss. Through the above training process, these models have obtained strong cross-scene generalization capabilities, can effectively extract structural prior information in scenes not seen during training, and produce monocular depth maps with rich structural details. However, these works focus on estimating relative depth. Although the monocular depth maps they output have good structural details, the scale of the depth value has a large deviation from the actual value. Summary of the invention
[0005] In view of the deficiencies in the above-mentioned prior art, the present invention provides a visible light image-guided millimeter-wave radar depth enhancement method based on structure perception, which achieves advanced depth enhancement effect. Given a pair of visible light images and a sparse millimeter-wave radar depth map, the method first fully extracts the structural prior information in the visible light image through a zero-sample monocular depth estimation model, then uses the depth distribution of the monocular depth estimation result to guide the generation of radar depth point-related areas, and then diffuses the radar depth points in these radar depth point-related areas, and finally inputs the diffused radar depth and visible light image into a deep neural network to learn the radar point-pixel association (Radar-Camera Pixel Association) for noise filtering to obtain enhanced millimeter-wave radar depth. The present invention guides the generation of radar depth point-related areas through the structural information extraction capability of the zero-sample monocular depth estimation model, and learns the radar point-pixel association through a deep neural network, which can densify the millimeter-wave radar depth in a reasonable spatial area and finely filter the noise, thereby obtaining a high-quality depth enhancement result.
[0006] The present invention is implemented by the following technical solutions:
[0007] A visible light image guided millimeter wave radar depth enhancement method based on structure perception, firstly inputting a visible light image into a zero-sample monocular depth estimation model to generate a monocular depth map, then using the monocular depth map to guide the generation of a radar depth point related area, then diffusing the radar depth points in the radar depth point related area to obtain the diffused radar depth, and finally using the diffused radar depth and the visible light image to estimate the radar point-pixel association to perform noise filtering, and obtain the enhanced millimeter wave radar depth. The visible light image guided millimeter wave radar depth enhancement method based on structure perception has the following specific steps:
[0008] S1: Using a zero-shot monocular depth estimation model from an RGB image I∈R 3×H×W Extract structural prior information from the image and generate a monocular depth map d with rich structural details. m ∈R H×W ;
[0009] S2: with d m (its depth distribution information) is used as a guide to generate radar depth point related areas;
[0010] S21: For the millimeter wave radar depth map d r ∈R H×W Each radar depth point in d is centered at the radar depth point. m Under the guidance of , the sub-correlation area of the radar depth point is generated;
[0011] S22: Integration r The sub-correlation area of all radar depth points in d r Full radar depth point correlation area;
[0012] S3: Enhance the sparse and noisy millimeter-wave radar depth in the area related to the radar depth point;
[0013] S31: For d r For each radar depth point in , its depth is diffused in the sub-correlation area of the corresponding radar depth point to obtain a relatively dense radar depth map d dr ∈R H×W ;
[0014] S32: Based on I and d dr , use deep neural network to estimate radar point-pixel association and obtain an association confidence map c∈R H×W ;
[0015] S33: Using the radar point-pixel association confidence in c as a reference, filter out d dr Potential noise, get enhanced mmWave radar depth map d er ∈R H×W .
[0016] In the above technical solution, further, the zero-sample monocular depth estimation model used in step S1 uses affine invariant loss to perform joint training on massive data containing multiple types of scenes and depth ranges. Affine invariant loss eliminates the influence of differences in depth scales and offsets between different sub-datasets, so that model training can fully utilize the information of massive data. This training method gives the model the ability to fully extract the structural priors of unseen scenes, and can obtain monocular depth with rich structural details, clear depth boundaries and reasonable relative geometric relationships from RGB images.
[0017] Furthermore, the specific steps of step S21 are as follows:
[0018] 1) For the radar depth map d r A radar depth point d r (i,j), set it in the monocular depth map d m The monocular depth point d at the corresponding position in m (i,j) is the seed point;
[0019] 2) With d m (i, j) as the center, the radar depth points are diffused pixel by pixel in four directions through the seed point growth strategy to obtain the sub-correlation area of the radar depth points. In the expansion process, d m The depth distribution in is used as the connection relationship to control the expansion; specifically, the following steps are included:
[0020] 21) Calculate d m The neighboring pixels of (i,j) and d m The depth difference of (i, j) is included in d if the depth difference is less than the preset depth difference threshold. r The sub-correlation region of (i,j);
[0021] 22) For inclusion of d r Calculate the depth difference between these pixels and their adjacent pixels in the sub-correlation area (i, j). If the depth difference is less than the preset depth difference threshold, include it in the sub-correlation area.
[0022] As shown in formula (1):
[0023]
[0024] in, represents the d obtained after expansion r The sub-correlation region of (i,j); τ 1 is the preset depth difference threshold; (u,v) is d m The set of pixels in the image, whose depth values obey d mDepth distribution in
[0025] 23) Repeat the method of step 22) for the pixels newly included in the sub-correlation area until there is no pixel with a depth difference less than a preset depth difference threshold among the adjacent pixels in the sub-correlation area.
[0026] Furthermore, the specific operation in step S22 is: for d r Each depth radar point d in r Sub-correlation area of radar depth point corresponding to (i,j) By taking the union of all sub-correlation areas, the radar depth point correlation area can be obtained As shown in formula (2):
[0027]
[0028] in Refers to d r The set of corresponding positions of all radar depth points in .
[0029] Furthermore, the specific steps of step S31 are: After that, d r The depth value of (i,j) is in the relevant area of its corresponding radar depth point Diffusion is performed inside to obtain the depth obtained by diffusion of the depth at position (i, j). The process is as shown in formula (3):
[0030]
[0031] in, Indicates d r The dense depth of (i,j);
[0032] For d r The above operation is performed on all radar depth points in (i, j) to obtain a relatively dense depth map d diffused by all radar depth points. dr .
[0033] Furthermore, in step S32, the details of the deep neural network are as follows:
[0034] The deep neural network is a UNet structure, including two encoders, a decoder and a fusion module based on self-attention mechanism and cross-attention mechanism; the inputs of the two encoders are respectively an RGB image obtained by a monocular camera and a sparse radar depth map obtained by millimeter-wave radar point cloud projection, and the outputs are respectively RGB image features and radar depth features; the fusion module based on self-attention mechanism and cross-attention mechanism is used to fuse RGB image features and radar depth features, and output fusion features; the input of the decoder is fusion features, and the output is a confidence map, the value of the confidence map represents the radar point-pixel association confidence (representing the probability that the position may be projected by the radar point).
[0035] Furthermore, the fusion module based on the self-control mechanism and the cross attention mechanism is used to fuse the RGB image features and the radar depth features, and output the fusion features. The specific method is as follows: for the RGB image feature f i and radar depth feature f d , first perform self-attention mechanism on each, then perform cross-attention mechanism between the two, and finally obtain the fusion feature f f The specific process is shown in formula (4):
[0036]
[0037] in, Q i , K i 、V i are the query, key, and value obtained by the linear layer of the RGB image features; Q d , K d and V d are the query, key and value obtained by the linear layer of radar depth features; SA represents the self-attention mechanism; d k Represents the feature dimension.
[0038] Furthermore, the training method of the deep neural network is as follows:
[0039] 1) To obtain the true value of the confidence map for supervised deep neural network training, the lidar depth is used as the true depth value. In the relevant area of the radar depth point The difference between the relatively dense radar depth and lidar depth is calculated in The difference is less than the predetermined threshold τ 2 The pixel at the position is set to 1, and the others are set to 0, and the true value of the confidence map c(u,v) is obtained. The process can be expressed as formula (5):
[0040]
[0041] Among them, d l(u,v) is the lidar depth; d dr (u,v) is a relatively dense radar depth map;
[0042] 2) Use the obtained confidence map truth value to supervise the training of the deep neural network, using the fork entropy loss measure The difference between the output of the deep neural network and the true value within the range is calculated, and the learnable parameters of the deep neural network are updated using stochastic gradient descent. The specific form of the loss function is shown in formula (6):
[0043]
[0044] in, Confidence map for deep neural network predictions.
[0045] Furthermore, the specific operation of step S32 is to input the RGB image into the deep neural network trained in S32 to estimate a confidence map.
[0046] Further, the step S33 is specifically as follows: using the confidence map obtained in step S32 as a reference, dr The depths of the positions with confidence values less than the threshold (the threshold can be set as needed) are deleted. The final enhanced radar depth map d er , d er Not only is it densified, but the noise is also filtered out.
[0047] The beneficial effects of the present invention are:
[0048] In response to the sparsity and noise problems of millimeter-wave radar depth data, the present invention introduces a zero-sample monocular depth estimation model to fully extract the structural prior information in the scene, and then generates a radar-related area for each radar depth point guided by the structural prior information. Then, the depth is densified in these more targeted radar depth point-related areas, and the uncertainty of the radar points is modeled by learning the degree of correlation between radar points and pixels (i.e., radar point-pixel correlation), thereby filtering out noise. This radar depth enhancement method allows the entire enhancement process to be carried out in a more targeted area (i.e., radar depth point-related area), reducing the additional noise that may be introduced during the densification process, while reducing the difficulty of the neural network learning the correlation between radar points and pixels, thereby effectively improving the quality of the enhanced radar depth. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 The figure is an overall operation flow chart of an embodiment of the present invention.
[0050] Figure 2 This is the training process of the deep neural network according to an embodiment of the present invention.
[0051] Figure 3 The present invention is a flowchart of a fusion module for performing feature fusion based on a self-attention mechanism and a cross-attention mechanism in an embodiment of the present invention.
[0052] Figure 4 The figure is a flow chart of generating radar depth point related areas according to an embodiment of the present invention.
[0053] Figure 5 The following are example results of the embodiments of the present invention. All example results are projected onto an RGB image to visualize the effect: a) input RGB image; (b) input sparse noisy millimeter-wave radar depth map; (c) monocular depth map obtained by the zero-sample monocular depth estimation model; (d) radar depth point correlation area; (e) radar point-pixel correlation confidence map (the brighter the color, the higher the confidence); (f) radar depth map after diffusion; (g) diffuse radar depth map after filtering out noise. All the above depth maps are visualized using the Viridis color map. DETAILED DESCRIPTION
[0054] The technical solution of the present invention is further described below in conjunction with specific embodiments and drawings.
[0055] Example
[0056] The present invention provides a visible light image-guided millimeter-wave radar depth enhancement method based on structure perception, which fully extracts structural prior information in the image to guide the generation of radar depth point related areas, and then enhances the radar depth in these targeted spatial areas.
[0057] The overall process of the structure-aware visible light image-guided millimeter-wave radar depth enhancement method is as follows: Figure 1 As shown, the specific steps include:
[0058] 1) Extract the RGB image I( Figure 5 (a)) generates a monocular depth map d with rich structural details. m ( Figure 5 (c)); The zero-shot monocular depth estimation model uses an affine invariant loss and is jointly trained on a large amount of data containing multiple types of scenes and depth ranges. The affine invariant loss eliminates the impact of differences in depth scales and offsets between different sub-datasets, allowing model training to fully utilize the information of massive data. This training method gives the model the ability to fully extract the structural priors of unseen scenes, and can obtain monocular depth with rich structural details, clear depth boundaries, and reasonable relative geometric relationships from RGB images.
[0059] 2) With d m To guide, generate radar depth point related areas. The overall process is as follows Figure 4 As shown; First, for the millimeter wave radar depth map d r ( Figure 5 (b)) Each radar depth point d r (i,j), set it in the monocular depth map d m The monocular depth point d at the corresponding position in m (i,j) is the seed point; d m (i, j) as the center, the radar depth point is diffused pixel by pixel in four directions through the seed point growth strategy to obtain the sub-correlation area of the radar depth point. m The depth distribution in is used as the condition to control the expansion. Specifically, d m The neighboring pixels of (i,j) and d m The depth difference of (i, j) is included in d if the depth difference is less than the preset depth difference threshold. r (i,j) sub-correlation area; for inclusion in d r For the pixels in the sub-correlation area of (i, j), calculate the depth difference between these pixels and their adjacent pixels. If the depth difference is less than the preset depth difference threshold, include them in the sub-correlation area. Repeat the above steps for the pixels newly included in the sub-correlation area until there are no pixels in the adjacent pixels of the sub-correlation area whose depth difference is less than the preset depth difference threshold. r Each depth radar point d in r Sub-correlation area of radar depth point corresponding to (i,j) By taking the union of all sub-correlation areas, the radar depth point correlation area can be obtained like Figure 5 (d) as shown.
[0060] 3) In the area related to the radar depth point, the millimeter wave radar depth is enhanced. r (i,j), whose depth value is in the sub-correlation area of the corresponding radar depth point Diffusion is performed inside to obtain the depth obtained by diffusion of the depth at position (i, j). r The above operation is performed on all radar depth points in the image to obtain a relatively dense radar depth map d obtained by diffusion of all radar depth points. dr ,like Figure 5 (f) As shown in Figure 2. dr Input the deep neural network and get an associated confidence map c∈R H×W ,like Figure 5 (e) As shown in Figure 5, the confidence level of radar point-pixel association in (c) is used as a reference to filter out the pixels in (d). dr Potential noise will be drThe depths of the positions with confidence values less than the threshold are deleted to obtain the enhanced millimeter-wave radar depth map d er ∈R H×W ,like Figure 5 (g) as shown.
[0061] The deep neural network is designed based on the UNet structure, which includes two encoders, a decoder, and a fusion module based on the self-attention mechanism and the cross-attention mechanism, which is used to learn the association between radar points and image pixels (i.e., radar point-pixel association), such as Figure 1 As shown in the figure. The inputs of the two encoders are the RGB image obtained by the monocular camera and the sparse radar depth map obtained by the projection of the millimeter-wave radar point cloud, and the outputs are RGB image features and radar depth features. The fusion module based on the self-control mechanism and the cross-attention mechanism is used to fuse the RGB image features and the radar depth features, and output the fused features. The input of the decoder is the fused features, and the output is a confidence map. The value of the confidence map represents the confidence of the radar point-pixel association.
[0062] The fusion module based on self-control mechanism and cross attention mechanism is used to fuse RGB image features and radar depth features. The process is as follows: Figure 3 As shown, for the RGB image feature f i and radar depth feature f d , first perform self-attention mechanism on each, then perform cross-attention mechanism between the two, and finally obtain the fusion feature f f .
[0063] The training process of deep neural network is as follows Figure 2 Specifically, in order to obtain the true value of the confidence map for supervising deep neural network training, the lidar depth is introduced, and the depth obtained by the lidar is considered to be accurate (the lidar depth is taken as the true depth value). In the relevant area of the radar depth point Calculate the difference between the millimeter wave radar depth and the lidar depth, and The pixels at the positions where the difference is less than the predetermined threshold are set to 1, otherwise they are set to 0, and the true value of the confidence map is obtained. The obtained true value of the confidence map is used to supervise the training of the radar point-pixel association deep neural network, and the cross entropy loss is used to measure The difference between the deep neural network output and the true value within a certain range is calculated, and the learnable parameters of the deep neural network are updated using stochastic gradient descent.
[0064] The method of the present invention uses the structural information extraction capability of the zero-sample monocular depth estimation model to guide the generation of radar depth point related areas, and learns radar point-pixel associations through a deep neural network, which can densify the millimeter-wave radar depth and finely filter noise in a reasonable spatial area, thereby obtaining high-quality millimeter-wave radar depth enhancement results. Figure 5 As shown, Figure 5 (a) is the RGB image I input to the algorithm, Figure 5 (b) is the sparse and noisy millimeter wave radar depth map d input to the algorithm r . Figure 5 (c) is the monocular depth map d generated by extracting structural information from I using the zero-shot monocular depth estimation model. m , you can see d m It has rich structural details and clear depth boundaries, which reflects the powerful structural information extraction capability of the zero-sample monocular depth estimation model. Figure 5 (d) shows the radar depth point correlation area generated by the method of the present invention You can see Mainly focused on the target, and there is a clear boundary between the target and the background, which reflects The rationality of the radar depth point is to be able to reasonably reflect the area that may be affected by the radar depth point. Figure 5 (e) is the association confidence map learned by the deep neural network. The brighter the color, the higher the confidence. It can be seen that the positions with high confidence are concentrated on the target metal surface close to the sensor, which is consistent with the characteristics of millimeter wave radar. Figure 5 (f) shows the effect of d r The relatively dense radar depth map d obtained after diffusion dr , it can be clearly seen that its depth is Figure 5 d in (b) r It is much denser, and the depth does not spread randomly, but is concentrated on the target surface. Figure 5 (g) shows the effect of dr Enhanced millimeter-wave radar depth map after noise filtering er , which retains d dr The reasonable depth value in d dr The noise can be seen er Compare Figure 5 d in (b) r To be dense, but not Figure 5 d in (f) dr Dense because the noise is filtered out.
Claims
1. A visible light image-guided millimeter-wave radar depth enhancement method based on structure perception, characterized in that: Firstly, the visible light image is input into the zero-sample monocular depth estimation model to generate a monocular depth map. Then, the monocular depth map is used to guide the generation of radar depth point correlation area. Then, the radar depth points are diffused in the radar depth point correlation area to obtain the diffused radar depth. Finally, the diffused radar depth and the visible light image are used to estimate the radar point-pixel association for noise filtering to obtain the enhanced millimeter-wave radar depth.
2. The visible light image-guided millimeter wave radar depth enhancement method based on structure perception according to claim 1 is characterized in that: The specific steps are as follows: S1: Using a zero-shot monocular depth estimation model from an RGB image I∈R 3×H×W Extract structural prior information from the image and generate a monocular depth map d m ∈R H×W ; S2: with d m For guidance, radar depth point correlation areas are generated; S21: For the millimeter wave radar depth map d r ∈R H×W Each radar depth point in d is centered at the radar depth point. m Under the guidance of , the sub-correlation area of the radar depth point is generated; S22: Integration r The sub-correlation area of all radar depth points in d r S3: within the radar depth point related area, the millimeter wave radar depth is enhanced; S31: For d r For each radar depth point in , its depth is diffused in the sub-correlation area of the corresponding radar depth point to obtain a relatively dense radar depth map d dr ∈R H×W ; S32: Based on I and d dr , use deep neural network to estimate radar point-pixel association and obtain an association confidence map c∈R H×W ; S33: Using the radar point-pixel association confidence in c as a reference, filter out d dr Potential noise, get enhanced mmWave radar depth map d er ∈R H×W .
3. The method for depth enhancement of millimeter-wave radar guided by visible light image based on structure perception according to claim 2 is characterized in that: The specific steps of step S21 are as follows: 1) For the radar depth map d r A radar depth point d r (i,j), set it in the monocular depth map d m The monocular depth point d at the corresponding position in m (i,j) is the seed point; 2) With d m (i, j) is the center, and the radar depth points are diffused pixel by pixel in four directions through the seed point growth strategy to obtain the sub-correlation area of the radar depth points, which specifically includes the following steps: 21) Calculate d m The neighboring pixels of (i,j) and d m The depth difference of (i, j) is included in d if the depth difference is less than the preset depth difference threshold. r The sub-correlation region of (i,j); 22) For inclusion of d r The pixels in the sub-correlation area of (i, j) are calculated, and the depth difference between these pixels and their adjacent pixels is calculated. If the depth difference is less than the preset depth difference threshold, it is included in the sub-correlation area; specifically, as shown in formula (1): in, represents the d obtained after diffusion r (i, j) is the sub-correlation area; τ1 is the preset depth difference threshold; (u, v) is d m The set of pixels in the image, whose depth values obey d m Depth distribution of 23) Repeat the method of step 22) for the pixels newly included in the sub-correlation area until there is no pixel with a depth difference less than a preset depth difference threshold among the adjacent pixels in the sub-correlation area.
4. The visible light image-guided millimeter wave radar depth enhancement method based on structure perception according to claim 2 is characterized in that: The specific operation in step S22 is: r Each depth radar point d in r Sub-correlation area of radar depth point corresponding to (i,j) Take the union of all sub-correlated areas to get the radar depth point related area The specific formula is as shown in formula (2): in Refers to d r The set of corresponding positions of all radar depth points in .
5. The visible light image-guided millimeter wave radar depth enhancement method based on structure perception according to claim 2 is characterized in that: The specific steps of step S31 are: r The depth value of (i,j) is in the relevant area of its corresponding radar depth point Diffusion is performed inside to obtain the depth obtained by diffusion of the depth at position (i, j). The process is shown in formula (3): in, Indicates d r The dense depth of (i,j); For d r The above operation is performed on all radar depth points in the image to obtain a relatively dense radar depth map d obtained by diffusion of all radar depth points. dr .
6. The visible light image-guided millimeter wave radar depth enhancement method based on structure perception according to claim 2 is characterized in that: In step S32, the deep neural network is specifically: The deep neural network includes two encoders, a decoder and a fusion module based on self-attention mechanism and cross-attention mechanism; the inputs of the two encoders are RGB images obtained by a monocular camera and sparse radar depth maps obtained by millimeter-wave radar point cloud projection, and the outputs are RGB image features and radar depth features, respectively; the fusion module based on self-attention mechanism and cross-attention mechanism is used to fuse RGB image features and radar depth features, and output fusion features; the input of the decoder is fusion features, and the output is a confidence map, and the value of the confidence map represents the radar point-pixel association confidence.
7. The visible light image-guided millimeter wave radar depth enhancement method based on structure perception according to claim 6 is characterized in that: The training method of the deep neural network is as follows: 1) To obtain the true value of the confidence map for supervising deep neural network training, the lidar depth is used as the true depth value; in the relevant area of the radar depth point The difference between the relatively dense radar depth and lidar depth is calculated in The pixels at the positions where the difference is less than the predetermined threshold τ2 are set to 1, and the others are set to 0, and the true value of the confidence map c(u,v) is obtained; the process is expressed as formula (5): Among them, d l (u,v) is the lidar depth; d dr (u,v) is a relatively dense radar depth map; 2) Use the confidence map obtained in 1) to supervise the training of the deep neural network, using the cross entropy loss measure The difference between the output of the deep neural network and the true value within the range is calculated, and the learnable parameters of the deep neural network are updated using stochastic gradient descent. The loss function is specifically shown in formula (6): in, Confidence map for deep neural network predictions.
8. The visible light image-guided millimeter wave radar depth enhancement method based on structure perception according to claim 6 is characterized in that: The fusion module based on the self-control mechanism and the cross-attention mechanism is used to fuse the RGB image features and the radar depth features and output the fused features. The specific method is as follows: For the RGB image feature f i and radar depth feature f d , first perform self-attention mechanism on each, then perform cross-attention mechanism between the two, and finally obtain the fusion feature f f , the specific process is shown in formula (4): in, Q i , K i 、V i are the query, key, and value obtained by the linear layer of the RGB image features; Q d , K d and V d They are respectively the query, key and value obtained by the linear layer of radar depth features; SA represents the self-attention mechanism; d k Represents the feature dimension.
9. The visible light image-guided millimeter wave radar depth enhancement method based on structure perception according to claim 2 is characterized in that: The specific operation of step S33 is: using the association confidence map obtained in step S32 as a reference, dr The depth of the position where the confidence is less than the threshold is deleted; finally, the enhanced radar depth map d is obtained er .