A Pseudo-Radar Vehicle Detection Method Based on Deep Completion

Abstract

This invention relates to the field of autonomous driving technology, and in particular to a pseudo-radar vehicle detection method based on fused depth completion. The method includes the following steps: converting scene information captured by a camera into an RGB image; constructing a deep learning model to obtain a dense depth map, and calculating distance data of objects in the scene using the dense depth map; constructing an end-to-end two-stage depth map completion model to obtain a color-dominated depth map and a depth-dominated depth map; processing the color-dominated depth map and the depth-dominated depth map to obtain a scale-based depth map and an optimized depth map, respectively; fusing the scale-based depth map and the optimized depth map to obtain a fused depth map; and converting the fused depth map into a pseudo-laser point cloud to calculate the distance, speed, and direction information between the vehicle and the camera using a 3D detection algorithm. This invention improves target detection accuracy by constructing a depth map completion model to complete depth information.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method for detecting fake radar vehicles by incorporating deep completion. Background Technology

[0002] 3D vehicle detection refers to the identification and localization of objects of interest from 3D data, and is an important task in fields such as autonomous driving, robot navigation, and augmented reality. The main purpose of 3D object detection is to perceive the position and orientation of surrounding vehicles or obstacles using data collected by sensors. Current 3D object detection technologies mainly include the following methods: Visual sensor-based methods, which use cameras to capture scene information and perform object detection and localization through deep learning models; LiDAR-based methods, which use the ranging principle of LiDAR to obtain distance information of objects in the scene; Data fusion-based methods, which combine the advantages of multiple data sources, using high-resolution RGB images to provide contextual information and LiDAR point clouds to provide spatial coordinate information; and Pseudo-Lidar-based methods, which convert depth information from monocular images into Pseudo-Lidar form and combine this Pseudo-Lidar with the image to predict 3D bounding boxes.

[0003] 3D object detection is a highly complex task. Visual sensor-based methods primarily use camera imaging, but due to a lack of depth information, their 3D detection accuracy is relatively low, resulting in inaccurate 3D object detection. LiDAR-based methods extract deep features by processing point cloud data, but they are expensive, significantly increasing the cost of autonomous driving. Furthermore, LiDAR's large size and susceptibility to environmental interference limit its widespread adoption in practical applications. Data fusion-based methods also face cost issues, requiring the deployment of multiple sensors for data fusion, and errors and inconsistencies between different sensors exacerbate the difficulties of data fusion. Pseudo-radar-based methods achieve 3D detection on monocular or binocular images by converting depth estimation maps into LiDAR point cloud formats, but they are significantly affected by the inaccuracy of depth estimation, resulting in low object detection accuracy. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a pseudo-radar vehicle detection method that integrates depth completion. By constructing a depth map completion model to complete the depth information, the method improves target detection accuracy.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for detecting fake radar vehicles by incorporating depth completion includes the following steps:

[0007] S1. Convert the scene information captured by the camera into an RGB image;

[0008] S2. Construct a deep learning model, extract depth feature information from RGB images through a convolutional neural network to obtain a dense depth map, and calculate the distance data of objects in the scene using the dense depth map;

[0009] S3. Construct an end-to-end two-stage depth map completion model. The depth map completion model includes a color-dominant branch and a depth-dominant branch. A color-dominant depth map is obtained through the color-dominant branch, and a depth-dominant depth map is obtained through the depth-dominant branch.

[0010] S4. Process the color-dominated depth map and the depth-dominated depth map to obtain a scale depth map and an optimized depth map respectively, and fuse the scale depth map and the optimized depth map to obtain a fused depth map;

[0011] S5. Convert the fused depth map into a pseudo-laser point cloud to calculate the distance, speed, and direction information between the vehicle and the camera using a 3D detection algorithm.

[0012] Further, in step S2, dense features in the RGB image are extracted using a ResNet model. Then, multi-scale feature fusion is performed using an Encoedr with fully connected layers and a hollow spatial pyramid pooling module to achieve a comprehensive understanding of the RGB image. Depth is represented using logarithmic space, and depth prediction is performed to generate five-dimensional tensor features. The dense depth map is obtained based on the five-dimensional tensor features.

[0013] The depth prediction is calculated as follows:

[0014]

[0015] Where SID represents the prediction strategy; t i ∈{t0,t1,…,t k} represents the discretization threshold after discretization of the depth interval; [α,β] represents the depth interval to be discretized; k represents the number of sub-intervals after discretization of the depth interval [α,β].

[0016] Furthermore, a softmax regression loss is used to learn the parameters of the dense depth map in order to calculate the distance data of objects in the scene.

[0017] Further, in step S3, the RGB image and the dense depth map are input into the color-dominant branch. By constructing an encoder and a decoder, the depth around the line boundary of the RGB image is learned, and the pixel information in the image is analyzed through the U-Net network architecture to obtain the color-dominant depth map of the color-dominant branch.

[0018] The dense depth map of the color-dominant branch and the dense depth map are input into the depth-dominant branch, and the pixel information in the image is analyzed through a residual network to obtain the depth-dominant depth map of the depth-dominant branch.

[0019] Furthermore, in step S4, a learnable linear transformation is added to the depth map completion model, and a fixed transformation calculated based on camera intrinsics and image resolution is used to convert the color-dominant depth map into a scaled depth map.

[0020] Further, in step S4, an adaptive weight term is set according to the reliability of the depth map, and a weight function is constructed using the gradient, variance, and edge features of the depth map to optimize the reliability of the depth map dominated by depth. The weight function is as follows:

[0021]

[0022] Where d is the depth map; α is the gradient operator; β is a hyperparameter controlling the rate of weight decay; and ||α·d|| is the norm of the gradient of the depth map.

[0023] Based on the consistency and contrast characteristics between the depth map and the RGB image, a feature function is constructed to perform continuity and smoothness optimization on the depth-dominated depth map after reliability optimization. The feature function is as follows:

[0024]

[0025] Where d is the depth map; c is the RGB image; and γ is a hyperparameter.

[0026] Further, in step S4, the scale depth map and the optimized depth map are fused using FusionNet to obtain a fused depth map. The fusion method is as follows:

[0027]

[0028] Where (u,v) is the position of the pixel in the depth map; D f(u,v) The fused dense depth map output by the two-stage depth completion model; C cd Confidence plot generated for the color-dominant branch; C dd Confidence plot generated for the color-dominant branch; D cdThe scale depth map output for the color-dominant branch; D dd An optimized depth map output for depth-guided branches.

[0029] Further, in step S5, the method for converting the fused depth map into a pseudo-laser point cloud is as follows:

[0030]

[0031] Where (u,v) is the position of the pixel in the depth map; (cu,cv) is the pixel position corresponding to the camera center; fv is the vertical focal length; fu is the horizontal focal length;

[0032] All pixels are back-projected onto 3D coordinates using formula (5) to obtain the 3D position (x, y, z) of each pixel. Based on the 3D position of each pixel, the following is obtained: The pseudo-laser point cloud, where N is the number of pixels.

[0033] The beneficial effects of this invention are:

[0034] A dense depth map is obtained by extracting depth features from an RGB image. The RGB image and the dense depth map are input to the color-dominant branch. A Resblocks encoder and a decoder containing five deconvolutional layers and one convolutional layer are constructed. The RGB image is used to learn the depth around line boundaries, and pixel information in the image is analyzed according to the U-Net network architecture to infer the shape of occluded objects. The dense depth map from the color-dominant branch and the dense depth map are input to the depth-dominant branch, and pixel information in the image is analyzed through a residual network to infer the position of occluded objects. By analyzing the shape and position of occluded objects, missing depth information is filled in, reducing the impact of depth estimation and effectively improving the accuracy of subsequent object detection. Since dense depth maps differ from RGB images, a learnable linear transformation is added to the depth map completion model, enabling the calculation of a scale-based depth map based on camera intrinsics and image resolution. By optimizing the reliability, continuity, and smoothness of the depth-dominant depth map, the accuracy of depth estimation can be improved. Fusing the processed scale-based depth map with the optimized depth map can overcome errors and inconsistencies between different feature representations, reducing the impact of inaccurate depth estimation. Attached Figure Description

[0035] Figure 1 This is a flowchart of a pseudo-radar vehicle detection method based on fusion depth completion, according to a preferred embodiment of the present invention.

[0036] Figure 2 This is a flowchart of the depth map processing of a pseudo-radar vehicle detection method with fusion depth completion according to a preferred embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0039] Please see Figure 1 A preferred embodiment of the present invention provides a method for detecting fake radar vehicles using fusion depth completion, comprising the following steps:

[0040] S1. Convert the scene information captured by the camera into an RGB image.

[0041] S2. Construct a deep learning model to extract depth feature information from RGB images using a convolutional neural network to obtain a dense depth map. Calculate the distance data of objects in the scene using the dense depth map.

[0042] In step S2, dense features in the RGB image are extracted using a ResNet model. Then, multi-scale feature fusion is performed using an Encoedr module with fully connected layers and a hollow spatial pyramid pooling module to achieve a comprehensive understanding of the RGB image. Depth is represented using logarithmic space and depth prediction is performed to generate five-dimensional tensor features. A dense depth map is obtained based on the five-dimensional tensor features.

[0043] The depth prediction is calculated as follows:

[0044]

[0045] Where SID represents the prediction strategy; t i ∈{t0,t1,…,t k} represents the discretization threshold after discretization of the depth interval; [α,β] represents the depth interval to be discretized; k represents the number of sub-intervals after discretization of the depth interval [α,β].

[0046] Softmax regression loss is used to learn the parameters of the dense depth map in order to calculate the distance data of objects in the scene.

[0047] S3. Construct an end-to-end two-stage depth map completion model. The depth map completion model includes a color-dominant branch and a depth-dominant branch. The color-dominant branch is used to obtain the color-dominant depth map, and the depth-dominant branch is used to obtain the depth-dominant depth map.

[0048] In step S3, an RGB image and a dense depth map are input into the color-dominant branch. By constructing an encoder and decoder, the depth around the line boundaries of the RGB image is learned, and the pixel information in the image is analyzed through the U-Net network architecture to obtain the color-dominant depth map of the color-dominant branch.

[0049] The depth-dominant branch takes the dense depth map and dense depth map of the color-dominant branch as input, and analyzes the pixel information in the image through a residual network to obtain the depth-dominant branch's depth-dominant depth map.

[0050] In the color-dominant branch, an RGB image and a dense depth map are input. A Resblocks encoder and a decoder containing five deconvolutional layers and one convolutional layer are constructed to learn the depth around the line boundaries using the RGB image. In the depth-dominant branch, the dense depth map and the dense depth map from the color-dominant branch are input. The pixel information in the image is analyzed through the U-Net network architecture and residual network to infer the shape and position of occluded objects and fill in the missing depth information.

[0051] S4. Process the color-dominated depth map and the depth-dominated depth map to obtain the scale depth map and the optimized depth map respectively. Then, fuse the scale depth map and the optimized depth map to obtain the fused depth map.

[0052] The processing flow of color-dominated depth maps and depth-dominated depth maps is as follows: Figure 2 As shown,

[0053] In step S4, a learnable linear transformation is added to the depth map completion model, and a fixed transformation calculated based on camera intrinsics and image resolution is used to convert the color-dominated depth map into a scaled depth map. Considering the differences between dense depth maps and RGB images, the consistency between dense depth maps and RGB images is ensured after linear transformation. Based on the camera intrinsic matrix K and image resolution W×H, the focal length f and baseline b of the depth map are calculated, and then d×W×f×b is used to obtain the scaled depth map.

[0054] In step S4, an adaptive weight term is set according to the reliability of the depth map, and a weight function is constructed using the gradient, variance, and edge features of the depth map to optimize the reliability of the depth map dominated by depth. The weight function is as follows:

[0055]

[0056] Where d is the depth map; α is the gradient operator; β is the hyperparameter controlling the rate of weight decay; and ||α·d|| is the norm of the gradient of the depth map.

[0057] Based on the consistency and contrast characteristics between the depth map and the RGB image, a feature function is constructed to perform continuity and smoothness optimization on the depth-dominated depth map after reliability optimization. The feature function is as follows:

[0058]

[0059] Where d is the depth map; c is the RGB image; and γ is a hyperparameter.

[0060] The feature function indicates that the greater the difference between the depth map and the color map, the weaker the relationship between depth and color, and thus it is given a smaller weight; the smaller the difference between the depth map and the color map, the stronger the relationship between depth and color, which may be more reliable, and thus it is given a larger weight.

[0061] In step S4, the scale depth map and the optimized depth map are fused using FusionNet to obtain a fused depth map. The fusion method is as follows:

[0062]

[0063] Where (u,v) is the position of the pixel in the depth map; D f(u,v) The fused dense depth map output by the two-stage depth completion model; C cd Confidence plot generated for the color-dominant branch; C dd Confidence plot generated for the color-dominant branch; D cd The scale depth map output for the color-dominant branch; D dd An optimized depth map output for depth-guided branches.

[0064] S5. Convert the fused depth map into a pseudo-laser point cloud to calculate the distance, speed, and direction between the vehicle and the camera using a 3D detection algorithm.

[0065] In step S5, the method for converting the fused depth map into a pseudo-laser point cloud is as follows:

[0066]

[0067] Where (u,v) is the position of the pixel in the depth map; (cu,cv) is the pixel position corresponding to the camera center; fv is the vertical focal length; fu is the horizontal focal length;

[0068] All pixels are back-projected onto 3D coordinates using formula (5) to obtain the 3D position (x, y, z) of each pixel. Based on the 3D position of each pixel, the following is obtained: The pseudo-laser point cloud. Where N represents the number of pixels.

[0069] The 3D detection algorithm is used to calculate information such as distance, speed and direction between the vehicle and the camera to realize vehicle detection and tracking; the performance of the pseudo-radar vehicle detection method with fusion deep completion technology is evaluated using evaluation indicators such as accuracy, recall, precision and F1 score.

Claims

1. A method for detecting fake radar vehicles by fusing depth completion, characterized in that, Includes the following steps: S1. Convert the scene information captured by the camera into an RGB image; S2. Construct a deep learning model, extract depth feature information from RGB images through a convolutional neural network to obtain a dense depth map, and calculate the distance data of objects in the scene using the dense depth map; In step S2, dense features in the RGB image are extracted using a ResNet model. Then, multi-scale feature fusion is performed using an Encoedr with fully connected layers and a hollow spatial pyramid pooling module to achieve a comprehensive understanding of the RGB image. Depth is represented using logarithmic space, and depth prediction is performed to generate five-dimensional tensor features. The dense depth map is obtained based on these five-dimensional tensor features. The depth prediction is calculated as follows: Where SID represents the prediction strategy; t i ∈{t0,t1,…,t k } represents the discretization threshold after discretization of the depth interval; [α,β] represents the depth interval to be discretized; k represents the number of sub-intervals after discretization of the depth interval [α,β]. S3. Construct an end-to-end two-stage depth map completion model. The depth map completion model includes a color-dominant branch and a depth-dominant branch. A color-dominant depth map is obtained through the color-dominant branch, and a depth-dominant depth map is obtained through the depth-dominant branch. In step S3, the RGB image and the dense depth map are input into the color-dominant branch. By constructing an encoder and decoder, the depth around the line boundary of the RGB image is learned, and the pixel information in the image is analyzed through the U-Net network architecture to obtain the color-dominant depth map of the color-dominant branch. The dense depth map of the color-dominant branch and the dense depth map are input into the depth-dominant branch, and the pixel information in the image is analyzed through a residual network to obtain the depth-dominant depth map of the depth-dominant branch; S4. The color-dominant depth map and the depth-dominant depth map are processed to obtain a scale depth map and an optimized depth map respectively, and the scale depth map and the optimized depth map are fused to obtain a fused depth map; In step S4, an adaptive weight term is set according to the reliability of the depth map, and a weight function is constructed using the gradient, variance, and edge features of the depth map to optimize the reliability of the depth map dominated by depth. The weight function is as follows: Where d is the depth map; α is the gradient operator; β is a hyperparameter controlling the rate of weight decay; and ||α·d|| is the norm of the gradient of the depth map. Based on the consistency and contrast characteristics between the depth map and the RGB image, a feature function is constructed to perform continuity and smoothness optimization on the depth-dominated depth map after reliability optimization. The feature function is as follows: Where d is the depth map; c is the RGB image; and γ is a hyperparameter. In step S4, a learnable linear transformation is added to the depth map completion model, and a fixed transformation calculated based on camera intrinsics and image resolution is used to convert the color-dominant depth map into a scaled depth map. S5. Convert the fused depth map into a pseudo-laser point cloud to calculate the distance, speed, and direction information between the vehicle and the camera using a 3D detection algorithm.

2. The method for detecting fake radar vehicles using fusion depth completion according to claim 1, characterized in that: The parameters of the dense depth map are learned using softmax regression loss to calculate the distance data of objects in the scene.

3. The method for detecting fake radar vehicles using fusion depth completion according to claim 1, characterized in that: In step S4, the scale depth map and the optimized depth map are fused using FusionNet to obtain a fused depth map. The fusion method is as follows: Where (u,v) is the position of the pixel in the depth map; D f(u,v) The fused dense depth map output by the two-stage depth completion model; C cd Confidence plot generated for the color-dominant branch; C dd Confidence plot generated for the color-dominant branch; D cd The scale depth map output for the color-dominant branch; D dd An optimized depth map output for depth-guided branches.

4. The method for detecting fake radar vehicles using fusion depth completion according to claim 1, characterized in that: In step S5, the method for converting the fused depth map into a pseudo-laser point cloud is as follows: Where (u,v) is the position of the pixel in the depth map; (cu,cv) is the pixel position corresponding to the camera center; fv is the vertical focal length; fu is the horizontal focal length; All pixels are back-projected onto 3D coordinates using formula (5) to obtain the 3D position (x, y, z) of each pixel. Based on the 3D position of each pixel, the following is obtained: The pseudo-laser point cloud, where N is the number of pixels.

Images