A monocular depth estimation method based on target surface point height and uncertainty

Abstract

The application discloses a monocular depth estimation method based on target surface point height and uncertainty, proposes to replace the height of an object itself with the physical height and the projection height of a surface point of the object, so that a plurality of depth candidate values are obtained. In addition, the uncertainty of the height is estimated, and the inferred instance depth with high uncertainty is replaced with a directly-regressed depth. According to the uncertainty, the depth predictions are combined to obtain a final target depth. The application achieves the best performance on the KITTI dataset, and the reasoning time meets the real-time requirement of automatic driving.

Description

Technical Field

[0001] This invention relates to the field of deep learning monocular 3D target detection in the field of computer technology, specifically a monocular depth estimation method based on the height and uncertainty of target surface points. Background Technology

[0002] Monocular 3D object detection is a fundamental task in the fields of autonomous driving and robot navigation. With the rapid development of deep neural networks, 2D object detection has achieved satisfactory performance. However, 3D object detection is more difficult and challenging because it requires predicting the object's position, size, and orientation in 3D space.

[0003] Recent LiDAR-based 3D target detection methods have achieved good performance due to accurate depth estimation; however, these methods are costly and computationally intensive for autonomous vehicles. Monocular methods, with their advantages of economy and portability, are gaining increasing attention in the computer vision community. However, due to the lack of depth information, a significant gap still exists between image-based monocular methods and LiDAR-based methods.

[0004] Predicting instance depth directly from an image is an ill-posed problem. In recent years, some works have focused on depth estimation based on geometric projection. For example, MonoRCNN and GUPNet regress the heights of 2D and 3D bounding boxes, respectively, and then infer instance depth using projection constraint equations. However, due to the influence of the viewing angle and object position, the height of a 2D bounding box is difficult to represent the projection of the physical height, which should be the distance from the top of the object to the ground below. Furthermore, this method only estimates the depth of a single instance, which may be unreliable.

[0005] Some works predict the eight corners (keypoints) of a 3D bounding box and generate depth candidates using the known physical height. However, the corners of the 3D bounding box are often not physical points on the object; some may be occluded, while others may belong to the background. Furthermore, the predicted height error propagates to the final instance depth, amplifying the depth estimation error. Summary of the Invention

[0006] The purpose of this invention is to estimate the physical height and pixel height of sampling points on the target surface based on the GUPNet architecture, thereby achieving high-precision depth estimation. It provides a monocular depth estimation method based on the target surface point height and uncertainty, which achieves the highest detection performance among monocular methods on the 3D target detection dataset KITTI.

[0007] To achieve the above objectives, the specific technical solution is as follows:

[0008] A monocular depth estimation method based on target surface point height and uncertainty, comprising the following steps:

[0009] Step 1: During the training phase, each target in the road traffic scene image, namely cars, pedestrians and cyclists, is decomposed into n*n surface sampling points. The physical height and pixel height of the surface sampling points are modeled, and the true values ​​of the two heights are calculated using camera parameters and projection transformation, which serve as the true labels during training.

[0010] Step 2: Inference stage, after the input image is processed by the backbone network to extract features, multiple detectors are connected to predict the physical height and pixel height of each surface sampling point and the Laplacian uncertainty corresponding to the two heights, and predict the distance from each surface sampling point to the target 3D center. For each pair of physical height and pixel height, a candidate surface depth value and the uncertainty corresponding to this depth value are calculated using the camera pinhole model.

[0011] Step 3: Add the depth of the surface sampling point to the distance from the surface point to the target 3D center to obtain n*n candidate depth values ​​for instances;

[0012] Step 4: Based on the uncertainty of the instance depth of each surface sampling point, the depth candidate values ​​are weighted and combined to form the final instance depth;

[0013] Step 5: Train the network using the 3D bounding box real labels provided by the KITTI training set and the real labels generated in Step 1; after training, perform performance testing on the KITTI test set.

[0014] The physical height and pixel height of the surface sampling point mentioned in step 1 refer to the distance h between the surface point and the bottom surface in the camera coordinate system. 3d and h 3d The projected pixel height h in the image 2d The specific calculations are as follows:

[0015] a1: Project the radar point cloud onto the image plane to obtain a pixel-level depth map, and use RoI-Align to obtain the pixel coordinates of n*n surface sampling points and the corresponding depth labels;

[0016] a2: Reproject surface points into 3D space using camera intrinsics:

[0017]

[0018] Among them, (u sur v sur (c) represents the pixel coordinates of the surface sampling point. u c v (f) represents the pixel coordinates of the image center. x f y (x) represents the horizontal focal length and vertical focal length of the camera, respectively.sur y sur , z sur () represents the coordinates of a point on the surface in 3D space;

[0019] a3: Subtract the y-coordinate of the center point of the bottom face of the target's 3D bounding box from the y-coordinate of the surface point in 3D space to obtain the true label of the physical height.

[0020]

[0021] a4: Using camera perspective geometry, the image plane is projected to obtain the true label of pixel height.

[0022]

[0023] Where f represents the camera focal length.

[0024] Step 2 describes the prediction of the physical height h. 3d and pixel height h 2d A candidate surface depth value is calculated as follows:

[0025]

[0026] Where f represents the camera focal length.

[0027] The instance depth candidate values ​​for the n*n instances mentioned in step 3 are obtained by adding the depth of the surface point and the distance from the surface point to the target 3D center:

[0028] z geo =z sur +δ z (5)

[0029] Where, δ z This represents the distance from a point on the surface to the 3D center of the target.

[0030] Step 4, which involves weighting and combining the candidate depth values ​​to arrive at the final instance depth, specifically involves:

[0031]

[0032] in, This represents the instance depth of the surface sampling point at position (i, j) among the n*n surface sampling points in the 2D bounding box, where i is the x-coordinate and j is the y-coordinate; ω ij The weight z represents the depth of the surface sampling point instance. ins This indicates the final instance depth.

[0033] Compared with existing technologies, the advantages of this invention are as follows: Based on the GUPNet framework, the depth candidate values ​​inferred from the physical height and pixel height of surface sampling points are more accurate and reliable than traditional camera pinhole model methods, while maintaining the inference speed required for real-time operation. Specifically:

[0034] 1) The physical height and pixel height of the surface sampling points predicted by this invention are not affected by the observation angle and the length of the target vehicle body, resulting in smaller errors and more reliable inferred instance depths. This invention achieves the best 3D detection and BEV detection performance on the KITTI 3D object detection dataset.

[0035] 2) Traditional methods based on camera pinhole models only predict depth at the instance level, resulting in fewer depth candidate values. This invention generates n*n depth candidate values ​​for a single target. When some surface sampling points are occluded or truncated, the remaining visible points can still be used to infer reliable depth values.

[0036] 3) While modeling the height of surface sampling points, it can also perceive the three-dimensional contour features of the target. Therefore, the 3D detection results of this invention in the KITTI dataset are more significantly improved than the BEV detection results.

[0037] 4) The inference time of this invention for a single frame image on an Nvidia GPU 3090 card is 40ms, which far exceeds some methods of monocular 3D detection. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of monocular depth estimation according to the present invention;

[0039] Figure 2 This is a flowchart of the present invention;

[0040] Figure 3-4 This is a scenario diagram of an embodiment of the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings. The process and principles of the present invention are very clear to those skilled in the art. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0042] This invention proposes a monocular depth estimation method (SHUD) based on the height and uncertainty of target surface points. This method models and predicts the physical height and projected pixel height of the target surface sampling points. It is based on a deep neural network model with GUPNet network as the basic architecture and achieves excellent test performance.

[0043] This invention includes the following steps:

[0044] Step 1: During the training phase, each target in the road traffic scene image, namely cars, pedestrians and cyclists, is decomposed into n*n surface sampling points. The physical height and pixel height of the surface sampling points are modeled, and the true values ​​of the two heights are calculated using camera parameters and projection transformation, which serve as the true labels during training.

[0045] Step 2: Inference stage, after the input image is processed by the backbone network to extract features, multiple detectors are connected to predict the physical height and pixel height of each surface sampling point and their Laplacian uncertainty, and predict the distance from each surface sampling point to the target 3D center. For each pair of physical height and pixel height, a candidate surface depth value and the uncertainty corresponding to this depth value are calculated using the camera pinhole model.

[0046] Step 3: Add the depth of the surface sampling point to the distance from the surface point to the target 3D center to obtain n*n candidate depth values ​​for instances;

[0047] Step 4: Based on the uncertainty of the instance depth of each surface sampling point, the depth candidate values ​​are weighted and combined to form the final instance depth;

[0048] Step 5: Train the network using the 3D bounding box real labels provided by the KITTI training set and the real labels generated in Step 1; after training, perform performance testing on the KITTI test set.

[0049] Example

[0050] 1. Building network infrastructure

[0051] like Figure 2 As shown, a GUPNet network structure is constructed, and RoI-Align is used to divide the detected 2D bounding boxes into n*n grids. The center of each grid represents a surface sampling point. The detection head is used to predict the height information and corresponding uncertainty of each surface sampling point.

[0052] 2. Depth of the inference surface

[0053] like Figure 1 As shown, the target depth is estimated from the physical height and pixel height of a set of target surface points; the physical height and pixel height regressed from each surface sampling point are used to calculate n*n candidate surface depth values ​​using a camera pinhole model.

[0054] 3. Depth of candidate instances for inference

[0055] The distance from the surface sampling point predicted by the network to the target 3D center is added to the depth of the surface sampling point to obtain the depth of n*n candidate instances.

[0056] 4. The final instance depth of inference

[0057] They are weighted and combined into a final single instance depth based on the uncertainty of the depth of each candidate instance.

[0058] 5. Train a deep convolutional neural network.

[0059] 6. Test after training is completed.

[0060] Once the model is trained, its parameters are fixed, and the model is validated on the test set to obtain test results.

[0061] The following is a specific example of the training process. In this example, the image dataset used for training is KITTI.

[0062] i) Input Figure 3 The image shown contains a car, and the detected 2D bounding box is as follows: Figure 4 As shown;

[0063] ii) Set the surface point sampling rate to 5*5. For example... Figure 2 As shown, after the input image is processed by the backbone network to extract features, multiple detectors are connected to predict the physical height and pixel height of each surface sampling point and their Laplacian uncertainty, and the distance from each surface sampling point to the target 3D center is predicted. For each pair of physical height and pixel height, a candidate surface depth value and the uncertainty corresponding to this depth value are calculated using the camera pinhole model.

[0064] iii) Add the depth of the surface sampling point to the distance from the surface point to the target 3D center to obtain 5*5 candidate depth values ​​for instances;

[0065] iv) Based on the uncertainty of the instance depth of each surface sampling point, the depth candidate values ​​are weighted and combined to obtain the final instance depth of the vehicle;

[0066] v) Finally, the Laplacian uncertainty loss function is used to backpropagate the gradient of the loss calculated between the output and the true labels of the 3D bounding boxes provided by the KITTI dataset, updating the network parameters. After training, performance is tested on the test set.

[0067] 1. The KITTI sub-training set containing three categories—cars, pedestrians, and cyclists—was used for training. Different surface point sampling rates were set. The experimental results on the validation set after training are compared as shown in Table 1 below.

[0068] Table 1 Comparison of model performance at different surface point sampling rates

[0069]

[0070] 2. The training was performed on the complete KITTI training set containing three categories. The experimental results of Car on the test set after training are compared as shown in Table 2 below.

[0071] Table 2 Performance Comparison of Different Monocular 3D Target Detectors

[0072]

[0073]

Claims

1. A monocular depth estimation method based on target surface point height and uncertainty, characterized in that, The method comprises the following steps: Step 1: training phase, each target in the image of the road traffic scene, i.e. car, pedestrian and rider, is decomposed into n*n surface sampling points, the physical height and pixel height of the surface sampling points are modeled, the real values of the two heights are calculated by using the camera parameters and projection transformation as the real labels during training; Step 2: inference phase, after the input image is extracted by the backbone network, the multiple detection heads are connected to predict the physical height and pixel height of each surface sampling point and the Laplace uncertainty corresponding to the two heights, and the distance from each surface sampling point to the 3D center of the target is predicted, for each pair of physical height and pixel height, a candidate surface depth value and the uncertainty corresponding to the depth value are calculated by using the camera pinhole model; Step 3: after the depth of the surface sampling point is added to the distance from the surface point to the 3D center of the target, n*n instance depth candidate values are obtained; Step 4: according to the uncertainty of the instance depth of each surface sampling point, the depth candidate values are weighted and combined into the final instance depth; Step 5: the network is trained by using the 3D bounding box real label provided by the KITTI training set and the real label generated in step 1; After training, the performance is tested on the KITTI test set; wherein: The physical height and pixel height of the surface sampling point in step 1 refer to the distance from the surface point to the bottom in the camera coordinate system and The projected pixel height in the image ; The specific calculation is as follows: a1: the radar point cloud is projected to the image plane to obtain the pixel-level depth map, and the pixel coordinates of n*n surface sampling points and the corresponding depth label are obtained by using RoI-Align; a2: the surface points are reprojected to the 3D space by using the camera intrinsic parameters: ； in,( , ) represents the pixel coordinates of the surface sampling point, ( , ) represents the pixel coordinates of the image center, ( , ) represent the horizontal focal length and vertical focal length of the camera, respectively. , , () represents the coordinates of a point on the surface in 3D space; a3: subtract the y coordinate of the center point of the bottom surface of the target 3D bounding box from the y coordinate of the surface point in the 3D space to obtain the real label of the physical height : ； a4: using camera perspective geometry model to get real label's pixel height from projection to image plane : ； wherein denotes the camera focal length; Step 2 Calculate a candidate surface depth value based on the predicted physical height and the pixel height A candidate surface depth value is calculated as follows: ； wherein denotes the camera focal length; The n*n instance depth candidate values in step 3 are obtained by adding the depth of the surface point and the distance from the surface point to the 3D center of the target: ； wherein, represents the distance of the surface point to the target 3D center.

2. The monocular depth estimation method based on target surface point height and uncertainty as claimed in claim 1, wherein, The final instance depth is obtained by weighting and combining the depth candidate values in step 4, specifically: ； wherein, denotes the instance depth of a surface sample at position (i, j) in the n * n surface samples of the 2D bounding box, i being the horizontal coordinate and j being the vertical coordinate denotes the weight of the instance depth of this surface sample, denotes the final instance depth.

Images