A Method for Constructing a Monocular 3D Detection Model Based on Target Surface Points and Ground Mapping

Abstract

This invention discloses a method for constructing a monocular 3D detection model based on the mapping of target surface points to the ground. Monocular 3D target detection has attracted much attention due to its simplicity and low cost. This invention maps object surface points to the ground plane and decomposes the object depth problem into ground depth estimation and surface point height estimation. During the training phase, dense ground depth labels are provided by the target surface depth. During the inference phase, the surface point depth is recovered by querying the ground depth map. This results in a large number of candidate target depths, and the final instance depth can be combined based on the uncertainty of the candidate target depths. Finally, the 3D bounding box of the target is obtained by combining other predicted 3D attributes. Furthermore, since most mapped ground points are occluded by objects, which may mislead network learning, this invention designs a depth augmentation strategy to expand the ground labels. This invention achieves superior performance on the KITTI dataset, showing a significant performance improvement compared to similar methods.

Description

Technical Field

[0001] This invention relates to the fields of computer technology and artificial intelligence, specifically to the field of deep learning monocular 3D target detection, and more specifically to a method for constructing a monocular 3D detection model based on the mapping of target surface points to the ground. Background Technology

[0002] 3D object detection is gaining increasing attention in autonomous driving and other visual perception tasks. While LiDAR-based methods involve expensive sensors and stereo detectors require precise online calibration, monocular 3D object detection stands as a promising solution due to its simplicity and cost advantages. A core challenge of monocular methods is the lack of explicit depth cues, which significantly limits detection performance.

[0003] Inferring object depth directly from an image is an ill-posed problem. Various depth cues have been used, for example, for depth estimation. Some works employ 3D-2D geometric projection constraints, including height-based and keypoint-based methods to recover depth. A few works consider ground-plane priors to estimate target depth from ground depth.

[0004] In mainstream 3D object detection frameworks, objects are represented by their center points. Therefore, 3D object detection is based on local features, which is beneficial for learning the intrinsic properties of objects. However, depth estimation depends on global scene information. Furthermore, during training, only instance depth or keypoint depth is used for supervision, resulting in depth sparsity. Summary of the Invention

[0005] The purpose of this invention is to predict dense ground depth maps based on the CenterNet detection architecture. During the inference phase, the target depth is queried and sampled from the ground depth map to obtain a more accurate and reliable target depth, thus constructing a monocular 3D detection model based on the mapping between target surface points and the ground. This model achieves state-of-the-art performance on the mainstream 3D target detection dataset KITTI, showing a significant performance improvement compared to similar methods.

[0006] The specific technical solution for achieving the objective of this invention is as follows:

[0007] A method for constructing a monocular 3D detection model based on the mapping of target surface points to the ground, the method comprising the following steps:

[0008] Step 1: During the training phase, the 2D bounding box of each target in the road scene image, namely vehicles, pedestrians and cyclists, is divided into s*s grids. The center of each grid represents a surface point. The surface point is mapped to the ground below the target using the camera perspective geometry formula to obtain the true labels of the depth of s*s ground points.

[0009] Step 2: For each mapped ground point, keep the v-axis coordinate of the pixel coordinate unchanged, expand each ground point into m points on the u-axis, and keep the expanded ground points at the same depth as the original ground points. For each target, m*s*s ground point depth can be generated as true labels.

[0010] Step 3: Based on the real labels generated in Step 2, train a ground depth map with the same size as the road scene image using dilated convolution;

[0011] Step 4: Inference phase, the depth of each target surface point is queried from the trained ground depth map, and then the depths of s*s surface points of each target are combined to form the depth of the final target 3D center, which is combined with other 3D attributes predicted by the 3D base head to form the target's 3D bounding box.

[0012] Step 1, mapping surface points onto the ground below the target, refers to first back-projecting the surface sampling points into 3D space, i.e., the camera coordinate system, then mapping their 3D coordinates vertically downwards onto the ground, and finally projecting the mapped ground points onto the image plane; the specific process is as follows: a1: Back-project the surface points into the camera coordinate system based on their pixel coordinates and corresponding known true depth labels:

[0013]

[0014] Among them, (u sur v sur (c) represents the pixel coordinates of a surface point. u c v (f) represents the center pixel coordinates of the image. x f y (x) represents the horizontal focal length and vertical focal length of the camera, respectively. sur y sur , z sur () represents the coordinates of a point on the surface in 3D space;

[0015] a2: Step a1 yields the 3D coordinates (X) of the surface point. sur Y sur Z sur ), to Z sur Replace with the Z-axis of the bottom center point of the target 3D bounding box label, which is the actual target 3D bounding box label provided in the KITTI dataset. c Coordinates, to obtain the mapping point (X) of the surface point on the ground. sur Y sur Z c );

[0016] a3: Projecting ground mapping points onto the image plane using camera parameters:

[0017] Z sur [u grd v grd ,1] T =K 3×3 [X sur Y c Z sur ] T (2)

[0018] Where Y c It is the height of the ground; the pixel coordinates of the projected ground points are (u grd v grd The depth of this ground point is Z. sur T represents the transpose operation.

[0019] Step 4 involves querying the depth of each target surface point from a ground depth map, specifically including:

[0020]

[0021] Among them, (u grd v sur () represents the pixel coordinates of a point on the surface. h represents the pixel coordinates of the mapped ground points. sur This represents the offset along the v-axis from the surface point to the ground point predicted by the network; then from... The location is sampled from the ground depth map to obtain the depth of the corresponding surface point.

[0022] Compared with existing technologies, the advantages of this invention are as follows: Based on the CenterNet framework, this invention maps target surface points onto the ground during the training phase, constructing realistic labels for ground depth for network learning and training. During the inference phase, depth is sampled from the ground depth map predicted by the network based on the target's ground contact points. This model predicts ground depths with continuity and greater reliability, while its inference speed meets the real-time requirements of autonomous driving. Specifically:

[0023] 1) Targets in an image are independent of each other, therefore their depth is discrete. Ground depth, however, is inherently global and independent of objects in the scene. A more accurate target depth estimation is achieved by decoupling target depth estimation into ground depth prediction and target depth querying and sampling based on ground contact points.

[0024] 2) By constructing a mapping between target surface points and ground contact points, the network can perceive the three-dimensional shape features of the target, and the prediction deviation of the ground contact point position is smaller, which is beneficial for querying the target depth.

[0025] 3) The present invention demonstrates a significant performance improvement over methods utilizing ground priors in the KITTI 3D object detection dataset. It also outperforms other state-of-the-art monocular detection methods, proving the reliability and superiority of the present invention.

[0026] 4) The inference time for a single RGB image on a single Nvidia GPU 3090 graphics card is 40ms, which is faster than other methods. Attached Figure Description

[0027] Figure 1 This is a schematic diagram illustrating the depth query and sampling of target surface points in this invention;

[0028] Figure 2 The model structure diagram constructed for this invention;

[0029] Figure 3 A schematic diagram illustrating the mapping from target surface points to ground contact points and the strategy for expanding the depth of ground points;

[0030] Figure 4-5 This is a scenario diagram of an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings. The model structure and principles of the present invention are very clear to those skilled in the art. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0032] This invention proposes to construct a monocular 3D detection model (MonoSGC) based on the mapping of target surface points to the ground. The construction process is as follows: it uses ground prior information to help estimate the depth of the target, and is built on the CenterNet network architecture. It achieves excellent performance while meeting the requirements of real-time inference time.

[0033] This invention includes the following steps:

[0034] Step 1: Training phase, such as Figure 3 As shown, the 2D bounding box of each target in the road scene image, namely vehicles, pedestrians and cyclists, is divided into s*s grids. The center of each grid represents a surface point. The surface point is mapped to the ground below the target using the camera perspective geometry formula to obtain the true labels of the depth of s*s ground points.

[0035] Step 2: For each mapped ground point, keeping the v-axis coordinate of the pixel coordinate unchanged, expand each ground point into m points on the u-axis, such as... Figure 3 As shown, the expanded ground points maintain the same depth as the original ground points, and can generate m*s*s ground point depth true labels for each target.

[0036] Step 3: Based on the ground truth labels generated in Step 2, train a ground depth map with the same size as the road scene image using dilated convolution, such as... Figure 2 As shown in the upper middle branch;

[0037] Step 4: Reasoning stage, such as Figure 1 As shown, the depth of each target surface point is queried from the trained ground depth map, and then the depths of s*s surface points of each target are combined to form the final depth of the target's 3D center. Figure 1 The 3D bounding box of the target is obtained by combining the attributes predicted by the base head.

[0038] Example

[0039] 1. Building the model

[0040] Based on CenterNet architecture, DLA-34 is used as the backbone network to extract image features, and the head part uses a Conv+BN+ReLU hierarchical structure.

[0041] 2. Constructing the true labels needed for training

[0042] like Figure 3 As shown, for each target, it is decomposed into s*s surface points using uniform sampling, and each surface point is mapped onto the ground. The depth of the resulting ground contact point is the same as the depth of the surface point. Therefore, for each target, s*s ground point depth labels are generated. Then, a depth augmentation strategy is used to obtain m*s*s ground point depth labels.

[0043] 3. Obtain the target depth during the inference phase.

[0044] like Figure 1 As shown, for each surface point of the target, the corresponding ground contact point position is predicted, and the depth information of the corresponding position is queried from the trained ground depth to obtain the depth of the surface point.

[0045] 4. Obtain the 3D bounding box of the target.

[0046] The surface point depths of each target are combined to form the final target depth, which, together with other predicted 3D attributes, constitutes the target's 3D bounding box.

[0047] The following is a specific example of training this model on the KITTI dataset.

[0048] i) Input Figure 4 The image shown; a white car was detected in this image, and the corresponding 2D bounding box is as follows. Figure 5 As shown;

[0049] ii) During the training phase, the surface point sampling rate is set to 50*50. Uniform sampling is used to decompose this 2D bounding box into 50*50 surface points, and each surface point is mapped onto the ground. The depth of the resulting ground contact point is the same as the depth of the surface point. Therefore, for this target, 50*50 ground point depth labels are generated. Then, a depth augmentation strategy is used to obtain 10*50*50 ground point depth labels.

[0050] iii) In the inference phase, the surface point sampling rate is set to 5*5. For each surface point of this target, the corresponding ground contact point position is predicted, and the depth information of the corresponding position is queried from the trained ground depth, thereby obtaining the depth of 5*5 surface points.

[0051] iv) Combine the surface point depths of this target to form the final target depth, and combine it with other predicted 3D attributes to form the target's 3D bounding box.

[0052] v) Finally, the generated gradients are backpropagated based on the model's output and the corresponding Ground Truth to update the network parameters. After training, performance is tested on the test set.

[0053] During the training phase, the sampling rate of the target surface points was set to 50*50, and the network was trained using L1, L2, Focal Loss, and Laplacian Aleatoric_Uncertainty_Loss. During the testing phase, the sampling rate of the target surface points was set to 5*5. The test results for the Car category on the official KITTI test set are shown in Table 1 below.

[0054] Table 1 Performance Comparison of Different Monocular 3D Target Detectors

[0055]

[0056]

Claims

1. A method for constructing a monocular 3D detection model based on the mapping of target surface points to the ground, characterized in that, The method includes the following steps: Step 1: During the training phase, the 2D bounding box of each target in the road scene image, namely vehicles, pedestrians and cyclists, is divided into s * s grids. The center of each grid represents a surface point. The surface point is mapped to the ground below the target using the camera perspective geometry formula to obtain the true depth labels of s * s ground points. Step 2: For each mapped ground point, keep the v-axis coordinate of the pixel coordinate unchanged, expand each ground point into m points on the u-axis, and keep the expanded ground points at the same depth as the original ground points. For each target, m * s * s ground point depth true labels can be generated. Step 3: Based on the real labels generated in Step 2, train a ground depth map with the same size as the road scene image using dilated convolution; Step 4: Inference phase, the depth of each target surface point is queried from the trained ground depth map. Then, the depths of s * s surface points of each target are combined to form the final depth of the target's 3D center. This depth is then combined with other 3D attributes predicted by the 3D base head to form the target's 3D bounding box; where: Step 1, mapping surface points onto the ground below the target, refers to first back-projecting the surface sampling points into 3D space, i.e., the camera coordinate system, then mapping their 3D coordinates vertically downwards onto the ground, and finally projecting the mapped ground points onto the image plane; the specific process is as follows: a1: Based on the pixel coordinates of the surface points and their corresponding known true depth labels, back-project them into the camera coordinate system: ； in,( , ) represents the pixel coordinates of a surface point, ( , ) represents the center pixel coordinates of the image, ( , ) represent the horizontal focal length and vertical focal length of the camera, respectively. , , () represents the coordinates of a surface point in 3D space; a2: Step a1 yields the 3D coordinates of the surface points ( , , ),Will Replace with the center point of the bottom face of the target 3D bounding box label, which is the actual target 3D bounding box label provided in the KITTI dataset. Coordinates, to obtain the mapping point of the surface point on the ground ( , , ); a3: Projecting ground mapping points onto the image plane using camera parameters: ； in It is the height of the ground; the pixel coordinates of the projected ground points are ( The depth of this ground point is T represents the transpose operation.

2. The method for constructing a monocular 3D detection model based on the mapping of target surface points to the ground as described in claim 1, characterized in that, Step 4 involves querying the depth of each target surface point from a ground depth map, specifically including: ； in,( , ) represents the pixel coordinates of a surface point, ( , () represents the pixel coordinates of the mapped ground point. The offset on the v-axis from the surface point to the ground point predicted by the network; then from ( , The location is sampled in the ground depth map to obtain the depth of the corresponding surface point.

Images