A 3D object detection method based on monocular camera

Abstract

The present invention discloses a 3D target detection method based on a monocular camera, comprising: (1) according to the imaging rule of the monocular camera, for each real target O, generating a pseudo target O′ with similar imaging in proportion to form an image data set; (2) calculating the quality score of the pseudo target O′ according to a designed quality score function; (3) extracting the feature F of each image in the image data set using a deep network; (4) predicting the 3D information of the object and the quality score of the predicted target based on the image feature F using a multi-layer perceptron; (5) using the pseudo target O′ with the quality score obtained in step (2) and the real target O as supervision information, and calculating the loss function with the prediction result of step (4); (6) training the deep network and the multi-layer perceptron through gradient backpropagation; (7) after the deep network and the multi-layer perceptron are trained, performing 3D target detection. The present invention can greatly improve the quality of monocular target detection.

Description

Technical Field

[0001] The present invention belongs to the fields of computer image processing and safety assisted driving, and in particular relates to a 3D target detection method based on a monocular camera. Background Art

[0002] In autonomous driving or safety-assisted driving systems, accurate perception of the vehicle's surroundings is crucial and is one of the main bases for the driving system to make reasonable decisions. Among them, 3D object detection is one of the important tasks of perception. Specifically, 3D object detection refers to using the information returned by the sensor to obtain the 3D properties of the target of interest (the target of interest generally refers to categories related to driving, such as cars, people, and roadblocks), including: the position of the target (coordinates XYZ in the world coordinate system, where Z represents depth and the orientation angle θ) and the dimensions of the target (length L, width W, height H). Monocular camera-based 3D object detection refers to the task of completing 3D object detection using only 2D images obtained by a single RGB camera. Due to the low cost, rich semantic information, high frame rate, and the lack of calibration requirements of RGB cameras, they have received extensive attention and research in recent years.

[0003] Currently, the mainstream 3D object detection methods based on monocular cameras can be divided into direct regression and the use of additional depth information. The former directly learns the various 3D properties of the target based on the features extracted from the image. For example, the paper "Rtm3d: Real-time monocular 3D detection from object keypoints for autonomous driving", published at the European Conference on Computer Vision in 2020, achieves real-time monocular 3D target detection through keypoint prediction. The paper "Geometry-based distance decomposition for monocular 3D object detection", published at the IEEE / CVF International Conference on Computer Vision in 2021, proposes a decomposition method based on geometric distance, which infers the distance of the target by its physical height and visual height in the image. The paper "MonoJSG: Joint Semantic and Geometric Cost Volume for Monocular 3D Object Detection", published at the IEEE / CVF Conference on Computer Vision and Pattern Recognition in 2022, reformulates depth estimation as a problem of progressive improvement and proposes a joint semantic and geometric cost volume to model depth errors. The latter uses additional depth information to assist the monocular 3D target detection task. For example, "Rethinking pseudo-lidar representation", published at the European Conference on Computer Vision, a top international computer vision conference in 2020, combines depth maps and RGB images in channels as new input data; "MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer", published at the IEEE / CVF Conference on Computer Vision and Pattern Recognition, a top international computer vision conference in 2022, designs a lightweight module to predict depth features, and then combines them with semantic information to predict the 3D properties of the target.

[0004] However, 3D target detection based on a monocular camera is an ill-posed problem. This is because the image captured by the monocular camera is 2D, while 3D target detection predicts the position and dimension information in the world coordinate system, and the dimensions of the input and output are different. Figure 2 As shown in Figure 1, different objects in the world coordinate system may have similar semantic features in the image coordinate system. Previous monocular 3D object detection methods used a one-to-one supervision approach, whereby a 3D object in the world coordinate system was used to supervise the learning of a 2D object in the image. This resulted in suboptimal detection performance. Summary of the Invention

[0005] The present invention provides a 3D target detection method based on a monocular camera. By using pseudo labels with quality scores and optimizing the model using a deep learning model, the network can learn a more general one-to-many relationship, greatly improving the quality of target detection, especially the accuracy of depth values.

[0006] A 3D target detection method based on a monocular camera includes the following steps:

[0007] (1) According to the imaging rules of the monocular camera, for each original real target O, a series of pseudo targets O′ with similar imaging are generated proportionally to form a new target detection dataset;

[0008] (2) According to the designed quality score function, the quality score of the pseudo target O′ is calculated to indicate the similarity between the pseudo target and its true target, where the quality score of the true target O is assumed to be 1.0 by default;

[0009] (3) Use a deep network to extract the features F of each image in the image dataset;

[0010] (4) Using a multi-layer perceptron, the 3D information of the object and the quality score of the target are predicted based on the image feature F;

[0011] (5) Using the pseudo target O′ with quality scores obtained in step (2) and the real target O as supervision information, and the prediction result of step (4) to calculate the loss function;

[0012] (6) Training deep networks and multi-layer perceptrons through gradient backpropagation;

[0013] (7) After the deep network and multi-layer perceptron are trained, the image data to be detected is passed into the deep network and multi-layer perceptron in turn to predict the 3D information of the object and the quality score of the predicted target.

[0014] In step (1), the pseudo target O′ is generated as follows:

[0015] Define the coordinates of the camera as (C X ,C Y ,C Z ), the default coordinate is the origin of the world coordinate system; the real target O(X,Y,Z,H,W,L,θ) is based on the center point (X,Y,Z) and moves along the light (XC X ) / f X =(YC Y ) / f Y =ZC Z , forward and backward displacement Δz·Z;

[0016] The coordinates of the pseudo target O′ are set to ((1+Δz)·X+Δz·f X ·C Z ,(1+Δz)·Y+Δz·f Y ,(1+Δz)·Z,H,W,L,θ).

[0017] Preferably, the value of Δz is: Δz = {±8%, ±4%}.

[0018] Alternatively, in step (2), there are two specific ways to calculate the quality score of the pseudo target O′. The first way is:

[0019] Based on the depth offset value, a linear quality score strategy is designed, and the formula is:

[0020]

[0021] Among them, Δz is the offset value of the depth, Z is the depth of the true target, and c is a hyperparameter.

[0022] The second method is:

[0023] Based on the IoU value between the projection boxes of the pseudo target O′ and the real target O on the 2D image, the IoU quality score strategy is designed, and the formula is:

[0024] IoU Label Score=IoU(B gt ,B pseudo )

[0025] Among them, B gt is the outer bounding box of the original real target projected on the 2D plane; B pseudo It is the outer bounding box of the pseudo target projected on the 2D plane.

[0026] The deep network adopts the DLA-34 network, which consists of an iterative deep aggregation module IDA and a hierarchical deep aggregation module HAD; the IDA module is used to refine the resolution and aggregation scale step by step, and fuse them in terms of resolution and scale; the HDA module aggregates each level into different levels of representation by integrating its own tree-like connection structure, and fuses features on various modules and channels.

[0027] In step (4), the multilayer perceptron includes a convolutional layer, a normalization layer, a nonlinear layer and a convolutional layer connected in sequence;

[0028] Among them, the convolution kernel size of the first convolution layer is 3×3, and the padding size is 1; the normalization layer uses batch normalization BatchNorm, and the nonlinear layer uses linear rectification ReLU; the convolution kernel size of the second convolution layer is 1×1.

[0029] In step (5), a variety of different loss functions can be used, and the loss function can be used with any monocular object detection method. In this invention, the results of PatchNet or Pseudo-LiDAR based on depth maps and RTM3D, Ground-aware or GUPNet based on direct regression are presented.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. This invention points out a phenomenon that has been ignored by previous methods, that is, multiple 3D objects may have the same (similar) 2D projection information, such as Figure 2 As shown in Figure 2. This phenomenon is likely to occur in real scenarios, which will confuse the network during training and thus limit the detection performance.

[0032] 2. This invention solves the problem that “multiple 3D objects may have the same (similar) 2D projection information”. By adopting deep learning methods, the training samples are expanded, allowing the model to learn more general rules.

[0033] 3. The present invention can be added as a plug-in to other baseline algorithms to achieve better detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a 3D target detection method based on a monocular camera of the present invention;

[0035] Figure 2 It is a theoretical schematic diagram of the present invention;

[0036] Figure 3 This is a structural diagram of the multi-layer perceptron in the present invention. DETAILED DESCRIPTION

[0037] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0038] like Figure 1 As shown in FIG, a 3D target detection method based on a monocular camera includes the following steps:

[0039] S01, for each original target O(X, Y, Z, H, W, L, θ) in an image, calculate the coordinates of the pseudo target O′.

[0040] The specific process of generating the pseudo target O′ is as follows: the target O(X, Y, Z, H, W, L, θ) is taken as the reference and along the light (XC X ) / f X =(YC Y ) / f Y =ZC Z (Where the coordinates of the camera are (C X ,C Y ,C Z ), which defaults to the origin of the world coordinate system), and is offset forward (backward) by Δz·Z.

[0041] Since we move on the extended line of the light, for X, we have

[0042] Z=(XC X ) / f X +C Z

[0043] and

[0044] (1+Δz)·Z=(X′-C X ) / f X +C Z

[0045] Similarly, for Y, we have

[0046] Z=(YC Y ) / f Y +C Z

[0047] and

[0048] (1+Δz)·Z=(Y′-C Y ) / f Y +C Z

[0049] Therefore, the coordinates of the pseudo target O′ are ((1+Δz)·X-Δz·C X +Δz·f X ·CZ ,(1+Δz)·Y-Δz·C Y +Δz·f Y ,(1+Δz)·Z,H,W,L,θ).

[0050] When the camera position is the default coordinate, the coordinate of the pseudo target O′ can be simplified to ((1+Δz)·X+Δz·f X ·C Z ,(1+Δz)·Y+Δz·f Y ,(1+Δz)·Z,H,W,L,θ). The recommended value of Δz is: Δz={±8%,±4%} (this is the optimal value tested on the KITTI dataset). With this value of Δz, the pseudo target is generated at a ratio of 1:4 to the original target.

[0051] S02, calculating the quality score of the generated pseudo target O′.

[0052] a) A linear quality score strategy based on the depth offset value can be used:

[0053]

[0054] In the present invention, the value of c is 4. This is determined based on the numerical range of the depth Z and the value of Δz in the data set, so that the value range of the second item can fall relatively evenly within [0, 1].

[0055] b) You can use the IOU quality score strategy of projecting the box IOU value on the 2D image:

[0056] IoU Label Score=IoU(B gt ,B pseudo )

[0057] Among them, B gt is the outer bounding box of the original target O projected on the 2D plane; B pseudo It is the outer bounding box of the pseudo target O′ projected on the 2D plane.

[0058] S03: Use a deep neural network to extract features F for each image in the dataset. The DLA-34 backbone network is used here. DLA fuses multi-scale information, capturing information about objects at different scales. To enable faster network convergence, the present invention initializes the network to parameters pre-trained on ImageNet.

[0059] S04, the multi-layer perceptron predicts the 3D information of the target and the quality score of the target based on the image feature F. Figure 3 As shown in Figure 2, two convolutional layers, a normalization layer, and a nonlinear layer are used here.

[0060] In S05, the pseudo target O′ with quality score obtained in S02 and the original target O are used as supervision information to train the deep network and multi-layer perceptron.

[0061] S06, apply the trained monocular 3D object detection system to actual scenarios.

[0062] To verify the effectiveness of this invention, we conducted quality experiments on the KITTI test set. The KITTI dataset, jointly developed by the Karlsruhe Institute of Technology and Toyota Research Institute of America (ATI) in Germany, is a widely used international dataset for evaluating computer vision algorithms in autonomous driving scenarios. KITTI contains real-world image data captured in multiple scenarios, including urban areas, rural areas, and highways, covering various degrees of occlusion and truncation. Each image contains up to 15 vehicles and 30 pedestrians. The entire dataset consists of 389 pairs of stereo images and optical flow maps, 39.2 km of visual odometry sequences, and over 200,000 images with 3D annotated objects.

[0063] For 3D object detection, the dataset categories are broken down into cars, vans, trucks, pedestrians, (seated) pedestrians, cyclists, trams, and others.

[0064] This example was compared with the best published methods on a test set. The monocular object detector used here is GUPNet. The comparison results are shown in Table 1. The average precision of each method was compared under two different view types: 3D view and BEV view. In Table 1, the top column shows the currently published methods; the bottom column shows the effectiveness of the present invention. It is not difficult to see that the present invention achieves the best results in all indicators, and the present invention (OBMO) has a higher detection accuracy than other methods.

[0065] Table 1

[0066]

[0067] Table 2 shows the detection performance of the present invention as a plug-in combined with other mainstream monocular object detectors on the validation set. The experimental results show that the present invention is effective on different detectors.

[0068] Table 2

[0069]

[0070] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 3D target detection method based on a monocular camera, characterized in that: The steps include: (1) According to the imaging rules of the monocular camera, for each original real target O, a series of pseudo targets O′ with similar imaging are generated proportionally to form a new target detection dataset; (2) According to the designed quality score function, the quality score of the pseudo target O′ is calculated to indicate the similarity between the pseudo target and its true target, where the quality score of the true target O is assumed to be 1.0 by default; (3) Use a deep network to extract the features F of each image in the image dataset; (4) Using a multi-layer perceptron, the 3D information of the object and the quality score of the target are predicted based on the image feature F; (5) Using the pseudo target O′ with quality scores obtained in step (2) and the real target O as supervision information, and the prediction result of step (4) to calculate the loss function; (6) Training deep networks and multi-layer perceptrons through gradient backpropagation; (7) After the deep network and multi-layer perceptron are trained, the image data to be detected is passed into the deep network and multi-layer perceptron in turn to predict the 3D information of the object and the quality score of the predicted target.

2. The 3D target detection method based on a monocular camera according to claim 1, characterized in that: In step (1), the pseudo target O′ is generated as follows: Define the coordinates of the camera as (C X , C Y , C Z ), the default coordinate is the origin of the world coordinate system; the real target O (X, Y, Z, H, W, L, θ), with the center point (X, Y, Z) as the reference, along the light (XC X ) / f X =(YC Y ) / f Y =ZC Z , forward and backward displacement Δz·Z; The coordinates of the pseudo target O′ are set to ((1+Δz)·X+Δz·f X ·C Z , (1+Δz)·Y+Δz·f Y , (1+Δz)·Z, H, W, L, θ).

3. The 3D target detection method based on a monocular camera according to claim 2, characterized in that: The value of Δz is: Δz = {±8%, ±4%}.

4. The 3D target detection method based on a monocular camera according to claim 1, characterized in that: In step (2), the quality score of the pseudo target O′ is calculated as follows: Based on the depth offset value, a linear quality score strategy is designed, and the formula is: Among them, Δz is the offset value of the depth, Z is the depth of the true target, and c is a hyperparameter.

5. The 3D target detection method based on a monocular camera according to claim 1, wherein: In step (2), the quality score of the pseudo target O′ is calculated as follows: Based on the IoU value between the projection boxes of the pseudo target O′ and the real target O on the 2D image, the IoU quality score strategy is designed, and the formula is: IoU Label Score=IoU(B gt ,B pseudo ) Among them, B gt is the outer bounding box of the original real target projected on the 2D plane; B pseudo It is the outer bounding box of the pseudo target projected on the 2D plane.

6. The 3D target detection method based on a monocular camera according to claim 1, characterized in that: In step (3), the deep network adopts the DLA-34 network, which is composed of an iterative deep aggregation module IDA and a hierarchical deep aggregation module HAD; the IDA module is used to refine the resolution and aggregation scale step by step, and fuse them in terms of resolution and scale; the HDA module aggregates each level into different levels of representation by integrating its own tree-like connection structure, and fuses features on various modules and channels.

7. The 3D target detection method based on a monocular camera according to claim 1, characterized in that: In step (4), the multilayer perceptron includes a convolutional layer, a normalization layer, a nonlinear layer and a convolutional layer connected in sequence; Among them, the convolution kernel size of the first convolution layer is 3×3, and the padding size is 1; the normalization layer uses batch normalization BatchNorm, and the nonlinear layer uses linear rectification ReLU; the convolution kernel size of the second convolution layer is 1×1.

Images