A monocular 3D target detection method based on deep dynamic center difference convolution

Abstract

The application discloses a monocular 3D target detection method based on deep dynamic center difference convolution, and belongs to the technical field of automatic driving. The application comprises the following steps: acquiring an image, determining an anchor frame pre-design, inputting the preprocessed image into a main feature extraction network, extracting semantic information, performing classification branch and regression branch on the output feature map, performing post-processing operation on the category information and regression information, and obtaining the category and 3D bounding box of the target in the RGB image. In the depth estimation of monocular 3D target detection, the deep dynamic center difference convolution is used to replace the traditional convolution, the deep dynamic operation makes the network consider the depth information, the local size information and the geometric information of each sample at the same time, the center difference further improves the refinement degree of the network at each pixel, and only simple calculation is used without occupying too much memory. The accuracy of depth estimation is improved, and the precision of monocular 3D target detection is finally improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a monocular 3D target detection method based on deep dynamic central difference convolution. Background Technology

[0002] Accurate 3D information is essential for autonomous driving. Previous 3D object detection methods have mostly relied on LiDAR (Light Detection and Ranging) devices, which improve detection accuracy by generating precise 3D point cloud depth information. However, due to the high cost and short lifespan of LiDAR sensors, there is a growing demand for cheaper alternatives, such as stereo cameras and monocular camera methods. Monocular 3D object detection offers a more economical solution compared to traditional LiDAR-dependent methods. Compared to stereo cameras, monocular cameras have a larger visual range and stronger robustness to extrinsic parameters. Therefore, even in the absence of depth information, monocular 3D object detection remains a hot research area.

[0003] Existing monocular 3D object detection methods can be categorized into three types based on depth information estimation methods: RGB image-based methods, combined depth map and image-based methods, and pseudo-LiDAR methods. However, RGB image-based methods suffer from significant differences in depth information between the background and objects in driving scenarios; depth maps and image-based methods do not share parameters, easily leading to the loss of some network information and affecting performance; and pseudo-LiDAR methods cannot effectively utilize the high-level semantic information in RGB images. Since point clouds only provide spatial information and not semantic information, this method may produce erroneous detections, misdetecting road signs, power boxes, etc., as vehicles.

[0004] Therefore, improving the performance and accuracy of depth estimation in monocular 3D target detection is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a monocular 3D target detection method based on depth dynamic central difference convolution, which improves the accuracy of depth estimation and ultimately enhances the results of monocular 3D target detection. The method includes the following steps:

[0006] Step I. Acquire the image, determine the anchor box pre-design, and input the pre-processed image into the backbone feature extraction network;

[0007] Step II. Extract semantic information from the backbone feature extraction network and output feature maps;

[0008] Step III. Perform classification and regression branch predictions on the feature map to obtain category and regression information;

[0009] Step IV. Perform post-processing operations on the category and regression information to obtain the category and 3D bounding box of the target in the RGB image.

[0010] Preferably or optionally, in the anchor frame pre-design described in step I, 32 anchor frames are pre-set, with each 2D anchor frame corresponding to a 3D anchor frame. The preset values ​​of each parameter of the anchor frame are selected from the traversal of the KITTI dataset. The mean and variance of the sine value sinα and the cosine value cosα,z of the 3D target observation angle are calculated. The IoU value between the ground real 3D frame and its projection onto the two-dimensional plane is calculated. The 3D frame is assigned to the anchor frame with IoU ≥ 0.5.

[0011] Preferably or optionally, the transformation formula between the image coordinate system and the camera coordinate system is: Where F represents the camera's intrinsic parameters, [x, y, z]. 3D Let [x, y] represent the horizontal position, height, and depth of the 3D coordinates in camera coordinates. 2D For the projected coordinates in the 2D image, z 3D For true depth.

[0012] Preferably or optionally, the size of the output feature map in step II is 18×80, and the 16x downsampled feature map output by the backbone network is selected as the output map.

[0013] Preferably or optionally, the classification branch in step III uses two convolutional modules to predict the feature map input to the backbone feature extraction network, resulting in an output tensor C(B, W, H, K*num_anchors), where B is the batch size, W and H are the size of the feature map, K is the number of classes, and num_anchors is the number of anchors per pixel.

[0014] Preferably or optionally, the regression branch in step III uses one deep dynamic center difference convolutional module and one convolutional module to predict the feature map input to the backbone feature extraction network, and obtains the output tensor R(B,H,W,12*num_anchors), where B is the batch size, W and H are the size of the feature map, and num_anchors is the number of anchors per pixel.

[0015] Preferably or optionally, the post-processing operation in step IV involves post-processing the classification tensor C and the regression tensor R using a non-maximum suppression algorithm to obtain the final target detection result.

[0016] Preferably or optionally, after calculating the feature map from the backbone, a depth map is output based on the ground hypothesis; the depth map is then calculated using a depth dynamic central difference convolution module. The depth dynamic part is based on an attention mechanism and expert knowledge, equivalent to a linear combination of multiple ordinary convolutions, learning specific parameters for each example. The input depth map (N, h, w, c_in) is processed by the attention module to obtain a tensor (N, num_experts), where num_experts is the number of experts, with a default value of 8. The convolution kernels are weighted to obtain a tensor (N, h*w*c_in*c_out), resulting in the desired dynamic convolution kernel. Central difference is then utilized to emphasize the difference between local objects and the background, without introducing excessive computation or memory usage. Unique convolution kernels are implemented on different pixels, improving the accuracy of depth estimation.

[0017] Preferably or optionally, the formula for the depth dynamic central difference convolution is: Where, output cd 'W is the output of the center difference convolution. DD (x n ) represents the depth-dynamically derived convolution kernel, x c With the center position, x n σ represents the pixel values, and σ is a hyperparameter of 0.7.

[0018] Beneficial Effects: This invention provides a monocular 3D object detection method based on depth dynamic central difference convolution. An end-to-end monocular 3D object detection network is designed based on the depth dynamic central difference convolution module, improving the accuracy of depth information estimation. The depth dynamic convolution module is a depth map processing module that improves the accuracy of depth map estimation based on RGB image methods. This module utilizes the depth information of surrounding pixels to expand the semantics of the depth map, alleviating the shortcomings of traditional two-dimensional convolution. The depth dynamic part uses an attention mechanism to learn each sample, and then each sample generates a unique convolution kernel, utilizing the depth information around all center pixels to learn the geometric information of each image. The central difference part highlights the depth difference between the target and the background, focusing more on pixel-level object localization. Therefore, the depth dynamic central difference convolution module not only solves the problems of scale sensitivity and inability to capture local objects in two-dimensional convolution, but also obtains high-level semantic information from RGB images. Attached Figure Description

[0019] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0020] Figure 1 This is a flowchart of the detection steps of the present invention;

[0021] Figure 2 This is a schematic diagram of the depth dynamic convolution module of the present invention;

[0022] Figure 3 This is a comparison of the visualization results of ordinary convolution and convolution with depth dynamic center difference. Detailed Implementation

[0023] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0024] Example 1

[0025] like Figure 1 As shown, the present invention includes the following steps:

[0026] Step S1: Determine the pre-design of the anchor box, preprocess the input image, and input it into the backbone feature extraction network ResNet-101.

[0027] The KITTI object detection dataset was selected, comprising 7481 frames from the training set and 7518 frames from the test set, containing a total of 80,256 labeled objects, primarily cars, bicycles, pedestrians, and trucks. Based on object occlusion and truncation levels, the ground truth labels were categorized into three levels: easy, medium, and hard. Following the split1 method, the training set was divided into 3712 training images and 3769 validation images. All input images were cropped to 288×1280 resolution, and random horizontal mirroring was used as data augmentation during training.

[0028] The default value is 32 anchor boxes, and each 2D anchor box on the image corresponds to one 3D anchor box. Each anchor box has 12 parameters, including x, y, and w. 2d ,h 2d For a corresponding 2D bounding box, x and y correspond to the center position of the 2D bounding box, and w 2d ,h 2d The corresponding width and height of the 2D bounding box; parameter c x ,c y ,z corresponds to the center of the 3D anchor box, c x ,c y The projection of the center of the 3D object onto the image is given by z, where z is the depth value of the center of the 3D object; the parameters w, h, and l are the length, width, and height of the 3D bounding box, in meters; and the parameters sinα and cosα are the sine and cosine values ​​of the 3D target observation angle.

[0029] The preset parameters of the anchor boxes are based on a traversal of the KITTI dataset, recording the data of all ground truth 3D bounding boxes in the dataset, where the mean and variance of sinα, cosα, and z are calculated. Then, all ground truth 3D bounding boxes are projected onto a 2D plane, their IoU values ​​are calculated, and the corresponding 3D bounding boxes are assigned to anchor boxes with IoU ≥ 0.5.

[0030] Step S2: Select the 16x downsampled feature map output by the backbone network ResNet-101 as the output result.

[0031] Step S3: The feature map output by the backbone network is input into the classification branch and the regression branch. The classification branch consists of two convolutional modules, and the regression branch consists of a deep dynamic center convolutional module (DDCDC) and a convolutional module, which respectively obtain the category information and the regression information.

[0032] The hardware used is an Nvidia 2080Ti GPU, the batch size is 8, and the training memory usage is about 10500MB.

[0033] Step S4: Post-process the predicted classification tensor C and regression tensor R obtained from the two branches, and use the non-maximum suppression algorithm to obtain the final target detection result.

[0034] The loss function formula for this network is: L = L cls +L reg The classification loss function L cls It is the focal loss function, the regression loss function L. reg It is the smooth-L1 loss function.

[0035] like Figure 2 As shown, the specific principle of the depthwise dynamic convolution module is as follows:

[0036] In traditional 2D convolution methods, all samples have the same convolution parameters. To improve model capacity and performance, it is inevitable to increase the number of model parameters and channels, leading to a surge in computational complexity and increased difficulty in deploying the network model. A deep dynamic convolution module based on an attention mechanism is designed, incorporating expert knowledge. This is equivalent to a linear combination of multiple ordinary convolutions, learning the specific convolution kernel parameters for each sample.

[0037] The formula for depthwise dynamic convolution is as follows:

[0038] Where α i =t i (x) is learned from the attention module, k is the number of experts (default value is 8), and w i W is the initial convolution value. DDIt is the weight of the final output.

[0039] The formula for the attention mechanism is as follows:

[0040] t(x) = sigmoid(fc(flatten(AdaptiveAvgPool2d(x)))). Obtaining t(x) from the attention module can be divided into four steps: adaptive average pooling, flattening layer, fully connected layer, and sigmoid normalization.

[0041] Deep dynamics only requires one convolution computation, and the convolution kernel for each sample is a linear combination of k experts. The network capacity can be increased by increasing the value of k, and only one additional multiplication and addition is needed.

[0042] Depth-dynamic convolutions use different kernels for each sample. To further emphasize the difference between local objects and the background, we designed a central difference convolution without introducing too much computation and memory usage, and implemented unique kernels on different pixels to improve performance. Central difference convolutions can better represent center-oriented gradients. In depth maps, there is a clear difference between objects and the background. Applying central difference can emphasize the edge depth of objects to highlight them, and it also compensates for the problem of not considering different kernels for each pixel. A simple calculation gives a specific kernel for each pixel.

[0043] The central difference formula is as follows:

[0044] output cd This represents the output convolution after center difference, where n represents the output feature at the current input position, and W... DD It is the weight (obtained after depth dynamics), x c It is the central position.

[0045] Experiments revealed that using only central difference convolution is not optimal, and combining it with traditional convolution is possible. Since central difference cannot effectively distinguish approximate semantic information, adding a regular convolution can address this issue.

[0046] The formula for central difference convolution combined with ordinary convolution is:

[0047] The hyperparameter σ∈[0,1] balances the contributions of ordinary semantic information and gradient information.

[0048] The higher the σ value, the more important the central differential gradient information becomes. A value of 1 indicates central difference convolution, while a value of 0 indicates ordinary convolution. Experiments showed that a value of 0.7 yielded the best results.

[0049] Further optimization of the central difference formula:

[0050] It is easier to implement at the code level.

[0051] Central difference learns the depth features of local objects from the depth information of surrounding pixels. It enhances the feature representation capability of the network without requiring any additional parameters, solving the problem of pixel-level depth dynamic convolution.

[0052] Table 1 below shows the performance comparison of this method with other algorithms tested on the KITTI split1 dataset. Experiments were evaluated at IoU (Intersection over Union) thresholds of 0.7 and 0.5.

[0053] Table 1

[0054]

[0055] The evaluation metric is the precision-recall curve (AP) (with two IoU thresholds, 0.7 and 0.5). AP 3D The AP value represents the accuracy of the 3D bounding box detection, calculated using the AP|R40 standard. A higher AP value indicates more accurate target detection. The method of this invention shows the most significant improvement in simple categories, with improvements also seen in normal and hard difficulty levels, demonstrating that the method effectively improves the accuracy of target detection.

[0056] like Figure 3 As shown in Figure A, the visualization result of the network after ordinary convolution is shown in Figure B, and the result after introducing depth dynamic center difference convolution is shown in Figure B, which shows a significant performance improvement.

[0057] This invention proposes a monocular 3D object detection method that incorporates surrounding depth information into the 3D object detection process. Since methods for acquiring 3D information using ordinary convolution are limited, a dynamic convolution module is designed so that all samples have different convolution kernels. Combined with center difference calculation, the depth difference between the background and the target can be emphasized, down to the pixel level. This improves the accuracy of depth information estimation, ultimately enhancing the performance of 3D object detection.

[0058] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present invention will not describe the various possible combinations separately.

Claims

1. A monocular 3D target detection method based on depth dynamic central difference convolution, characterized in that, Includes the following steps: Step I. Acquire images, determine anchor box pre-design, preprocess the input images, and then input them into the backbone feature extraction network; Step II. Extract semantic information from the backbone feature extraction network and output feature maps; Step III. Predict classification and regression branches on the feature map to obtain category and regression information; The regression branch uses one deep dynamic center difference convolutional module and one convolutional module to predict the feature map input to the backbone feature extraction network, and obtains the output tensor R(B,H,W,12*num_anchors), where B is the batch size, W and H are the size of the feature map, and num_anchors is the number of anchors per pixel. After the feature map is computed from the backbone, a depth map is output based on the ground assumption; the depth map is computed using the depth dynamic center difference convolution module. Step IV. Perform post-processing operations on the category and regression information to obtain the category and 3D bounding box of the target in the RGB image; The post-processing operation described in step IV involves post-processing the classification tensor C and the regression tensor R using a non-maximum suppression algorithm to obtain the final target detection result. The formula for the depth dynamic central difference convolution is: , where output cd ’ W is the output of the center difference convolution. DD (x) n ) represents the depth-dynamically derived convolution kernel, x c With the center position, x n Here are the values ​​for each pixel, and σ is a hyperparameter of 0.

7. ;in It is learned from the attention module, and k is the number of experts (default value is 8). These are the initial convolution values. These are the weights of the final output; The process of obtaining t(x) from the attention module can be divided into four steps: adaptive average pooling, flattening layer, fully connected layer, and sigmoid normalization.

2. The monocular 3D target detection method based on depth dynamic central difference convolution according to claim 1, characterized in that, In the anchor frame pre-design described in step I, 32 anchor frames are pre-set, with each 2D anchor frame corresponding to a 3D anchor frame. The preset values ​​of each parameter of the anchor frame are selected from the traversal of the KITTI dataset. The mean and variance of the sine value sinα and the cosine value cosα of the 3D target observation angle are calculated, where z is the depth value of the center of the 3D object. The IoU value between the ground real 3D frame and its projection onto the 2D plane is calculated, and the 3D frame is assigned to the anchor frame with IoU ≥ 0.

5.

3. The monocular 3D target detection method based on depth dynamic central difference convolution according to claim 2, characterized in that, When acquiring the image in step I, the transformation formula between the image coordinate system and the camera coordinate system is: Where F represents the camera's intrinsic parameters, [x, y, z]. 3D Let [x, y] represent the horizontal position, height, and depth of the 3D coordinates in camera coordinates. 2D z represents the projected coordinates in the 2D image; 3D For true depth.

4. The monocular 3D target detection method based on depth dynamic central difference convolution according to claim 1, characterized in that, In step II, the size of the output feature map is 18×80, and the 16x downsampled feature map output by the backbone network is selected as the output map.

5. The monocular 3D target detection method based on depth dynamic central difference convolution according to claim 1, characterized in that, In step III, the classification branch uses two convolutional modules to predict the feature map input to the backbone feature extraction network, resulting in an output tensor C(B, W, H, K*num_anchors), where B is the batch size, W and H are the size of the feature map, K is the number of classes, and num_anchors is the number of anchors per pixel.

Images