Anchor-guided 3D target detection method and device for autonomous driving

By introducing an anchor guide method of sparse anchor points and anchor boxes in binocular 3D object detection, combining key point generation network and stereo regressor, 3D bounding box detection is optimized, and the calculation speed problem caused by excessive anchor points is solved. Achieve efficient and high-precision 3D object detection.

CN113989758BActive Publication Date: 2024-11-01AUTOMOBILE RES INST OF TSINGHUA UNIV IN SUZHOU XIANGCHENG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111250715.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-26
Publication Date
2024-11-01
Estimated Expiration
2041-10-26

AI Technical Summary

Technical Problem

At this stage, there are too many anchor points in the anchor-based 3D object detection algorithm, which leads to the problem of slow calculation speed.

Method used

A method of anchor-guided 3D object detection based on Stereo RCNN is proposed. By generating sparse anchor points and anchor boxes, combining key points to generate networks and stereo regressors, the detection process of 3D bounding boxes is optimized, and the instance parallax and instance mask are carried out. Pixel-level optimization.

Benefits of technology

It improves computing efficiency, reduces computing cost, enhances the accuracy of 3D bounding boxes, and shows good generalization and robustness in practical applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113989758B_ABST
    Figure CN113989758B_ABST
Patent Text Reader

Abstract

The present invention proposes an anchor-guided 3D target detection method and device for autonomous driving. In the first stage, the left and right images are input to generate corresponding probability maps to generate sparse anchor points and sparse anchor boxes, and then the left and right anchors are used as a whole to generate 2D pre-selected boxes. In the second stage, the key point generation network uses the sparse anchor point information to generate a key point heat map, and combines it with a stereo regressor to generate a 3D pre-selected box. In order to solve the problem of pixel-level information loss in the original image after convolution, the instance segmentation mask generated by Mask Branch is combined with instance-level disparity estimation for pixel-level optimization. Experiments show that the present invention can reduce the amount of calculation while maintaining a high recall rate. Compared with existing methods, it is faster, more robust, and has stronger generalization ability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to computer technology, and in particular to the field of autonomous driving, and specifically discloses an anchor-guided 3D target detection method and device for autonomous driving. Background Art

[0002] As one of the foundations of autonomous driving, object detection has been a hot topic in recent years. 2D object detection guided by Faster R-CNN, Mask R-CNN, SSD and YOLO series has achieved a high level in both accuracy and detection speed. However, 2D object detection alone still cannot meet the needs of autonomous driving, so 3D object detection came into being. 3D object detection can be vigorously developed in fields including visual perception, autonomous driving and motion prediction because it has information that is crucial for autonomous driving, such as the spatial position and distance of objects that 2D object detection cannot provide. However, powerful 3D detectors at this stage are heavily dependent on data information provided by radar, and the price of high-precision radar is very expensive, which greatly limits the development of object detection. In contrast, stereo cameras have high cost-effectiveness and practicality, which makes them attract more and more attention in many complex application scenarios at this stage. Stereo cameras determine the perception range by focal length and baseline, so they can ensure high depth accuracy for objects with huge differences.

[0003] At present, the mainstream 3D object detection methods are mainly divided into four categories: methods based on monocular images, methods based on stereo images, methods based on image and point cloud fusion, and methods based on raw point clouds. Among the methods based on monocular images, the M3D-RPN algorithm proposed by G. Brazil et al. improves the performance of 3D scenes by designing depth-aware convolutional layers, and uses the geometric relationship between 2D and 3D perspective images to generate convolutional features in the image space. Z. Junning et al. merged the network structure with perspective projection constraints and proposed a monocular 3D object detection algorithm based on perspective projection. The 3D object boundary model is solved through the transformation relationship between the real world, the camera and the target. Among the methods based on stereo images, the Stereo RCNN algorithm proposed by P. Li et al. predicts the left and right 2D boxes through the proposed Stereo RPN module, and predicts the 3D bounding box by combining the additional branches to predict key points, viewpoints and object dimensions. FX Dong et al. proposed a new method for establishing joint calibration for binocular stereo vision and 3D laser scanning, which uses a binocular stereo camera to extract 3D coordinate information and a laser ranging radar to obtain the center point coordinates. Methods based on image and point cloud fusion: The SPLATNet algorithm proposed by H. Su et al. first transforms the point cloud from the Euclidean matrix to another space that can efficiently organize sparse and disordered data through a transformation matrix, then extracts and fuses features, and finally regresses the 3D box through a deep network. The F-PointNet algorithm of CRQi et al. uses a pre-trained 2D detector to identify the 2D box and category of objects from RGB images, and uses PointNet to obtain the object mask and estimate the 3D bounding box based on camera parameters and 3D cone point clouds. ZSWu et al. proposed an algorithm based on laser point cloud and image information fusion for optimal matching through the Hungarian algorithm, which solves the defect that monocular 3D detection cannot constrain the 3D center. Methods based on raw point clouds: PIXOR proposed by B. Yang et al. obtains a 2D bird's eye view map (Bird's Eye View Map, BEV) through point clouds, and then performs object detection and positioning through RetinaNet. STD proposed by Z. Yang et al. first uses the raw point cloud as input, and then generates accurate proposals by seeding each point with a spherical anchor box. The method proposed by Z.Yi et al. only uses the location information of the radar point cloud to voxelize and construct a bird's-eye view, and then uses the residual structure to extract multi-level features containing high-level semantic features to output a dense feature map. The current anchor-based stereo 3D object detection algorithm has a large number of anchor points, which affects the online calculation speed. Summary of the invention

[0004] Aiming at the problem that the current anchor-based binocular 3D object detection algorithm has a large number of anchor points, which affects the online calculation speed, this paper proposes an anchor-guided 3D object detection algorithm FGASRCNN based on Stereo RCNN. Compared with uniform anchor point detectors such as Stereo RCNN, FGAS RCNN can adjust the distribution and size of anchor points according to the size and position of the object.

[0005] The present invention adopts the following technical solution:

[0006] An anchor-guided 3D target detection method for autonomous driving comprises: in the first stage, inputting left and right images, and then generating corresponding foreground position probability maps respectively to generate sparse anchor points and sparse anchor frames, and then generating 2D pre-selected frames by taking the left and right anchors as a whole to obtain a new feature map; in the second stage, a key point generation network generates a key point heat map using sparse anchor point information, and combines it with a stereo regressor to generate a 3D pre-selected frame; in the third stage, pixel-level optimization is performed by combining instance segmentation masks generated by Mask Branch with instance-level disparity estimation to complete 3D target detection.

[0007] In this invention, the input left and right images are called feature maps, and the maximum coverage IOU (Intersection over Union) refers to the highest coverage of the predicted shape with the closest ground truth (GT) bounding box; in the first stage, a 1×1 convolution is applied to the input feature map to generate a probability map of the same size as the feature map P To obtain the objectivity score, it is converted into a probability value through the Sigmod function. Formula (1) represents the probability of the detected object at that position.

[0008]

[0009] Where: x a ,y a ) corresponds to the coordinates on the input feature map, S Represents the step size of the feature map;

[0010] The points with probability higher than the preset threshold are regarded as the activity area where the object may exist. Then, the shape of the activity area where the object may exist is predicted based on the feature map and the probability map, and the shape with the maximum coverage is selected as the predicted anchor box.

[0011] Furthermore, according to the feature pyramid architecture, anchor points are collected in the multi-scale feature map, the basic anchor points are set, and the object type and frame offset are regressed to the input position using 3×3 convolution to reduce the channel and two fully connected layers, and then the deformable convolution with offset is applied to the original feature map to obtain a new feature map; the new feature maps of the left and right outputs are connected at each scale, and the concatenated features are fed into the Stereo RPN network to obtain an accurate detection frame. Specifically, the left and right feature maps are concatenated and input into the guided stereo RPN network, and the corresponding detection frames are matched by the maximum coverage. In the present invention, when sampling the region of interest (ROI), only the left and right detection frames at the same anchor position will be used as the foreground if they correspond to each other at the same time and the maximum coverage is greater than 0.5; if the detection frames on both sides correspond to the same one and the maximum coverage is greater than 0.8, then the two detection frames are considered to predict the same target object.

[0012] Preferably, the left and right feature maps are concatenated and input into the guided stereo RPN network, and the 2D bounding box is regressed using the six coordinate parameterizations in formula (2), and then the detection box is screened again by using non-maximum suppression (NMS) on the left and right ROI regions to obtain a 2D pre-selected box;

[0013] ;

[0014] Where: x , y , w , h Represents the horizontal and vertical coordinates, width and height of the prediction box; x , x ’ , x a Represent the horizontal coordinates of the left prediction box, the right prediction box, and the anchor box respectively.

[0015] In the present invention, anchor points are only assigned to observed objects; compared with the conventional method of relying on uniform distribution of anchor points and predefined multi-scale boxes, the difference of the present invention is that anchor points are only assigned to observed objects instead of being uniformly distributed on the image, and features can be adjusted according to the geometric shape of the target.

[0016] In the present invention, in the second stage, the key point generation network only predicts the key points of the right image, and only uses the new feature map generated by the multi-level anchor scheme as input. Each scale new feature map generated in the first stage is upsampled three times through three bilinear interpolations, and a 1×1 convolution layer is added after each upsampling to reduce the channel. Before upsampling, the corresponding feature maps are cascaded, and the obtained multi-scale feature maps are adjusted to the maximum scale, and the soft weight ∂ is generated by the Softmax function operation. The importance of each scale can be intuitively seen through the generated soft weight, and then the scale space score map S can be obtained through the linear weighting formula (4) scale ,

[0017]

[0018] The present invention defines the 2D bounding box center point heat map as ,in H , W Represents the width and height of the input image, C represents the number of object categories, S represents the step length; the heat map of the 9 perspective points projected from the 3D bounding box vertices and the center point is defined as ; The local offset regressed from the 2D bounding box center is defined as ; will be closest to V C The 9 key points of the coordinates are considered to be a set of key point coordinates of the same object, and the 3D bounding box of the object is restored through the 18 constraints of these 9 key points.

[0019] In the present invention, in the third stage, the new feature maps (left and right) of the first stage are respectively subjected to the ROI Align proposed in Mask RCNN, and then the left and right ROI features are merged and input into two fully connected layers to extract semantic information. The stereo regressor can obtain four outputs including object category, stereo bounding box, object size and viewpoint angle; the car's forward direction and ROI viewing angle are used as the viewpoint regression angle, and [sin β ,cos β ] as a regressor to avoid discontinuity, and then the vehicle direction is obtained by decoupling the relationship between the regression angle and the 3D position;

[0020] The above 3D bounding box can be , 3D box center coordinates and regression size , combined with the given perspective key point information, the corresponding relationship between the 3D box vertices and the 2D box vertices can be derived, and the 3D-2D relationship formula (7) obtained by the projection transformation can be solved by the Gauss-Newton method;

[0021] The present invention transforms the problem of minimizing the reprojection of 3D key points and 2D key points into a nonlinear least squares optimization problem;

[0022] ;

[0023] In the formula: The key point is expressed as , whose dimensions and directions are ; K is the given camera intrinsic matrix, R Represents the rotation angle of the 3D box The present invention takes ;

[0024] When using ROI Align to extract object features from the feature map on the left image, an instance segmentation mask is generated. Then, the 2D bounding box and instance segmentation mask provided above can be used to crop and align the left and right ROIs in the horizontal direction from the full image according to formula (8);

[0025] ;

[0026] Where: b L and b R Represents the normalized coordinates of the left and right bounding boxes of the left 2 bounding boxes respectively ; D i ( p ) represents the predicted instance disparity value;

[0027] By calculating the disparity of all pixels in the mask area and combining it with the baseline B With camera intrinsics f The 3D position and depth value of each pixel in the mask can be calculated. The calculation formula of the 3D position is (9),

[0028] ;

[0029] Where: u c , v c ) represents the camera center pixel position; ( f u , f v ) are the horizontal and vertical focal lengths of the camera respectively;

[0030] The depth calculation formula is:

[0031] ;

[0032] Where: xpl , x pr Represents the pixels p Horizontal coordinates in the left and right borders;

[0033] The total matching samples set by the present invention are the sum of square differences of all pixels in the mask area:

[0034] ;

[0035] Where: Z p Represents the pixels within the mask P The depth difference from the center of the 3D box, I L and I R They represent the 3-channel RGB vectors in the left and right images respectively.

[0036] Preferably, the optimized center depth is obtained by minimizing the total matching cost, and preferably an enumeration method is used to generate the optimal depth. Z Specifically, the 3D box depth value Z estimated in the previous article p Based on this, 40 rough depths are selected at intervals of 0.5 meters, and then the rough depths are enumerated again at intervals of 0.05 meters to obtain the optimal depth. By fixing the alignment depth of all pixels in the mask area, the entire 3D box can be corrected to achieve 3D target detection.

[0037] The present invention discloses a device for anchor-guided 3D target detection for autonomous driving, which is used for the above-mentioned anchor-guided 3D target detection method, and includes an image acquisition module and a data processing module. The data processing module includes a stereo RPN module, a key point generation network module, a stereo regression and 3D box estimation module, and a center depth correction module; specifically, the data processing module is a module for generating corresponding foreground position probability maps for pictures collected by the image acquisition module, a module for generating sparse anchor points and sparse anchor boxes, a module for generating 2D pre-selected boxes by taking left and right anchors as a whole, a module for generating key point heat maps by using sparse anchor point information, a module for generating 3D pre-selected boxes by combining stereo regressor fusion, a module for generating instance segmentation masks by Mask Branch, and a module for performing pixel-level optimization by instance-level disparity estimation; the image acquisition module is a radar sensor and a camera.

[0038] The present invention discloses an electronic device, comprising a processor and a memory, wherein the memory stores instructions executed by the processor, so that the processor can execute the anchor-guided 3D target detection method for autonomous driving.

[0039] The present invention discloses a computer-readable storage medium storing computer instructions, wherein the computer instructions enable a computer to execute the anchor-guided 3D target detection method for autonomous driving.

[0040] The present invention has carried out the following innovations:

[0041] (1) To address the problem that existing anchor point selection methods have large computational complexity and slow speed, a stereo RPN method based on adaptive anchor boxes is proposed. The anchor point selection range is determined by generating foreground position probability maps for the left and right input images respectively, and the anchor point selection range is guided to learn the target object shape that is beneficial to the 2D box. The method of combining sparse anchor points with stereo RPN can effectively save computational costs and improve computational efficiency.

[0042] (2) To address the problem of low 3D bounding box accuracy in stereoscopic 3D object detection methods, a key point generation network based on a pyramid network is proposed. By introducing a feature map of a multi-level anchor scheme that integrates high-level features, the 9 key point constraints of the 3D bounding box are extracted, and the number of negative input samples is reduced, thereby increasing the proportion of positive samples;

[0043] (3) To address the problem of pixel-level information loss in the original image after convolution, a 3D bounding box center depth correction method based on pixel-level instance disparity is proposed. By fusing the pixel-level instance disparity with the instance segmentation mask, the pixel-by-pixel depth within a limited range is estimated to optimize the center depth. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is the FGAS RCNN algorithm framework diagram;

[0045] Figure 2 Generate solutions for anchor points;

[0046] Figure 3 It is a multi-level anchor scheme;

[0047] Figure 4 Assignment for goals;

[0048] Figure 5 It is the upsampling process;

[0049] Figure 6 It is composed of a multi-task detection head;

[0050] Figure 7 For the trimming alignment process;

[0051] Figure 8 is the relationship between parallax error (pixels), depth error (m) and object distance (m);

[0052] Fig. 9The following are the visualization results. From top to bottom, they represent key point prediction, 3D bounding box prediction, and bird's-eye view prediction.

[0053] Fig.10 IoU distribution of RPN, Stereo-RPN, and GS-RPN;

[0054] Fig.11 PR curve for ROI classification;

[0055] Fig.12 RPN proposal (top) and GS-RPN proposal (bottom);

[0056] Fig.13 It is a vehicle-mounted experimental platform;

[0057] Fig.14 is the average accuracy of 3D detection under different standards;

[0058] Fig.15 To visualize the detection results. DETAILED DESCRIPTION

[0059] The present invention designs a method for proposing left and right regions of interest (ROI) using sparse anchors, called FGAS-RCNN. The network architecture is divided into three modules and uses ResNet-50 as the backbone network. Figure 1 As shown in the figure, the Guided Stereo RPN module (GS RPN) is first used to output the probability map of the foreground position of the feature map, and generate sparse anchor points to predict the object shape. Then the corresponding left and right ROI proposals are output based on the predicted anchor point position and anchor box shape. The key point detection network module uses a multi-level anchor scheme to generate a key point heat map and predicts 9 key perspective points including 3D bounding box vertices and center points. The 3D box constraints obtained by these key points are then used to correct the obtained rough 3D box. The final center depth correction module mainly uses the ROI and mask generated by ROIAlign on the left and right feature maps to generate instance segmentation masks, and combines the disparity to calculate the depth value of each pixel to optimize the 3D box center depth. Example

[0060] The FGAS RCNN of the present invention is specifically as follows.

[0061] 1.1 Stereo RPN Module

[0062] The Region Proposal Network (RPN) regresses the object type and box offset of the input position by reducing the channels using 3×3 convolutions and two fully connected layers after feature extraction. Compared with the conventional reliance on uniform distribution of anchor points and predefined multi-scale boxes, the difference of the present invention is that anchor points are only assigned to the observed objects instead of being evenly distributed on the image, and the features can be adjusted according to the geometry of the target. Figure 2 As shown in Figure 2, the method consists of two branches, namely, the anchor point positioning branch and the anchor box prediction branch.

[0063] In the anchor prediction branch, a 1×1 convolution is applied to the input feature map F I In , a probability map of the same size as the feature map is generated P To obtain the objectivity score, it is converted into a probability value through the Sigmod function. Formula (1) represents the probability of the detected object at this position.

[0064]

[0065] Where: x a ,y a ) corresponds to the input left feature map F I The coordinates on S Represents the stride of the feature map.

[0066] The present invention selects points with probabilities higher than a preset threshold as the active areas where objects may exist, which can greatly reduce the area where objects may exist while ensuring the recall rate. F I , and the probability map output by the anchor prediction branch P The best shape for these locations with probabilities above the threshold is predicted, that is, the shape that has the highest coverage with the closest ground truth (GT) bounding box. The present invention selects the shape that has the maximum coverage IOU (Intersection over Union) with the nearest GT box. w a , h a As the predicted anchor box size. Different from conventional methods, the method of the present invention has only one predicted anchor box of the best shape at each position, rather than a set of predefined anchor boxes.

[0067] Since all anchor frames in the present invention change with the position, the present invention adopts the following method: Figure 3The multi-level anchor scheme shown in Figure 1. This scheme collects anchors in multi-scale feature maps according to the Feature Pyramid Network (FPN) architecture, and these anchors can be shared in all scale feature maps. In order to achieve the correspondence between features of different sizes and the corresponding size ranges, this scheme predicts an offset from the output of the anchor box prediction branch based on the basic anchor shape and a 3×3 variable convolution layer. Then the deformable convolution with the offset is applied to the original feature map F I Get the new feature map F I ’ .

[0068] Connect the new feature maps of the left and right outputs at each scale, and feed the concatenated features into the Stereo RPN network to obtain an accurate detection frame. Figure 4 As shown, unlike conventional object targets, the present invention regards the concatenated left and right feature maps as the target of object classification. The present invention concatenates the left and right feature maps and inputs them into the GS RPN network, and matches the corresponding detection boxes through IOU. When performing ROI sampling, only the left and right detection boxes at the same anchor point position will be used as foregrounds if their IOU with the corresponding union GTBox is greater than 0.5. And if the IOU of the detection boxes on both sides with the same union GT Box is greater than 0.8, the present invention believes that the two detection boxes predict the same target object.

[0069] Traditional RPN regressors generally have only four outputs, while GS RPN regressors generally have six outputs. The present invention uses the six coordinate parameterizations in formula 2 to regress the 2D bounding box. Then, non-maximum suppression (NMS) is used on the left and right ROI regions to screen the detection box again.

[0070] ;

[0071] Where: x , y , w , h Represents the horizontal and vertical coordinates, width and height of the prediction box; x , x ’ , x a Represent the horizontal coordinates of the left prediction box, the right prediction box, and the anchor box respectively.

[0072] The present invention defines each ROI sample as a multi-task loss during training: L = L cls + Lreg + L ga In addition to the conventional classification loss L cls and regression loss L reg In addition, the present invention also introduces the guide anchor loss L ga The definitions of classification loss and regression loss are similar to [1] The anchor module outputs the probability of each pixel being the target object through a pixel-by-pixel S-shaped function, and w , h ) is sampled to simulate the traversal of all values, so the anchor loss is guided L ga As shown below:

[0073] ;

[0074] Where: i Represents the index corresponding to a certain anchor point; P i Represents the anchor point is the probability of the target object; weighting factor, focusing parameter , in the present invention, select ; ( w , h )and( w g , h g ) represent the width and height of the predicted anchor box and the corresponding GT box respectively.

[0075] 1.2 Keypoint Generation Network

[0076] The key point generation network of the present invention only predicts key points for the right image, and only takes the new feature map generated by the multi-level anchor scheme as input. The present invention generates perspective points from the vertices and center points of the 3D bounding box, and then uses the output center point heat map, vertex heat map, vertex coordinates and viewpoint angle as basic modules to perform 3D box regression and correction.

[0077] In order to avoid the key point overlap problem caused by small scale, the present invention adopts the following method: Figure 5 Since there is no size difference in the key points in the image, the present invention upsamples each scale feature map generated by the multi-level anchor scheme three times through three bilinear interpolations, and adds a 1×1 convolution layer after each upsampling to reduce the channel. Before upsampling, the corresponding feature maps should be cascaded to obtain F multi-scale feature maps. Adjust to the maximum scale size and generate soft weights ∂ through the Softmax function. The importance of each scale can be intuitively seen through these generated soft weights. Then, the scale space score map S can be obtained through linear weighting. scale ,

[0078] ;

[0079] like Figure 6 As shown in FIG, the detection head mainly consists of three components. By using the anchor point position as the possible key point position, it is possible to effectively avoid the situation where the 3D projection point of the object exceeds the image boundary in the case of truncation. The present invention defines the 2D bounding box center point heat map as ,in H , W Represents the width and height of the input image, C represents the number of object categories, S represents the step size. Another component of the detection head is the heat map of the 9 perspective points of the 3D bounding box vertices and center point projection .

[0080] The local offset regressed from the 2D bounding box center is , the present invention will be closest to V C The 9 key points of the coordinates are considered to be a set of key point coordinates of the same object, and the 3D bounding box of the object is restored through the 18 constraints of these 9 key points.

[0081] Although the present invention eliminates a large number of negative samples through a multi-level anchor scheme, the training goal of the key point generation network is still to solve the imbalance problem between positive and negative samples and focal loss.

[0082] ;

[0083] Where: N represents the number of center points and vertices in the image; n represents different key point channels (if n = c represents at the center point, n = 9 represents at the vertex); α , β represents the hyperparameter used to balance the weights of positive and negative samples;

[0084] ;

[0085] Where: p represents the estimated probability that the object is at the key point.

[0086] 1.3 Stereo Regression and 3D Box Estimation

[0087] After passing through GS-RPN, the ROI Align proposed in Mask RCNN is used for the left and right feature maps respectively. Then the left and right ROI features are merged and input into two fully connected layers to extract semantic information. Through the stereo regressor, the present invention can obtain 4 outputs including object category, stereo bounding box, object size and viewpoint angle. The present invention uses the car's forward direction and the ROI viewing angle as the viewpoint regression angle, and selects [sin β ,cos β ] as the regressor to avoid discontinuity. Then the vehicle direction is obtained by decoupling the relationship between the regression angle and the 3D position. The method of combining the stereo frame with the object size can better obtain the depth information.

[0088] In addition to the viewpoint angle and depth information, the present invention also provides an additional 18 constraints for 3D box estimation by projecting 9 perspective key points in the middle of the two frames. For an input image I, the key point detection network of the present invention will give 9 key points to represent a set of N objects. The corresponding 3D bounding box can be obtained by the viewpoint angle. , 3D box center coordinates and regression size Then, given the perspective key point information, the corresponding relationship between the 3D box vertices and the 2D box vertices can be derived, and the 3D-2D relationship formula obtained by the projection transformation can be solved by the Gauss-Newton method.

[0089] The present invention transforms the problem of minimizing the reprojection of 3D key points and 2D key points into a nonlinear least squares optimization problem.

[0090] ;

[0091] In the formula: The key point is expressed as Its dimensions and directions are ; K is the given camera intrinsic matrix, R Represents the rotation angle of the 3D box The present invention takes .

[0092] 1.4 Center Depth Correction

[0093] The present invention recovers approximate depth information from the disparity information provided by the left and right bounding boxes, but since a large amount of pixel-level information is lost in the previous processing, the present invention uses a large amount of pixel-level measurements to solve the correction problem of the depth of the 3D box center.

[0094] In order to exclude the influence of background and other object pixels on the correction, Figure 7As shown, the present invention generates an instance segmentation mask when extracting object features from the feature map using ROI Align on the left image. Then, the 2D bounding box and instance segmentation mask provided in FGAS RCNN can be used to crop and align the left and right ROIs in the horizontal direction from the complete image.

[0095] ;

[0096] Where: b L and b R Represents the normalized coordinates of the left and right bounding boxes of the left 2 bounding boxes respectively ; D i ( p ) represents the predicted instance disparity value.

[0097] By calculating the disparity of all pixels in the mask area and combining it with the baseline B With camera intrinsics f The 3D position and depth value of each pixel in the mask can be calculated. The calculation formula of the 3D position is,

[0098] ;

[0099] Where: u c , v c ) represents the camera center pixel position; ( f u , f v ) are the horizontal and vertical focal lengths of the camera respectively.

[0100] The depth calculation formula is:

[0101] ;

[0102] Where: x pl , x pr Represents the pixels p The horizontal coordinates of the left and right borders.

[0103] The total matching samples set by the present invention are the sum of square differences of all pixels in the mask area:

[0104] ;

[0105] Where: Z p Represents the pixels within the mask PThe depth difference from the center of the 3D box, I L and I R They represent the 3-channel RGB vectors in the left and right images respectively.

[0106] The optimized center depth can be obtained by minimizing the total matching cost. The present invention uses an enumeration method to generate the optimal depth. Z The overall process is to first select 40 rough depths at intervals of 0.5 meters based on the estimated 3D box depth value in the previous article, and then enumerate the rough depth again at intervals of 0.05 meters to obtain the optimal depth. By fixing the alignment depth for all pixels in the mask area, the entire 3D box can be corrected. And because each pixel in the mask area contributes to the depth estimate, the discontinuity and ill-posed problems in stereo depth estimation can be avoided.

[0107] The present invention defines the total multi-task loss of the proposed method as:

[0108] ;

[0109] Where: p , r As superscripts, they represent RPN and RCNN respectively. Subscript ga, sreg , key , ver They represent the losses of the guided anchor module, stereo regressor, 3D box center point and vertex respectively.

[0110] Minimize the total matching cost:

[0111] ;

[0112] Minimizing the total matching cost means continuously screening random numbers to find a depth interpolation that minimizes the total matching cost. Then add the 3D box center depth to this depth interpolation to get the optimized center depth.

[0113] Pre-set threshold points:

[0114] ;

[0115] This formula can calculate the probability that the detected object is located in a certain area of ​​the image.

[0116] Ground truth: In the data set, the ground truth is already included. For each target object in the data set, the data set itself has introduced it, including its position, direction, etc. This information can easily confirm the gap between the target object detected by the proposed method and the real target object; Key points: The key points of the present invention are found through the key point generation module. For each target object, the corresponding center point coordinates and vertex coordinates will be generated.

[0117] Experiment and analysis

[0118] The method proposed in the present invention is evaluated on the challenging Kitti and NuScenes3D object detection benchmarks and compared with the latest methods. Then the present invention conducts an ablation study to analyze the effectiveness of different components of the proposed method. Finally, the present invention also provides an introduction to the actual vehicle-mounted experimental platform and experimental site. The present invention uses Ubuntu18.04, equipped with i7-9700k CPU and dual 2080Ti GPU, and PyTorch to run the network of the present invention.

[0119] 3D object detection experiment on Kitti

[0120] The Kitti object detection benchmark contains 7481 training images and 7518 test images.

[18] The training images are roughly divided into a training set of 3712 images and a test set of 3769 images. Following the settings of Kitti, objects are divided into three levels: easy, medium and difficult according to the 2D bounding box size, occlusion and truncation degree. This paper uses the average precision ( AP ) for 3D detection ( AP 3d ) and bird's-eye view detection ( AP bev ) is used to evaluate the performance of 3D detection and localization. As shown in Tables 1 and 2, the proposed method is compared with the previous state-of-the-art image-based 3D detection methods for car categories using 0.7 and 0.5 as IOU thresholds.

[0121] Table 1 Comparison of the average precision (AP) of the bird's-eye view and 3D box evaluated using the Kitti validation set (IOU threshold is 0.5)

[0122]

[0123] Table 2 Comparison of the average precision (AP) of the bird's-eye view and 3D box evaluated using the Kitti validation set (IOU threshold is 0.7)

[0124]

[0125] During training, the method of the present invention outperforms Stereo RCNN in all indicators. Specifically, this advantage comes from the large number of constraints provided by the key point detection network.

[0126] Table 3 is a test conducted on the Kitti test set. The present invention uses 0.7 as the IOU threshold and compares it with previous stereo methods. Compared with Stereo RCNN, the method of the present invention achieves improvements in all indicators. Specifically, compared with the latest OC-Stereo method, the method of the present invention improves the average accuracy of the bird's-eye view at a simple level by 3.8%, and improves the average accuracy of simple and medium 3D boxes by about 11% and 2%, respectively. It can be observed from Table 3 that our method almost surpasses all previous stereo methods. Specifically, our method takes 0.11s for 2D detection and segmentation, 0.08s for key point generation network, and 0.13s for 3D bounding box regression and center optimization.

[0127] Table 3 Comparison of 3D object detection results and running time evaluated using the Kitti test set

[0128]

[0129] pass Figure 8 It can be seen intuitively that as the distance of the object increases, the error of disparity and depth show opposite trends. In addition, the 3D detection performance is also inversely proportional to the distance of the object. In order to solve the problem of pixel-level information loss in the original image, the method of the present invention hopes to use pixel-level disparity for optimization. However, the ordinary pixel-level disparity estimation problem has the problem of over-smoothing, so the present invention uses instance-level disparity estimation as the information to achieve sub-pixel matching. The visualization results of the method of the present invention are as follows: Fig. 9 shown.

[0130] 3D object detection experiments on NuScenes

[0131] The NuScenes dataset is the latest large-scale autonomous driving dataset. To increase the challenge of the dataset, it collects 1,000 driving scenes from Boston and Singapore. Compared with the Kitti dataset, the NuScenes dataset collects data through 6 multi-view cameras and 32-line Lidar, and provides 7 times more object annotations. The dataset contains 28,130 training samples and 6,019 validation samples. Unlike the Kitti dataset, the NuScenes dataset generally uses the NuScenesDetection Score (NDS) as a measurement indicator.

[0132] FGAS RCNN was tested on the NuScenes dataset to verify the generalization of the proposed method. Table 4 provides the three categories, mAP and NDS of vehicle categories achieved by the 3D object detector. It can be observed from Table 4 that compared with the latest 3D-CVF, the proposed method has improved mAP and NDS by about 5.99% and 2.97% respectively, and in the detection of vehicle categories, the proposed method is significantly better than other methods.

[0133] Table 4 3D detection map on NuScenes test set

[0134]

[0135] Ablation experiments show the effectiveness of each module of the method proposed in this invention.

[0136] GS RPN This paper proposes a method for region proposal based on guided sparse anchors. In order to intuitively understand the performance of this module, this paper uses the average recall rate ( AR 300 , %) and 2D detection average accuracy ( AP 2d , %) compares FGAS- RCNN, Stereo-RCNN and Faster-RCNN. The test uses the same backbone network and left and right feature fusion strategy.

[0137] The present invention also studies the IOU distribution of proposals generated using the method of the present invention and compares it with the traditional RPN method. Fig.10 As shown in Figure 2, by observing the IOU distribution of proposals generated by RPN, Stereo-RPN and GS-RPN, it can be clearly observed that GS-RPN provides more high IOU proposals.

[0138] Position Threshold The sparsity of the anchor point distribution can be controlled. The present invention chooses to compare the average number of anchor points generated for each image, the average recall rate (AR) and the inference running time by changing the position threshold. The results are shown in Table 5.

[0139] Fig.11 The curve in FIG. 1 shows the comparison between the method of the present invention and other detection methods in terms of ROI classification accuracy. From the figure, it can be seen that the method of the present invention can maintain a high performance while improving the detection efficiency.

[0140] From the results in Table 6, we can observe that compared with Stereo-RPN, GS-RPN has more positive proposals with high IOU, which makes the method of the present invention have a higher average precision. And because the method of the present invention can still maintain a high recall rate when there are fewer proposals, the present invention can still improve the final mAP after training only 300 proposals.

[0141] Table 5 Different position thresholds The experimental results

[0142]

[0143] In Table 7, the present invention uses the same backbone network, hyperparameters and enhancement methods, and shows the comparison data of Faster RCNN, Stereo RCNN and FGAS RCNN after NMS processing. From the data in the table, it can be seen that the use of GS RPN can reduce time consumption while maintaining a high recall rate. Compared with the previous method, the method of the present invention improves the 2D average accuracy by about 1.34% in all aspects.

[0144] Table 6 Experimental results of different proposals and IOU thresholds

[0145]

[0146] The proposed method has higher recall rate and detection accuracy on a single image, and can generate high-quality proposals in both left and right images without adding additional calculations. The traditional RPN is a sliding window-based method, while the GS-RPN is a guide anchor-based method. The present invention shows some examples of proposals based on sliding windows and guide anchor methods. Fig.12 As shown in Figure 2, compared to RPN, the anchor boxes of GS-RPN are more focused on foreground objects.

[0147] Table 7 Average recall rate ( AR 300 , %) and 2D detection average precision ( AP 2d , %)Compare

[0148]

[0149] Table 8 Comparison of 3D detection evaluation with and without key point constraints on the Kitti validation set

[0150]

[0151] Key Point Detection Network The method for providing key points proposed in the present invention can provide 18 constraints to correct the 3D box. In order to verify the benefits of this method, the present invention evaluates the performance of the rough 3D box without using key point constraints and the performance of the 3D box after correction using the key point network. In addition, the present invention also adds a method for generating key point constraints using regression perspective and 2D box information for comparison. As shown in Table 8, since the key point detection network can provide a large number of pixel-level constraints other than the 2D box, the method proposed in the present invention is significantly better than other key point detection methods, where when IOU=0.7, the simple and difficult difficulties are improved by 7.81% and 8.17% respectively.

[0152] Actual platform testing

[0153] In order to verify the effectiveness of the proposed method, the present invention also conducted a real test on an actual vehicle platform. Fig.13 As shown in the figure, the platform is composed of multiple sensors. In addition to the basic 16-line radar, the platform also adds two additional LIDARs, including Tele-15 radar and millimeter wave radar.

[0154] The main factor limiting the lidar-based detection method at this stage is the price of solid-state lidar. Unlike the high price of conventional lidar, Tele-15 has successfully solved the price problem. This radar achieves long-distance high-precision detection by sacrificing the detection angle, and can exist as a long-range millimeter-wave radar or additional safety redundancy. The radar's wiring harness is very concentrated, and concentrated illumination is exchanged for a high-density point cloud and an increased effective detection distance. The Tele-15 radar can reserve a processing distance of 250 meters for high-speed vehicles.

[0155] In the experimental platform of the present invention, the millimeter wave radar is mainly responsible for the speed and distance of the vehicle at a long distance ahead, while the Tele-15 radar serves as its additional safety redundancy to increase the safety of the vehicle; the 16-line radar is responsible for short-distance, large-angle, and low-power detection tasks.

[0156] In the experiment on the vehicle platform, the present invention selects three aspects to verify the good performance of the method of the present invention in BEV target detection and 3D target detection. Fig.14 Comparison of the 3D detection average accuracy of FGASRCNN and Stereo RCNN under different 3D IOU standards and different distances AP 3d . Fig.14 It is constructed from real data collected from actual platforms. Fig.14It can be observed that the proposed method has better performance than StereoRCNN at the same distance and the same 3D IOU. When the distance is 15m and IOU=0.65, the performance improvement is 10.17%. Fig.15 As shown in Figure 2, when IOU=0.7, the average accuracy of the proposed method at a medium level of bird's-eye view can reach a high level of 58.35%.

[0157] In order to more intuitively judge the quality of the model, the present invention takes the sensor factor into consideration when adding the comparison model. As can be seen from Table 9, the detection accuracy of the vehicle category of the method of the present invention is higher than that of other comparison models, reaching 86.25%, 79.57% and 73.60% respectively, and the average accuracy also reaches 79.81%. These data can show that the method proposed by the present invention has strong generalization ability.

[0158] Table 9 Comparison of 3D object detection performance for cars, vans, and buses

[0159]

[0160] The present invention relies on the vehicle-mounted platform to carry out testing and data collection tasks in real scenes, and inputs point cloud data into the model for real-time 3D target detection. The visualization results are as follows: Fig.15 As shown in the figure, the method of the present invention can output high-quality predictions with higher accuracy compared to other latest methods. In the case of medium and long distances, the method proposed by the present invention can achieve an average accuracy of 68.3%.

[0161] References are as follows:

[0162] [7] Li P, Chen X, Shen S. Stereo r-cnn based 3d object detection for autonomous driving [C]. Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. 2019: 7644-7652.

[0163]

[16] Geiger A, Lenz P, Urtasun R. Are we ready for autonomous drivingthe kitti vision benchmark suite [C]. 2012 IEEE Conference on Computer Visionand Pattern Recognition. IEEE, 2012: 3354-3361.

[0164]

[17] Caesr H, Bankiti V, Lang A H, et al. nuscenes: A multimodaldataset for autonomous driving [C]. Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition. 2020: 11621-11631.

[0165]

[18] Che X, Luo X, Weng J, et al. Multi-View Gait Image Generationfor Cross-View Gait Recognition [J]. IEEE Transactions on Image Processing,2021, 30: 3041-3055.

[0166]

[19] Yan C, Salman E. Mono3D: Open source cell library for monolithic3-D integrated circuits[J]. IEEE Transactions on Circuits and Systems I:Regular Papers, 2017, 65(3): 1075-1085.

[0167]

[20] Tung F, Little J J. MF3D: Model-free 3D semantic scene parsing[C]. 2017 IEEE International Conference on Robotics and Automation (ICRA).IEEE, 2017: 4596-4603.

[0168]

[21] Li P, Zhao H, Liu P, et al. RTM3D: Real-time monocular 3Ddetection from object keypoints for autonomous driving[J]. arXiv preprintarXiv: 2001.03343, 2020, 2.

[0169]

[22] Li B, Zhang T, Xia T. Vehicle detection from 3d lidar usingfully convolutional network[J]. arXiv preprint arXiv:1608.07916, 2016.

[0170]

[23] Wang Y, Chao W L, Garg D, et al. Pseudo-lidar from visual depthestimation: Bridging the gap in 3d object detection for autonomous driving[C]. Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. 2019: 8445-8453.

[0171]

[24] Pon A D, Ku J, Li C, et al. Object-centric stereo matching for3d object detection[C]. 2020 IEEE International Conference on Robotics andAutomation (ICRA). IEEE, 2020: 8383-8389.

[0172]

[25] Yan Y, Mao Y, Li B. Second: Sparsely embedded convolutionaldetection[J]. Sensors, 2018, 18(10): 3337.

[0173]

[26] Ye Y, Chen H, Zhang C, et al. Sarpnet: Shape attention regional proposal network for lidar-based 3d object detection[J]. Neurocomputing, 2020, 379: 53-63.

[0174]

[27] Lang AH, Vora S, Caesar H, et al. Pointpillars: Fast encoders for object detection from point clouds[C]. Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2019: 12697-12705.

[0175]

[28] Yoo JH, Kim Y, Kim JS, et al. 3d-cvf: Generating joint cameraand lidar features using cross-view spatial feature fusion for 3d objectdetection[J]. arXiv preprint arXiv:2004.12636, 2020, 3.

[0176] The present invention proposes an anchor-guided 3D object detection algorithm based on Stereo RCNN. The method of the present invention makes full use of the semantic information of the input image to guide the generation of sparse anchor points, and generates non-uniform anchor points by predicting the anchor point positions and their anchor box shapes. The 3D bounding box is generated by combining the key point generation network with the stereo regressor, and then the 3D bounding box accuracy is optimized by calculating the pixel-by-pixel depth in a specific area using instance disparity and instance mask. Experiments on public datasets show that the method of the present invention improves the computational efficiency while maintaining the high accuracy of similar algorithms. In addition, the generalization and portability of the method of the present invention are demonstrated through experiments on different datasets, and it has good robustness in different environments.

Claims

1. An anchor-guided 3D object detection method for autonomous driving, characterized in that: include: In the first stage, the left and right images are input, and then the corresponding foreground position probability maps are generated to generate sparse anchor points and sparse anchor boxes. Then, the 2D pre-selected boxes are generated by taking the left and right anchors as a whole to obtain a new feature map. In the second stage, the key point generation network uses the sparse anchor point information to generate a key point heat map, and combines it with the stereo regressor to generate a 3D pre-selected box. In the third stage, the instance segmentation mask generated by Mask Branch is combined with instance-level disparity estimation for pixel-level optimization; 3D object detection is completed; in the first stage, anchor points are collected in the multi-scale feature map according to the feature pyramid architecture, the basic anchor points are set, and the object type and box offset are regressed to the input position using 3×3 convolution channel reduction and two fully connected layers, and then the deformable convolution with offset is applied to the original feature map to obtain a new feature map; the new feature maps of the left and right outputs are connected at each scale, and the concatenated features are fed into the Stereo RPN network to obtain an accurate detection box; in the second stage, the key point generation network only predicts key points for the new feature map on the right; In the second stage, each scale feature map generated in the first stage is upsampled three times through three bilinear interpolations, and a 1×1 convolution layer is added after each upsampling to reduce the channel. Before upsampling, the corresponding feature maps are cascaded, the obtained multi-scale feature maps are adjusted to the maximum scale, and the soft weight ∂ is generated through the Softmax function operation. Then, the scale space score map S can be obtained through the linear weighting formula (4): scale , 。 2. The anchor-guided 3D target detection method for autonomous driving according to claim 1, characterized in that: In the first stage, a 1×1 convolution is applied to the input feature map to generate a probability map of the same size as the feature map. P , and converted into a probability value through the Sigmod function formula (1); ; Where: x a ,y a ) corresponds to the coordinates on the input feature map, S represents the step of the feature map; Points with probabilities higher than a preset threshold are regarded as activity areas where objects may exist. Then, the shape of the activity area where objects may exist is predicted based on the feature map and the probability map, and the shape with the maximum coverage is selected as the predicted anchor box.

3. The anchor-guided 3D target detection method for autonomous driving according to claim 1, characterized in that: In the first stage, the left and right feature maps are concatenated and input into the guided stereo RPN network, and the corresponding detection boxes are matched by the maximum coverage rate. The left and right feature maps are concatenated and input into the guided stereo RPN network, and the 6 coordinate parameterization in formula (2) is used to regress the 2D bounding box. Then, the detection boxes are screened again by using non-maximum suppression on the left and right ROI areas to obtain the 2D pre-selected box. ; Where: x , y , w , h Represents the horizontal and vertical coordinates, width and height of the prediction box; x , x ’ , x a Represent the horizontal coordinates of the left prediction box, the right prediction box, and the anchor box respectively.

4. The anchor-guided 3D target detection method for autonomous driving according to claim 1, characterized in that: In the second stage, the 2D bounding box center point heat map is defined as ,in H , W Represents the width and height of the input image, C represents the number of object categories, S represents the step length; the heat map of the 9 perspective points projected from the 3D bounding box vertices and the center point is defined as ; The local offset regressed from the 2D bounding box center is defined as ; will be closest to V C The 9 key points of the coordinates are considered to be a set of key point coordinates of the same object, and the 3D bounding box of the object is restored through the 18 constraints of these 9 key points.

5. The anchor-guided 3D target detection method for autonomous driving according to claim 1, characterized in that: In the third stage, the ROI Align proposed in Mask RCNN is used on the new feature maps in the first stage. Then the left and right ROI features are merged and input into two fully connected layers to extract semantic information. The car's forward direction and the ROI viewing angle are used as the viewpoint regression angle, and [sin β ,cos β ] as a regressor to avoid discontinuity, and then the vehicle direction is obtained by decoupling the relationship between the regression angle and the 3D position; the 3D bounding box is obtained by the viewpoint angle , 3D box center coordinates and regression size, and solve the 3D-2D relationship formula (7) obtained by the projection transformation using the Gauss-Newton method; ; In the formula: The key point is expressed as , whose dimensions and directions are ; K is the given camera intrinsic matrix, R Represents the rotation angle of the 3D box ;Pick ; In the third stage, the instance segmentation mask is generated when the object features are extracted from the feature map using ROI Align on the left image. Then, the left and right ROIs can be cropped from the full image and aligned in the horizontal direction according to formula (8) using the 2D bounding box and the instance segmentation mask; ; Where: b L and b R Represents the normalized coordinates of the left and right bounding boxes of the left 2 bounding boxes respectively ; D i ( p ) represents the predicted instance disparity value; By calculating the disparity of all pixels in the mask area and combining it with the baseline B With camera intrinsics f The 3D position and depth value of each pixel in the mask can be calculated. The calculation formula of the 3D position is (9), ; Where: u c , v c ) represents the camera center pixel position; ( f u , f v ) are the horizontal and vertical focal lengths of the camera respectively; The depth calculation formula is: ; Where: x pl , x pr Represents the pixels p Horizontal coordinates in the left and right borders; The total number of matching samples is set to the sum of the squared differences of all pixels in the mask area: ; Where: Z p Represents the pixels within the mask P The depth difference from the center of the 3D box, I L and I R They represent the 3-channel RGB vectors in the left and right images respectively; the optimized center depth can be obtained by minimizing the total matching cost.

6. A device for anchor-guided 3D target detection for autonomous driving, used in the anchor-guided 3D target detection method of claim 1, characterized in that: It includes an image acquisition module and a data processing module. The data processing module includes a stereo RPN module, a key point generation network module, a stereo regression and 3D box estimation module, and a center depth correction module.

7. An electronic device, comprising a processor and a memory, characterized in that: The memory stores instructions executed by the processor, so that the processor can execute the anchor-guided 3D target detection method for autonomous driving as described in claim 1; or a computer-readable storage medium storing computer instructions, wherein the computer instructions enable the computer to execute the anchor-guided 3D target detection method for autonomous driving as described in claim 1.