A detection method for improving the efficiency of automatic driving three-dimensional target detection

Abstract

The application discloses a detection method for improving the efficiency of automatic driving three-dimensional target detection, comprising the following steps: extracting a group of key points from the original point cloud by using the farthest point sampling method; quantifying the key points to be queried into voxels, and obtaining the adjacent voxel feature set of the key points by calculating the Manhattan distance; enhancing the features by matching and aggregating; connecting the original point cloud features, the multi-scale voxel aggregation features and the bird's eye view features to obtain the original key point features; optimizing the key point features; uniformly segmenting each region of interest by using a group of virtual grid points, and setting the key point threshold and the set abstract radius to screen the grid points; correcting the direction and boundary of the proposal by fitting the minimum enclosing rectangle and the weighted key point features, to obtain the corrected 3D box; repeating the above steps until all the key points are traversed, to obtain the final three-dimensional target detection result. The application reduces the calculation amount while improving the prediction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to a three-dimensional target detection technology, in particular to a target detection method for automatic driving, and more particularly to a three-dimensional target detection method for improving detection efficiency through point cloud voxel fusion. BACKGROUND

[0002] Various modules developed by using 3D target detection technology play an important role in the fields of automatic driving, robot technology and path planning. Traditional 3D target detection methods pay more attention to the original point cloud obtained by a laser radar sensor. Although the original point cloud can provide very accurate depth information, the sparsity of the point cloud will inevitably limit the expansion of its application field. The more popular research in recent years can be divided into three categories: point cloud (Point) based methods, voxel (Voxel) based methods and methods combining point cloud and voxel. These three methods all need to use convolutional neural network (CNN) to extract 3D representation, and the difference between them is how to convert the 3D representation of the target object.

[0003] The Point-based method directly takes the original point cloud as the processing object. They extract point-based features, so they can provide accurate 3D spatial representation. A series of methods based on PointNet solve the problem of unorderedness in the original point cloud by applying a symmetric operation function on the transformed elements in the set, and directly extracting features from the original point cloud will inevitably bring a huge amount of calculation. In order to solve this problem, the current Point-based method adopts a two-stage pipeline. Taking Point R-CNN as an example (see the literature: Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li, “Point R-CNN: 3D Object Proposal Generation And Detection From Point Cloud”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-779, 2019), it first estimates 3D target proposals by segmented foreground objects in the first stage. Then in the second stage, more accurate Point-based features are used to refine the 3D proposals to generate the final prediction proposals. Although the Point-based method has high prediction performance, it still cannot avoid the expensive calculation cost brought by extracting features from a single point.

[0004] The Voxel-based method converts the irregular point cloud into an ordered voxel grid, solving the problem of sparsity and irregularity of the original point cloud. In early research, the point cloud is projected into the pseudo bird's eye view (BEV) to obtain a relatively dense 3D representation. This method can only obtain high-precision 3D features, and the remaining information of the target object cannot be obtained. In order to solve this problem, recent methods will use 3D convolution to extract voxel features from voxel representation. VoxelNet (see Zhou Y, Tuzel O. Voxelnet: End-to-End Learning For Point Cloud Based 3d Object Detection [C]. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4490-4499) first proposed to deploy PointNet to the laser radar point cloud, and proposed to jointly process through 3D convolution layer, 2D backbone network and detection head. But its shortcomings are also very obvious, the running speed of VoxelNet is very slow. After that, a series of improvements have appeared to improve the running speed, so that the voxel-based method has been fully developed and used. However, these methods are still limited by the quantization error when segmenting voxels. Specifically, the voxel-based method has two main shortcomings: one is that a large amount of fine-grained 3D structure information is lost; the other is that the size of the voxel grid will greatly affect the performance of the algorithm.

[0005] To solve the problems of expensive computation cost and loss of fine-grained structure information at the same time, recent methods choose to combine Point-based methods and Voxel-based methods. PV R-CNN (see document Shi S, Guo C, Jiang L, et al. Pv-rcnn: Point-Voxel Feature Set Abstraction For 3d Object Detection [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 10529-10538) expands SECOND by adding Point-based feature. PV R-CNN is also a method of projecting 2D views, but it is different in that it obtains multi-scale 3D voxel features by using a voxel set abstraction module and integrates them into neighboring keypoints. Finally, the feature of each 3D region proposal is extracted by ROI grid pooling to refine the final proposal. The detection method combining Point and Voxel takes into account the advantages of Voxel-based and Point-based methods, but still cannot completely avoid the problems of large computation, slow computation speed and insufficient utilization of complementary information. Moreover, the methods in these three directions all have the problem of size ambiguity, that is, these methods may ignore the boundary size of the proposal, which will lead to the decline of algorithm performance when dealing with some complex scenes.

[0006] 3D target detection needs to estimate the 7 degrees of freedom (including coordinate position, dimension and direction) of the surrounding target object in the real complex 3D space. At present, most 3D target detection methods directly fit the target object through axis-aligned boxes, and the sparsity of point cloud will cause a large number of no-measurement value regions in 3D space, so only one 3D box in the output result cannot be aligned with the global coordinate system. The traditional Anchor-based method will cause the direction error of fitting the rotating object.

[0007] Therefore, in order to improve the detection efficiency of three-dimensional targets to adapt to the application requirements of automatic driving, it is necessary to solve the problems of large computation, size ambiguity and angle deviation, and imbalance between positive and negative samples in generating keypoints in the prior art. SUMMARY

[0008] The purpose of the present application is to provide a detection method for improving the detection efficiency of three-dimensional targets for automatic driving, which can improve the prediction accuracy while reducing the computation.

[0009] To achieve the above object, the technical scheme adopted by the present application is as follows: a detection method for improving the efficiency of automatic driving three-dimensional target detection, comprising the following steps:

[0010] (1) obtaining original point cloud data to be detected;

[0011] (2) using the farthest point sampling method to extract a group of key points P from the original point cloud, wherein the key points are p i , and i is the serial number of the key point;

[0012] (3) quantizing the key point p i to be queried into a corresponding voxel, and then judging whether the non-empty voxel around the key point is the adjacent voxel of the query key point by calculating the Manhattan distance between the non-empty voxel and the key point voxel, to obtain the adjacent voxel feature set of the key point p i wherein K represents the current downsampling multiple;

[0013] (4) calculating the matching probability of the point cloud (Point) based feature of the key point p i and the voxel (Voxel) based feature in the adjacent voxel feature set thereof, selecting the most similar k voxel features according to the matching probability, and aggregating the k aggregated voxel features with the corresponding matching probability for feature enhancement, and finally generating the feature of the key point p i through the PointNet-block method K is 1x, 2x, 4x;

[0014] (5) connecting the original point cloud feature multi-scale voxel aggregation feature and bird's eye view feature to obtain the original key point feature

[0015] (6) calculating the feature matching probability mean of the key point feature and its adjacent original point cloud feature multi-scale voxel aggregation feature respectively, and weighting the initial key point feature together with the predicted foreground probability to obtain the updated key point feature

[0016] (7) uniformly dividing each region of interest (ROI) by using a group of virtual grid points, and setting a key point threshold λ and a set abstraction radius r g screening the grid points; correcting the direction and boundary of the proposal by fitting the minimum enclosing rectangle and the weighted key point feature to obtain a corrected 3D box;

[0017] ​(8) repeating steps (3) to (7) until all key points are traversed to obtain a final three-dimensional target detection result.

[0018] In the technical solution, step (3) comprises:

[0019] The Kth set of voxel features obtained by 3D sparse convolution and its corresponding actual coordinates are expressed as:

[0020]

[0021] Where i represents the number of non-empty voxels in the Kth level;

[0022] The key point to be queried is taken as the origin of the local coordinate system, and 3 offsets are added to the non-empty voxels around the key point, That is,

[0023]

[0024] Then sample all non-empty voxels within the Manhattan threshold D K For any key point p i A neighboring voxel feature set can be obtained, that is:

[0025]

[0026] Wherein, represents the relative position of the semantic voxel feature; Kj represents the jth neighboring non-empty voxel of the key point pi in the Kth level; represents the Manhattan distance between the semantic voxel feature and the corresponding key point,

[0027]

[0028] In the technical solution, step (4) comprises:

[0029] The matching probability of the point cloud (Point) based feature of the key point pi and the voxel (Voxel) based feature in its neighboring voxel feature set is:

[0030]

[0031] Wherein, f(V n ,p i ) represents the nth voxel (Voxel) based similar feature of the key point p i .

[0032] The most similar k voxel features are selected according to the matching probability, and they are aggregated with the corresponding matching probability. Finally, the k aggregated voxel features are generated into the feature of the key point p i By PointNet-block,

[0033]

[0034] where M(·) denotes a multi-layer perceptron network used to encode the keypoint voxel features; max(·) denotes max-pooling along the channel.

[0035] In the technical solution, the keypoint p i obtained in step (5) has a feature,

[0036]

[0037] In step (6), for any keypoint p i , a set of neighboring raw point clouds can be obtained,

[0038]

[0039] wherein, respectively represent the point cloud (Point) based feature and the actual coordinates of the raw point cloud; represents the relative position of the raw point cloud; r raw represents a set radius range; C raw , F raw respectively represent the feature set of the raw point cloud and the corresponding actual coordinates; N represents the number of neighboring raw point clouds;

[0040] The point cloud based feature of the keypoint p i and the matching probability of the point cloud based feature of the neighboring raw point cloud are calculated in the following manner,

[0041]

[0042] The feature probability of the keypoint, i.e., the prediction probability of the keypoint belonging to the foreground, is obtained through a 3-layer multi-layer perceptron network and a Sigmod function The feature matching probability mean value of the keypoint and the neighboring raw point cloud is calculated by SA The feature matching probability mean value of the keypoint and the neighboring voxel in multiple scales is calculated by VPFA Then, the weight of the keypoint from the foreground region is improved through averaging and weighting operations, and the keypoint feature after reweighting is represented as:

[0043]

[0044] In the technical solution, step (7) includes:

[0045] For each 3D proposal, M×M×M virtual mesh points are set up, evenly distributed in length, width, and height, and the coordinates of the virtual mesh points are normalized to the coordinates of the real points. Then, SA is used for each virtual mesh point to select key point features to be aggregated, and the key point threshold λ and the set abstraction radius r of the virtual mesh points are set. g Aggregate all weighted key points within the set's abstract radius. For any virtual grid point g... i Each can obtain a set of adjacent weighted voxel features.

[0046]

[0047] Where, p j -g i N' represents the relative position of adjacent weighted key points; N' represents the total number of adjacent weighted key points.

[0048] If any grid point g i With r g If no λ weighted key points can be found within a sphere of radius λ, then the virtual grid point is deleted.

[0049] The virtual grid points set are also distributed on the proposal surface, which can capture adjacent weighted feature points outside the target proposal's 3D bounding box but within the set abstract radius;

[0050] Obtain the set of virtual grid points for each proposal.

[0051] G={g m ∈R 3 |m∈[0,M 3 -1]}

[0052] Then, a minimum bounding rectangle is fitted around all remaining points in the virtual mesh point set to correct the proposal direction and boundary. The features of the virtual mesh are obtained by interpolating the features in adjacent weighted voxel feature sets.

[0053]

[0054] Where d(·) represents the L2 distance;

[0055] The features of all virtual grid points in a proposal are added to the grid feature set, and then passed through an MLP with a channel dimension of [C+3,256,128,128] and global max pooling to obtain the ROI features of each proposal. Then, another MLP is used to obtain the Intersection over Union (IoU) estimate of each box. Finally, a two-layer multilayer perceptron is used to predict the foreground confidence of the target proposal and optimize the 3D box.

[0056] The key point threshold value can be selected according to experience, and preferably, the key point threshold value λ = 3.

[0057] Compared with the prior art, the application has the following advantages due to the technical scheme:

[0058] 1. The application proposes a feature aggregation method using offset and feature matching probability to solve the problem of large calculation amount in traditional voxel-point feature aggregation. The method queries neighbor voxels through offset and filters voxel features using voxel-key point matching probability, thereby improving prediction accuracy and reducing calculation amount.

[0059] 2. A pooling module based on a virtual grid point group is proposed to solve the size ambiguity and angle deviation problems existing in a large number of target detection algorithms. The module optimizes the boundary and angle of the generated boundary box proposal through a group of uniformly distributed virtual grid points.

[0060] 3. A key point analysis module based on multi-scale probability weighting is proposed to solve the imbalance problem between positive and negative samples in generated key points. The module calculates the mean of the matching probability in the multi-scale and then weights the key point features to determine whether the key point exists in the background or the foreground, and finally solves the imbalance problem between positive and negative samples. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a method framework diagram of the embodiment one of the application;

[0062] Figure 2 is a schematic diagram of neighbor voxel query in the embodiment one;

[0063] Figure 3 is a schematic diagram of the KP Analysis module in the embodiment one;

[0064] Figure 4 is a schematic diagram of common problems of 3D target detection algorithms;

[0065] Figure 5 is a schematic diagram of virtual grid point screening in the embodiment one;

[0066] Figure 6 is a qualitative result instance of the embodiment one on the KITTI dataset;

[0067] Figure 7 is a qualitative result instance of the embodiment one on the NuScenes dataset;

[0068] Figure 8 shows the influence of the number of key points on 3D mAP and inference time;

[0069] Figure 9The impact of different key point thresholds on the PR curve (IoU = 0.7) is shown.

[0070] Figure 10 is a performance comparison graph at different distances on the actual test platform in Example One. DETAILED DESCRIPTION

[0071] The application will be further described below in conjunction with the accompanying drawings and examples:

[0072] Example One: A detection method for improving the efficiency of automatic driving three-dimensional target detection, called APVR. The original point cloud is used as input, and 3D sparse convolution is used as the backbone network to extract voxel features. Then, offset is added and matching probability is calculated to fully utilize complementary information. Subsequently, the IOU-Grid Pooling module is applied to further refine the proposal generation based on the filtered key point features.

[0073] Referring to Figure 1 , the framework can be expressed as:

[0074]

[0075]

[0076] The specific modules involved are as follows:

[0077] 1. VPFA module

[0078] Traditional SA-based local feature extraction methods have been widely used in 3D target detection algorithms in recent years. Although this method can extract multi-scale features by designing different radius thresholds, it still cannot avoid the problem of large computational cost. Unlike the irregular distribution of original point clouds in the quantization space, voxels are generally arranged regularly. Therefore, the use of spherical search methods similar to SA results in a large amount of computation. In order to further reduce the computational cost, this method selects to add an offset to query adjacent voxels.

[0079] The original point cloud is used as input, and a set of key points P is selected from the original point cloud through Furtherest Point Sampling (FPS). As Figure 2 shown, first, the key points to be queried are quantized into corresponding voxels. Then, by calculating the Manhattan distance between the non-empty voxels around the key points and the key point voxels, it is determined whether the non-empty voxels are adjacent voxels of the query key points.

[0080] Specifically, the K-level voxel feature set obtained by 3D sparse convolution and its corresponding actual coordinates are represented as:

[0081]

[0082] where i represents the number of non-empty voxels in the K-th level.

[0083] Take the key point to be queried as the origin of the local coordinate system, and add 3 offsets to the non-empty voxels around it,

[0084]

[0085] Then sample all non-empty voxels within the Manhattan threshold D. K For any key point p i , a neighboring voxel feature set can be obtained, that is:

[0086]

[0087] wherein, represents the relative position of the semantic voxel feature; Kj represents the jth neighboring non-empty voxel of the key point p i in the K-th level;

[0088]

[0089] Compared with the traditional SA method, the method of the present application has an order of magnitude lower in calculation amount. Therefore, when performing voxel query, the method can be more efficient and have lower time complexity than the traditional method.

[0090] After using 3D sparse convolution, the original point cloud is gradually down-sampled to convert into 1x, 2x, and 4x 3D feature volumes (Feature Volumes). By setting three different Manhattan thresholds, feature aggregation can be performed in voxels of different scales.

[0091] In order to further enhance the relevant voxel features, the matching probability of the Point-based feature of the key point p i and the Voxel-based feature in the neighboring voxel feature set thereof is also calculated,

[0092]

[0093] wherein, f(V n ,p i ) represents the nth Voxel-based similar feature of the key point p i .

[0094] The most similar k voxel features are selected according to the matching probability, and the corresponding matching probability is aggregated. The aggregated features contain more accurate fine-grained information than the original Voxel-based features. Therefore, the aggregated features have the function of feature enhancement and can provide a better discriminative 3D representation. Finally, the k aggregated voxel features are generated into key points p i by a PointNet-block,

[0095]

[0096] where M(·) represents a multi-layer perceptron network used to encode the key point voxel features; max(·) represents an along-channel max-pooling operation.

[0097] The VPFA module of the present application can obtain 3-scale context information for each key point by setting different Manhattan thresholds, and the aggregated features enable the key point features to obtain more abundant fine-grained information.

[0098] 2、KP Analysis module

[0099] The key point features are connected by the original point cloud features multi-scale voxel aggregated features and the bird's eye view features. The bird's eye view features are mainly obtained by bilinear interpolation on the down-sampled bird's eye view feature map. Therefore, the features of the key point p i can be represented as,

[0100]

[0101] Connecting different features of the key point can enable the key point features to contain a large amount of fine-grained information, thereby being able to retain the 3D structure information of the entire target scene. In order to solve the problem of mixing foreground and background key points, the present application proposes a key point analysis module (KP Analysis Module).

[0102] Considering that the original point cloud generated by the laser radar is distributed on the surfaces of the background and foreground objects, a multi-scale matching probability weighted key point analysis module is designed. Specifically, unlike the traditional method of using only a multi-layer perceptron, the present application needs to calculate the matching probabilities of the key points with the original point cloud features and the multi-scale voxel features to further enhance the features to determine whether the key point exists in the foreground or the background. Compared with the foreground area, the surrounding real environment of the key points distributed in the background area is more complex. Therefore, the average matching probability of the key points in the foreground area is higher than that of the key points in the background area.

[0103] Using the traditional SA module to obtain the raw features of keypoints, therefore for any keypoint p i Similarly, a set of adjacent original point clouds can be obtained.

[0104]

[0105] in, These represent the point-based features and actual coordinates of the original point cloud, respectively. This represents the relative position of the original point cloud; r raw Indicates the set radius range; C raw ,F raw These represent the feature sets of the original point cloud and their corresponding actual coordinates, respectively; N represents the number of adjacent original point clouds.

[0106] Then calculate the key point p. i The probability of matching point-based features with the point-based features of the adjacent original point clouds.

[0107]

[0108] The KP Analysis module, such as Figure 3 As shown, this module can fully utilize the complementary information between Voxel-based and Point-based approaches. Specifically, it acquires different keypoint information in three ways: First, it obtains the feature probability of the keypoint, i.e., the predicted probability that the keypoint belongs to the foreground, through a 3-layer multilayer perceptron network and a Sigmad function. The mean feature matching probability between key points and adjacent original point clouds is calculated using SA. The mean feature matching probability of keypoints and adjacent voxels at multiple scales was calculated using VPFA. Then, by averaging and weighting operations, the weights of key points from the foreground region are increased, ultimately achieving the goal of optimizing the proposal.

[0109] Key features after reweighting It can be represented as:

[0110]

[0111] Compared to simply using a multilayer perceptron, the method of this invention offers significant improvements in both the accuracy of determining the location of key points and the resolution of key point features.

[0112] In the training stage, the application selects to use Ground-truth 3D Box to generate segmentation labels for each key point. And selects to use Fcoal Loss to solve the imbalance problem of the number of foreground and background key points.

[0113] 3, IoU-Grid Pooling module

[0114] As Figure 4 shown, there are usually two problems in current 3D target detection: size ambiguity and the problem of incorrect direction fitting caused by the fact that 3D boxes cannot be aligned with the global coordinate system. The first thing to consider is that the application does not use traditional 2D images, but pseudo-2D bird's eye view converted from 3D feature bodies obtained by 3D sparse convolution downsampling. Due to the sparsity and irregularity of the original point cloud, it cannot provide as dense location information as a 2D image. This will cause the proposal to be unable to obtain accurate size information. Specifically, as shown in (a) in Figure 4 , the middle box and the outer box extract the same points, so they have similar feature representations, but the lack of size information will cause different classification and regression targets for the same set of points.

[0115] The application selects the traditional RPN module when generating ROI (region of interest). Two different 3D anchor boxes are designed for objects of each class. And only 0° and 90° directions are evaluated on the bird's eye view. But this method will cause the direction of the fitted rotating object to be wrong, that is, this method can have higher performance when facing objects moving along a straight line. If the object is rotating, the anchor-based method cannot fit the axis-aligned bounding box.

[0116] In order to solve the above problems, the application proposes an IoU-Grid Pooling module. After a series of module processing, a set of weighted key point features and 3D ROI obtained by 3D sparse convolution have been obtained. Considering that the traditional RCNN model ignores the distance information between points, the application uses virtual grid points to utilize this ignored information.

[0117] For each 3D proposal, MxMxM virtual grid points are evenly distributed in length, width and height, and the coordinates of the virtual grid points are normalized with the coordinates of the real points. Then, SA is used for each virtual grid point to select the key point features that need to be aggregated. However, unlike the traditional way, the key point threshold λ and the set abstraction radius r g of the virtual grid point are set. Specifically, all weighted key points within the radius will be aggregated, so for any virtual grid point g iAll can obtain a set of adjacent weighted voxel features,

[0118]

[0119] where p j -g i represents the relative position of the adjacent weighted key points; N' represents the total number of adjacent weighted key points.

[0120] As Figure 5 shown, if any one grid point g i cannot find λ weighted key points in the sphere with r g as the radius, the virtual grid point is deleted. In practical applications, the key point threshold needs to be adjusted according to experience, and the key point threshold λ = 3 selected in this embodiment. In particular, the virtual grid points set by the method of the application are also distributed on the proposal surface, so that the adjacent weighted feature points outside the target proposal 3D bounding box but within the set abstraction radius can be captured through the IoU-GridPooling Moudle. After the virtual grid point screening, the virtual grid point set of each proposal can be obtained by the application,

[0121] G = {g m ∈ R 3 |m ∈ [0, M 3 -1]}

[0122] Then a minimum enclosing rectangle is fitted around all the remaining points in the virtual grid point set, which can realize the correction of the proposal direction and boundary. By interpolating the features in the adjacent weighted voxel feature set, the features of the virtual grid can be obtained,

[0123]

[0124] where d(·) represents the L2 distance.

[0125] The features of all virtual grid points in a proposal are added to the grid feature set by the application, and the ROI features of each proposal are obtained by passing through an MLP with a channel dimension of [C+3, 256, 128, 128] and a global max pool. Then, the Intersection over Union (IOU) estimate of each box is obtained by another MLP. Finally, a 2-layer multi-layer perceptron is used to predict the foreground confidence of the target proposal and optimize the 3D box, respectively.

[0126] In summary, by adding virtual grid points, this paper can obtain accurate boundary information of the proposal by judging whether the number of adjacent weighted key points of the proposal boundary grid point is higher than the key point threshold, and correct the direction of the 3D box by fitting a minimum enclosing rectangle.

[0127] Training target

[0128] The proposed model is trained using 4 terms, including regression (L reg ), classification (L cls ), region of interest (L rpn ) and grid update (L grid ),

[0129]

[0130] where N fg represents the number of anchors generated belonging to the foreground region; λ represents the balance parameter corresponding to different loss functions; λreg=λrpn=λcls=λgrid=1.

[0131] In order to solve the problem of imbalance of generated key points in foreground and background regions, Focal Loss is used as the classification term L cls . While for the frame regression term L reg and the RPN loss term L rpn , Huber Loss is selected.

[0132] For the grid update term, L2 loss function is selected,

[0133]

[0134] where ||·|| represents the L2 norm.

[0135] The APVR method of the embodiment is tested on the data sets Kitti, NuScenes and Waymo.

[0136] Related data set introduction:

[0137] (1) Kitti data set: Kitti has been widely concerned as the most classic 3D object detection benchmark. Kitti data set provides 7481 training samples and 7518 test samples to meet the needs of different object detection such as cars, pedestrians and bicycles. In the Kitti data set, objects will be divided into three levels of easy, medium and difficult according to object size, occlusion degree and truncation level. In most cases, the original training data will be divided into a training set of 3712 samples and a validation set of 3769 samples. However, the present application randomly selects 80% of the samples in the training samples as the training set and 20% of the samples as the validation set.

[0138] (2) NuScenes dataset: The NuScenes dataset, as the latest large-scale autonomous driving dataset, collects 1000 complex driving scenes from Boston and Singapore. These scenes can be split into 700 training scenes, 150 validation scenes, and 150 test scenes. Unlike the Kitti dataset, the NuScenes dataset collects data through 6 multi-view cameras and a 32-line LiDAR.

[0139] (3) Waymo dataset: The Waymo dataset, as the largest open dataset currently, includes 798 training sequences and 202 validation sequences. The dataset contains RGB images obtained by high-resolution cameras and 3D point clouds generated by a 64-line LiDAR. And the official also provides performance subdivision for LEVEL_1 (more than 5 laser radar points in the box) and LEVEL_2 (at least one laser radar point in the box).

[0140] Network architecture:

[0141] The 3D backbone network of this embodiment has 3 levels, and the feature dimensions of them are 16, 32, and 64, respectively. The Manhattan threshold values used in the VPFA module are 2, 4, and 6, respectively. The set abstraction radius used in the original point cloud is (0.4m, 0.8m). In the IOU-Grid Pooling module, 5x5x5 virtual grid points are uniformly sampled for each proposal. And the virtual grid key point threshold λ = 3 and the set abstraction radius rg = 0.6m are set.

[0142] The range of the point cloud in the 3D scene of the KITTI dataset is specified as [(0, 70.4)m, (-40, 40)m, (-3, 1)m], and (0.1m, 0.1m, 0.15m) is used as the voxel size. For the Nuscenes dataset, the 3D point cloud range of the X-axis, Y-axis, and Z-axis is cropped respectively. Among them, the X-axis and Y-axis are both [-49.6, 49.6]m, and the Z-axis is [-5, 3]m. The voxel size is (0.1m, 0.1m, 0.2m). For the Waymo dataset, the detection range of the X-axis and Y-axis is [-75.2, 75.2]m, and the Z-axis is [-2, 4]m. The voxel size is (0.1m, 0.1m, 0.1m).

[0143] Training and inference:

[0144] All models in this embodiment are trained end-to-end using PyTorch. For the KITTI dataset, the network is trained for 100 epochs, and the batch size of the network is 8. While for the Nuscenes dataset, the network is trained for 80 epochs, and the batch size is 6. The learning rate for both datasets is initialized to 3e-3. Then the learning rate is updated by a cosine annealing technique.

[0145] In the inference phase, the IOU threshold is set to 0.7 in the RPN. After optimization, non-maximum suppression (NMS) is used to remove redundant prediction boxes. It is worth noting that different IOU thresholds are designed for class perception predictions of three different classes: cars, pedestrians, and bicycles. The IOU thresholds for cars, pedestrians, and bicycles are 0.7, 0.5, and 0.5, respectively.

[0146] Experimental results on the KITTI dataset:

[0147] The APVR framework of this embodiment is first evaluated through the KITTI dataset. As shown in Table 1, the APVR is compared with several other state-of-the-art methods in the KITTI test set. By fully using the complementary information based on Voxel and Point, the APVR has a significant improvement in performance. In order to compare the performance of the proposed model, the latest Voxel-based method, Point-based method, and RGB and LiDAR fusion-based method are also reported. From the table, it can be observed that: (1) the method of this embodiment is the fastest among all two-stage methods. Specifically, the real-time processing frame rate of the method of this embodiment is more than 3 times higher than that of PV-RCNN. (2) The average precision (AP) of the method of this embodiment is significantly better than that of other methods in all difficulty levels of different categories. For example, the simple, medium, and difficult levels of the car category of the HVPR are improved by 5.07%, 4.25%, and 5.04%, respectively. This proves that 3D target detection by fully using complementary information can effectively improve the performance of the model in complex environments.

[0148] Table 1 Performance comparison on the KITTI test set

[0149]

[0150]

[0151] Figure 6are Example qualitative results of APVR on KITTI dataset. Actual bounding boxes, predicted bounding boxes of pedestrian and car classes are shown respectively. And 2D bounding boxes projected from 3D detection results are also provided.

[0152] Figure 6 Example qualitative results on KITTI dataset

[0153] Experimental results on NuScenes dataset:

[0154] The present embodiment is also tested on the more challenging NuScenes dataset. The effectiveness and generalization of APVR can be further proved by the test results in different datasets. In the NuScenes dataset, the present embodiment uses a new measurement indicator, i.e. NuScenes detection score (NDS).

[0155] Table 2 shows the comparison data between the method of the present embodiment and other latest methods on the NuScenes dataset. Among them, “CV”, “Ped.”, “Moto.”, “Bicy.”, “TC”, “Bar” respectively represent engineering vehicles, three-wheeled vehicles, motorcycles, bicycles, traffic cones, and obstacles. In Table 2, detailed AP data and other indicator data of all classes are given. APVR achieves a substantial improvement in mean Average Precision (mAP) and NDS compared to other latest lidar methods. Specifically, compared with CVCNet, the method of the present embodiment improves the AP of the car class by about 2%, improves the mAP by 2.8%, and improves the NDS by 1.7%.

[0156] Table 2 Performance comparison on NuScenes test set

[0157]

[0158]

[0159] Figure 7 are Example qualitative results of APVR on NuScenes dataset.

[0160] Experimental results on Waymo dataset:

[0161] The APVR is evaluated on LEVEL_1 and LEVEL_2 object and compared with the state-of-the-art methods on the Waymo dataset. In Table 3, the 3D bounding box mean average precision (3D bounding box mean average precision, mAP) and mAP weighted by heading accuracy (mAP weighted by heading accuracy, mAPH) are used for evaluation. The proposed method has significant improvement on both LEVEL_1 and LEVEL_2. Specifically, Table 3 reports the detection results on 3D vehicles and pedestrians classes on the test sequences. Compared with the state-of-the-art CenterPoint method, the proposed method improves 3.8% and 2.7% on mAP for the two difficulty levels of the pedestrian class, respectively. This shows that the proposed method also has good performance on detecting small objects in addition to the vehicle class.

[0162] Table 3 Vehicle and pedestrian class 3D detection results on the Waymo dataset test sequences

[0163]

[0164] Table 4 shows the vehicle class detection results of the proposed method on the validation sequences of the Waymo dataset. The vehicle class detection is performed on the validation sequences by 3D AP and BEV AP. Under the LEVEL_1 3D mAP indicator, the proposed method achieves the state-of-the-art performance with 77.2% mAP. It is noted that the proposed method has significant improvement on both LEVEL_1 and LEVEL_2 in the distance range of 50m-Inf. The high detection performance at a long distance proves that the proposed method can effectively detect objects with sparse points.

[0165] Table 4 Vehicle detection results on the Waymo dataset validation sequences

[0166]

[0167] Ablation experiments:

[0168] The performance of each proposed module is analyzed in detail by showing more experimental data. All models are evaluated on the car class of the val set of the KITTI dataset.

[0169] The VPFN module proposes a new module for local feature fusion using offset and feature matching probability. The performance of VPFN using different configurations is shown in Table 5. It can be found from the table that, compared with the method using traditional SA, the method of adding offset for voxel query around key points can reduce the calculation cost while maintaining the detection accuracy. After adding the feature matching probability, the performance can be observed to be improved, especially on the AP of difficult difficulty, which is increased by 1.91%.

[0170] Table 5 Effect of Voxel-Point Feature Aggregation module on KITTI val set

[0171]

[0172] The KP Analysis module reweights the key point features by the original point cloud matching probability, feature probability and multi-scale matching probability. The number of selected key points in the KITTI dataset is 2048. As shown in Figure 8 , changing the number of sampled key points cannot significantly improve the detection performance. Specifically, when the number of key points is too large, it will not greatly improve the performance, but will greatly increase the calculation amount. Reducing the number of key points will cause the performance to decrease. Therefore, the KP Analysis module is designed to improve the performance.

[0173] Table 6 shows the effect of key point features with different matching probabilities and reweighting on vehicle class detection performance. Compared with PV-RCNN, the method of the present embodiment improves the 3D AP of the vehicle class by 2.93%, 0.87% and 1.16% respectively in simple, medium and difficult difficulty. This shows that combining different information matching probabilities helps to improve the AP performance of the algorithm.

[0174] Table 6 Effect of KP Analysis module on vehicle class detection performance on KITTI val set

[0175]

[0176] The IoU-Grid Pooling module reports the effect of different key point thresholds λ on the PR curve in Figure 9 . The PR curve is a standard method for evaluating the quality of prediction. It can be observed from the PR curve that when λ = 3, the quality of the predicted object is significantly improved. Too large or too small key point threshold will have a negative impact on the prediction quality.

[0177] The effect of virtual grid point interpolation feature and minimum bounding rectangle constraint is shown in Table 7. Without changing the key point threshold, the performance can be further improved by inserting different sub-modules in the IoU-Grid Pooling module.

[0178] Table 7 Sub-module effectiveness investigation

[0179]

[0180] In the ablation experiment, the effectiveness of all independent modules is investigated. In Table 8, the performance gap of the proposed method under different configurations is summarized. Specifically, the independent modules proposed in this embodiment can improve 4.14%, 4.75%, and 3.84% respectively at simple, medium, and difficult difficulty levels, and the real-time processing frame rate can also reach 27.4Hz.

[0181] Table 8 Performance gap using different configurations on KITTI val set

[0182]

[0183] Actual platform research:

[0184] In order to test the effectiveness of the method of this embodiment in real scene, a series of tests are carried out on the actual vehicle platform. It is a platform integrated by multiple sensors. In addition to the basic 16-line radar, Tele-15 radar and millimeter wave radar are also added. The purposes of the three kinds of radars are different, among which the basic 16-line radar is mainly responsible for the detection task; the Tele-15 radar is used as an additional safety redundancy to increase the safety of the vehicle; the millimeter wave radar is responsible for measuring the speed and distance of the vehicle in front.

[0185] On the experimental platform, the performance of APVR and existing 3D detection methods in real complex environment is mainly tested. Figure 10 The 3D mAP and bird's eye view mAP gap of the method of this embodiment and the most representative PointPillar, PV RCNN in different distance ranges are shown. It can be observed from the figure that compared with PV RCNN, the 3D mAP and bird's eye view mAP of the method of this embodiment can be improved by about 2% in all distance ranges.

[0186] In order to intuitively show the advantages of the proposed method, the sensor factor is also considered. As can be seen from Table 9, the method of this embodiment is higher than other comparative methods in the detection of different vehicle categories, which are 88.25%, 81.26%, and 75.38% respectively. These data prove that the method in this paper can still maintain an advantage in different sensor methods.

[0187] Table 9. Comparison of 3D detection performance for cars, vans and buses

[0188]

[0189] Experiments on three public datasets demonstrate that the APVR of the present embodiment has good generalization and portability. And compared with other 3D target detection methods, it has significant improvement in detection accuracy and computational efficiency.

Claims

1. A detection method for improving the efficiency of automatic driving three-dimensional target detection, characterized in that, The method comprises the following steps: (1) obtaining original point cloud data to be detected; (2) Extract a set of key points P from the original point cloud using the farthest point sampling method, where the key points are p i , i is the serial number of the key points; (3) the key point p to be queried i is quantified into a corresponding voxel, and then whether a non-empty voxel around the key point is a neighboring voxel of the key point p to be queried is determined by calculating the Manhattan distance between the non-empty voxel and the key point voxel, so as to obtain a neighboring voxel feature set of the key point p i wherein K represents a current down-sampling multiple;​ (4) Calculate the matching probability of the point cloud-based feature of the key point p i and the voxel-based feature in its adjacent voxel feature set, select the most similar k voxel features according to the matching probability, aggregate them with the corresponding matching probability, perform feature enhancement, and finally generate the feature of the key point p i through the PointNet-block method K is 1x, 2x, 4x; (5) by concatenating the original point cloud features Multi-scale voxel aggregated features and the bird's eye view features to get the original keypoint features (6) respectively calculate keypoint features and its neighboring original point cloud features multi-scale voxel aggregated features feature matching probability mean, and the predicted foreground probability is weighted with the initial keypoint feature to obtain updated keypoint features (7) Uniformly divide each region of interest by a set of virtual grid points, and set the key point threshold λ and the set abstraction radius r g Screen the grid points; correct the direction and boundary of the proposal by fitting the minimum enclosing rectangle and the weighted key point features to obtain the corrected 3D box; (8) repeating steps (3) to (7) until all key points are traversed to obtain a final three-dimensional target detection result.

2. The detection method for improving the efficiency of automatic driving three-dimensional target detection according to claim 1, characterized in that, Step (3) comprises: The Kth level voxel feature set and its corresponding actual coordinate obtained by the 3D sparse convolution are expressed as: Wherein, i represents the number of non-empty voxels in the Kth level; Taking the key point to be queried as the origin of the local coordinate system, and adding 3 offsets to the non-empty voxels around it, Then the Manhattan threshold D is sampled K For any keypoint p i A neighboring voxel feature set is obtained, namely: wherein, represents the relative position of the semantic voxel feature; Kj represents the jth adjacent non-empty voxel of the key point p i in the Kth level; represents the Manhattan distance between the semantic voxel feature and the corresponding key point, 3.The detection method for improving the efficiency of automatic driving three-dimensional target detection according to claim 1, characterized in that, Step (4) comprises: Key point p i The matching probability of the point cloud based feature of the key point p with the voxel based feature in its neighboring voxel feature set is where f(V n , p i ) denotes the n-th voxel-based similarity feature of the key point p i . According to the matching probability, the most similar k voxel features are selected, and the corresponding matching probability is aggregated, and finally the k aggregated voxel features are generated by PointNet-block to generate the key point p i features, Wherein, M(·) represents a multi-layer perceptron network used to encode the key point voxel feature; max(·) represents the maximum channel pooling operation.

4. The detection method for improving the efficiency of automatic driving three-dimensional target detection according to claim 1, characterized in that: The key point p obtained in step (5) i characterized in that 5.The detection method for improving the efficiency of automatic driving three-dimensional target detection according to claim 1, characterized in that: In step (6), for any key point p i A neighboring raw point cloud set can be obtained, wherein, represent a point cloud based feature and actual coordinates of the original point cloud, respectively; represent a relative position of the original point cloud; r raw denotes a set radius range; C raw ,F raw represent a feature set of the original point cloud and its corresponding actual coordinates, respectively; N represents a number of adjacent original point clouds; Computing key points p i The matching probability of a point cloud based feature of a key point p with its neighboring point cloud based features is computed as follows, The feature probability of the key point, i.e. the prediction probability of the key point belonging to the foreground, is obtained by a 3-layer multi-layer perception network and a Sigmod function The feature matching probability mean value of the key point and the adjacent original point cloud is calculated by SA The feature matching probability mean value of the key point and the adjacent voxels in multiple scales is calculated by VPFA respectively Then the weight of the key point from the foreground region is improved by the average and weighting operation, and the key point feature after reweighting is expressed as: 6.The detection method for improving the efficiency of automatic driving three-dimensional target detection according to claim 1, characterized in that, Step (7) comprises: For each 3D proposal, MxMxM virtual grid points are set in the proposal, and the coordinates of the virtual grid points are normalized with the coordinates of the real points; then for each virtual grid point, SA is used to select the key point features that need to be aggregated, and the key point threshold λ and the set abstraction radius r of the virtual grid point are set g All weighted key points within the set abstraction radius are aggregated, and for any virtual grid point g i An adjacent weighted voxel feature set can be obtained where p j -g i denotes the relative position of the neighboring weighted keypoint; N' denotes the total number of neighboring weighted keypoints; If any one grid point g i cannot find λ weighted key points in the sphere with r g as the radius, the virtual grid point is deleted; The virtual grid points set are also distributed on the proposal surface, which can capture adjacent weighted feature points outside the target proposal 3D bounding box but within the set abstraction radius; Obtain the virtual grid point set of each proposal, G = {g m ∈ R 3 |m ∈ [0, M 3 - 1]} Then fit a minimum enclosing rectangle around all remaining points in the virtual grid point set, realize the correction of the proposal direction and boundary, and obtain the features of the virtual grid by interpolating the features in the adjacent weighted voxel feature set, Wherein, d(·) represents the L2 distance; Add the features of all virtual grid points in a proposal to the grid feature set, pass it through an MLP with channel dimensions of [C+3, 256, 128, 128] and global maximum pooling to obtain the region of interest feature of each proposal, then pass it through another MLP to obtain the intersection over union estimate of each box, and finally use a 2-layer multi-layer perceptron to predict the foreground confidence of the target proposal and optimize the 3D box respectively.

7. The detection method for improving the efficiency of automatic driving three-dimensional target detection according to claim 6, characterized in that: The key point threshold λ=3.

Images