A Real-time 3D Object Detection Method for Autonomous Driving Combining Point Cloud Shape Features

Abstract

The present invention discloses a real-time three-dimensional object detection method for autonomous driving that combines point cloud shape features. The method uses a triangular network in computer graphics to represent local features, thereby flexibly presenting complex local voxel features. In the present invention, for each point inside the voxel, the K-nearest neighbor algorithm is used to find the K nearest points, and these points are used to construct an umbrella-shaped surface centered on this point. The surface feature of each point is constructed using information such as the normal vector direction of the surface and the center coordinates of the surface. A surface feature is generated using a fully connected neural network and a max pooling layer, so as to use this surface feature to represent the local shape around the point cloud. The present invention uses the umbrella-shaped surface construction to model the point cloud shape around each point cloud, thereby supplementing the missing local shape learning features and reducing the phenomena of false detection and missed detection in the autonomous driving perception task. While completing the local shape modeling of the point cloud, the present invention ensures the real-time performance of the algorithm.

Description

Technical Field

[0001] The present invention relates to a three-dimensional object detection method in the field of autonomous vehicles, and particularly to a real-time three-dimensional object detection method for autonomous driving that combines point cloud shape features. Background Art

[0002] Environmental perception, as one of the most important modules in autonomous driving, plays a crucial role in ensuring vehicle safety. Among them, three-dimensional object detection obtains the position, size, and category information of objects through various sensors (lidar, millimeter-wave radar, cameras, etc.). Lidar obtains a large-scale unordered point cloud by performing high-speed scanning and sampling on surrounding objects, thereby obtaining a simple and accurate three-dimensional scene representation, and is one of the most important sensors for autonomous driving perception solutions. With the development of deep neural networks, the current mainstream three-dimensional object detection algorithms based on lidar point clouds are mainly divided into detection methods based on raw point clouds and detection methods based on voxels.

[0003] Among them, the detection method based on raw point clouds usually performs farthest point sampling and feature aggregation on all the point clouds obtained by lidar scanning: the sampling part extracts some point clouds from all the point clouds, and the feature aggregation aggregates the feature information of these extracted point clouds and the surrounding point clouds, so that each point cloud aggregates the feature information of the surrounding point clouds. However, the method based on raw point clouds will cause large memory and computational overheads, and does not meet the requirements for the real-time performance of detection algorithms in autonomous driving scenarios. With the development of deep neural networks and the proposal of three-dimensional sparse convolution, the current voxel-based detection methods can more efficiently extract the features of point clouds directly from the voxels generated by point clouds, such as methods like VoxelNet and PointPillar. These methods usually use information such as the coordinates, intensities, and offsets relative to the voxel center of the point clouds within each voxel as voxel features, and then use three-dimensional sparse convolution for downsampling of the voxel features, and finally obtain the classification information and regression information of the three-dimensional object bounding boxes. This method very efficiently extracts the features of point clouds from voxels rather than raw point clouds, greatly reducing the memory and computational overheads of the algorithm, making it possible to deploy real-time algorithm detection on the in-vehicle platforms of autonomous vehicles. However, although this method is relatively simple and intuitive, it cannot well represent the information of the local shape of point clouds by only using local coordinates, ignoring the interaction between point clouds and the local shape information formed by adjacent point clouds, thereby restricting the further improvement of the accuracy of three-dimensional detection algorithms in autonomous driving.

[0004] In order to enable the 3D object detection algorithm in autonomous driving to obtain both the detection accuracy of the original point cloud algorithm and the detection efficiency of the voxel algorithm, the present invention proposes a point cloud local shape feature extraction scheme, which can be combined with any voxel-based detection scheme, so that the local shape information of the point cloud can be fused into the voxel features of the point cloud, helping the algorithm to further improve the detection accuracy. Summary of the Invention

[0005] To solve the above problems existing in the prior art, the present invention aims to design a real-time 3D object detection method for autonomous driving that combines point cloud shape features and can avoid point cloud feature loss and improve detection accuracy.

[0006] To achieve the above object, the basic idea of the present invention is as follows:

[0007] To solve the problems existing in the prior art, the present invention uses a triangular network in computer graphics to represent local features, so as to flexibly present complex local voxel features. For each point inside the voxel, the K-nearest neighbor algorithm is used to find the K nearest points in terms of distance. These points are used to construct an umbrella-shaped surface centered on this point. The surface feature of each point is constructed using information such as the normal vector direction of the surface and the center coordinates of the surface. A surface feature is generated using a fully connected neural network and a max pooling layer, so as to represent the local shape around the point cloud using this surface feature.

[0008] The technical solution of the present invention is as follows: A real-time 3D object detection method for autonomous driving that combines point cloud shape features, comprising the following steps:

[0009] A. Allocate point clouds for any voxel in the 3D scene

[0010] All the point clouds P collected by the vehicle-mounted lidar are allocated to each voxel. When the number of point clouds in the voxel exceeds the preset maximum number of point clouds N, N point clouds are randomly sampled. When the number of point clouds is less than N, 0 is used to fill up the number of point clouds to N, so that each non-empty voxel has the same number of point clouds N. Each point cloud contains three-dimensional position coordinate information c and point cloud reflection intensity information i.

[0011] B. Find adjacent points for any point cloud in the voxel and construct a representative surface

[0012] Taking any point cloud in the voxel as the origin, use the K-nearest neighbor algorithm to find the K adjacent point clouds with the closest geometric distance to it. These K adjacent point clouds are sorted clockwise according to the angle relative to the current point cloud in the polar coordinate system, and are successively paired with the current point cloud to construct a triangular grid, so that each point cloud corresponds to K triangular grids. Calculate the normal vector direction v norm of the grid corresponding to each point cloud in the voxel and the grid center coordinate c center, finally, K normal vectors and grid center coordinates are generated for each point cloud, and the specific formulas are as follows:

[0013] v norm =(c k -c current )×(c k-1 -c current ) 0 ≤ k ≤ K

[0014]

[0015] In the formula: c current represents the coordinate of the current point cloud, and c k represents the coordinate of the k-th neighboring point

[0016] C. Extract features of the representative surface inside the voxel

[0017] Generate the normal vector direction v of the grid for each point norm , the grid center coordinate c center and the polar coordinate c of the grid center sphere Use a fully connected neural network and max pooling to generate an umbrella-shaped surface feature vector f representing the local shape for each point cloud umbella .

[0018] c sphere = xyz2sphere(c center )

[0019] f umbella = pooling{MLPs(v norm , c center , c sphere )}

[0020] In the formula: xyz2sphere is the polar coordinate conversion function, pooling is the max pooling function, and MLPs represents the fully connected neural network.

[0021] D. Calculate the voxel features containing the local shape of the point cloud

[0022] Use the global coordinate c, reflection intensity i, coordinate c relative to the clustering center cluster , and coordinate c relative to the voxel center voxel of each point inside the voxel to construct the point cloud coordinate feature f inside the voxel coordinate , and combine it with the local shape feature f umbella to further perform feature set abstraction to obtain the features of each point, and compress the features of N point clouds into voxel features through the max pooling layer to obtain the voxel feature F containing the local shape of the point cloud voxel , which is expressed as follows:

[0023] fcoordinate =(c,i,c cluster ,c voxel )

[0024] F voxel =pooling{MLPs(f coordinate ,f umbella )}

[0025] E. Detecting 3D targets

[0026] Project the obtained voxel features into the bird's-eye view perspective, and use a deep convolutional neural network to further downsample and extract features from the voxel features, and directly predict the position, size, and category of 3D targets on the downsampled bird's-eye view features. The 3D targets include cars, pedestrians, and cyclists, thereby completing the understanding of the surrounding environment by the autonomous driving vehicle and providing accurate detection results for subsequent autonomous driving.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] 1. Due to the disorder and irregularity of point clouds, previous voxel-based 3D object detection methods independently learned from point clouds and used symmetric functions to obtain global information. However, this method did not perceive the local shape of point clouds, and the local shape is crucial for the learning of point clouds in 3D object detection. The present invention uses an umbrella-shaped surface structure to model the shape of point clouds around each point cloud, thus well complementing the missing local shape learning features of previous methods and reducing the phenomena of false detection and missed detection in the autonomous driving perception task.

[0029] 2. In order to obtain information from the local structure of point clouds, some previous detection methods based on raw point clouds indirectly learned from the point cloud shape by performing multiple transformations on the point clouds through additional structures such as graph neural networks and distance relationships between point clouds. These artificially set components would cause high memory and computational overheads and result in information loss during the transformation process. The present invention represents the local shape by constructing a surface from the point clouds within the voxels, and can obtain a significant improvement in detection performance with a relatively small increase in parameters, while ensuring the real-time performance of the algorithm when completing the local shape modeling of point clouds, which is conducive to being deployed on the embedded platform of autonomous driving vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flowchart of the present invention.

[0031] Figure 2 is a schematic diagram of the local surface construction of the present invention.

[0032] Figure 3 is a schematic diagram of the local grid center and normal vector direction. Detailed implementation mode

[0033] The present invention will be further described below with reference to the accompanying drawings. As Figure 1 shown, the present invention first rasterizes all the point clouds scanned by a single-frame lidar to obtain a series of regular point cloud voxels, and then constructs a surface and abstracts local features for the point clouds inside each voxel to obtain shape features, and fuses the features with the point cloud coordinates, and finally aggregates the shape features of all the point clouds to obtain voxel features with fused shape information.

[0034] Among them, the method for constructing the surface inside the voxel is as Figure 2 shown. This method finds K neighboring point clouds for any point cloud and constructs a local umbrella-shaped surface for it using these neighboring point clouds. The surface contains K triangular meshes.

[0035] The feature extraction for each triangular mesh is as Figure 3 shown. By calculating the central coordinate c center 、central polar coordinate c sphere and normal vector direction v norm of each triangular mesh, and using a fully connected neural network and a max pooling layer to aggregate the features of K neighboring points for any point cloud, so as to obtain the local shape feature f umbella of any point cloud. Finally, the global coordinate c, reflection intensity i, relative voxel center coordinate c cluster 、relative clustering center coordinate c voxel of any point cloud are used to construct the coordinate feature f coordinate of the point cloud. Finally, the local shape features and coordinate features of all the point clouds in any voxel are merged, and a fully connected neural network and a max pooling layer are used again to aggregate all the point cloud features in the voxel to obtain voxel features F containing point cloud shape features and point cloud coordinate features voxel . Finally, these regular voxel features are downsampled and feature-extracted using a deep convolutional neural network, and a head network is used on the feature map in the bird's-eye view to complete the regression and classification of the position, size, and category information of three-dimensional targets (vehicles, pedestrians, cyclists, etc.), so as to obtain the perception of obstacles around the autonomous vehicle.

[0036] The present invention is not limited to this embodiment, and any equivalent conceptions or changes within the technical scope disclosed by the present invention are included in the protection scope of the present invention.

Claims

1. A real-time 3D object detection method for autonomous driving that combines point cloud shape features, characterized in that: It includes the following steps: A. Assign point clouds to any voxel in the 3D scene All point clouds P collected by the vehicle-mounted lidar are assigned to each voxel. When the number of point clouds in a voxel exceeds the preset maximum number of point clouds N, N point clouds are randomly sampled. When the number of point clouds is less than N, 0 is used to fill up the number of point clouds to N, so that each non-empty voxel has the same number of point clouds N. Each point cloud contains three-dimensional position coordinate information c and point cloud reflection intensity information i; B. Find adjacent points for any point cloud in the voxel and construct a representative surface Taking any point cloud within the voxel as the origin, use the K-nearest neighbor algorithm to find the K adjacent point clouds with the closest geometric distance to it; sort these K adjacent point clouds in clockwise order according to the angle relative to the current point cloud in the polar coordinate system, and construct triangular meshes with the current point cloud pairwise in turn, so that each point cloud corresponds to K triangular meshes, and calculate the normal vector direction v of the mesh corresponding to each point cloud within the voxel norm and the grid center coordinate c center , finally generate K normal vectors and grid center coordinates for each point cloud, and the specific formula is as follows: v norm = (c k - c current ) × (c k-1 - c current ) 0 ≤ k ≤ K where: c current represents the coordinates of the current point cloud, and c k represents the coordinates of the k-th neighboring point C. Extract features of the representative surface inside the voxel Generate the normal vector direction v of the grid for each point nom , the grid center coordinate c center and the polar coordinates c of the grid center sphere Use a fully connected neural network and max pooling to generate an umbrella-shaped surface feature vector f that characterizes the local shape for each point cloud umbella ; c sphere = xyz2sphere(C center ) f umbella = pooling{MLPs(v norm , c center , c sphere )} In the formula: xyz2sphere is a polar coordinate conversion function, pooling is a max pooling function, and MLPs represents a fully connected neural network; D. Calculate the voxel features containing the local shape of the point cloud Using the global coordinates c, reflection intensity i, and coordinates c relative to the clustering center of each point within the voxel cluster , and coordinates c relative to the voxel center voxel Construct the point cloud coordinate feature f inside the voxel coordinate , and combine with the local shape feature f umbella Further perform feature set abstraction to obtain the features of each point, and compress the features of N point clouds into voxel features through the max pooling layer to obtain the voxel feature F containing the local shape of the point cloud voxel , which is expressed as follows: f coordinate =(c,i,c cluster ,c voxel ) F voxel = pooling{MLPs(f coordinate , f umbella )} E. Detect three-dimensional targets The obtained voxel features are projected onto the bird's-eye view. A deep convolutional neural network is used to further downsample and extract features from the voxel features, and the position, size, and category of the three-dimensional targets are directly predicted on the downsampled bird's-eye view features. The three-dimensional targets include cars, pedestrians, and cyclists, thus completing the understanding of the surrounding environment by the autonomous driving vehicle and providing accurate detection results for subsequent autonomous driving.

Images