A three-dimensional target detection method based on point cloud structure perception

Abstract

The application discloses a three-dimensional target detection method based on point cloud structure perception, comprising the following steps: (1) obtaining original point cloud data; (2) performing voxel feature extraction by using a feature extraction network to obtain bird's eye view features; the feature extraction network is composed of a voxel space division layer, a VFE feature extraction layer and a sub-manifold sparse convolution layer; (3) providing an auxiliary training network for supervising the feature extraction network in the training process to learn foreground points and target center points; the auxiliary training network is divided into three parts: voxel coordinate conversion, foreground point segmentation and center coordinate prediction; (4) inputting the bird's eye view features obtained in the step (2) into a detection network to generate a three-dimensional target detection frame; the detection network is composed of a 2D convolution layer, a region proposal network and a position alignment module. The application improves the detection precision while ensuring real-time performance.

Description

Technical Field

[0001] This invention relates to three-dimensional target detection technology, specifically to a three-dimensional dynamic target detection method, and more particularly to a three-dimensional target detection method based on point cloud structure perception. Background Technology

[0002] In the field of autonomous driving, image-based 2D object detection and instance segmentation methods are relatively mature and widely used in practice. With the development of convolutional neural networks in 3D space in recent years, point cloud-based 3D object detection technology has gradually become a hot research topic, and its detection performance has also received widespread attention.

[0003] Current point cloud-based 3D object detection methods mainly fall into two categories: single-stage detection and two-stage detection. Single-stage detection methods process point cloud data using fully convolutional neural networks. They employ 2D convolutional networks on bird's-eye view (BEV) or frontal panoramic images, or utilize voxel grids for 3D convolutional networks to extract features. VoxelNet, a representative example of the voxel grid method, divides the point cloud into voxels, uses the classic PointNet network for feature extraction, and fuses features from multiple scales of voxel data. Kuang et al. combined the Feature Pyramid Network (FPN) with the VoxelNet method, proposing a single-stage detection method that can fuse feature maps of different scales from top to bottom. The advantage of single-stage detection is its high real-time performance; however, multiple downsampling of the feature maps leads to the loss of point cloud data and structural information, thus affecting detection accuracy.

[0004] Two-stage detection methods typically select a region of interest (ROI) in the first stage to generate a coarse pre-selected bounding box, and then use more detailed spatial information in the second stage to achieve more accurate localization and detection, such as performing secondary feature extraction on the ROI or processing each point in the region. This approach obviously increases computational costs, making it difficult for two-stage methods to meet real-time requirements.

[0005] If we can improve the model's ability to perceive point cloud structures based on the single-stage detection method, we can improve detection accuracy while maintaining the real-time performance of the method.

[0006] Furthermore, considering that the classification confidence of a target is related to the position of its features in the feature map, and that the detection boxes obtained by previous single-stage methods deviate from their current positions, resulting in a mismatch between the classification feature map and the localization feature map at corresponding locations, the commonly used non-maximum suppression (NMS) cannot effectively solve this problem. Jiang et al. improved NMS by using a new method to calculate the localization feature map and replacing the NMS filtering threshold with the product of the classification confidence and the localization confidence. This method can optimize the problem to some extent; however, it does not constrain the consistency between the two feature maps. Therefore, how to achieve more reliable confidence prediction is also a problem that needs to be solved. Summary of the Invention

[0007] The purpose of this invention is to provide a three-dimensional target detection method based on point cloud structure perception, which can ensure the real-time performance and detection accuracy of the method, and achieve more reliable confidence prediction.

[0008] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a three-dimensional target detection method based on point cloud structure perception, comprising the following steps:

[0009] (1) Obtain raw point cloud data;

[0010] (2) Voxel features are extracted using a feature extraction network to obtain bird’s-eye view features; the feature extraction network consists of a voxel space partitioning layer, a VFE feature extraction layer and a submanifold sparse convolutional layer.

[0011] (3) Provide an auxiliary training network to supervise the feature extraction network during training and learn the foreground points and target center points; the auxiliary training network is divided into three parts: voxel coordinate transformation, foreground point segmentation and center coordinate prediction.

[0012] (4) Input the bird's-eye view features obtained in step (2) into the detection network to generate a three-dimensional target detection box. The detection network consists of a 2D convolutional layer, a region proposal network and a position alignment module.

[0013] In the above technical solution, step (2) specifically includes:

[0014] (2-1) Divide the voxel space:

[0015] Construct a 3D spatial coordinate system along the Z, X, and Y axes in the point cloud space, and let the ranges of the three directions be H, D, and W, respectively, and the sampling step size be d. z ,d x ,d y Each coordinate axis is divided into H', D', and W', and the entire point cloud space is divided into N equally sized voxels.

[0016]

[0017] Introduce a reference threshold T. If the number of points within a certain voxel is greater than T, then use a random sampling method to select T points and remove the redundant points.

[0018] Encode each voxel using It means that p i This represents the X, Y, Z coordinates of the i-th point in the non-empty voxel and the reflectance r of that point. i Let t represent the number of points contained in each voxel, t∈(0,T). The mean of all points in each voxel is taken as the local centroid, denoted as V. c =(v x ,v y ,v z ), calculate p i The offset relative to the centroid is denoted as V. off =(x i -v x ,y i -v y ,z i -v z This yields the seven-dimensional input features for each voxel.

[0019] (2-2) VFE feature extraction:

[0020] First, use a fully connected layer to connect V in The mapping is 32-dimensional, so each voxel has t 32-dimensional feature vectors. These are then input into a max pooling layer to obtain the global variables for that voxel, and concatenated into the feature vectors V at each point. in After that, each point now has 64-dimensional features. The above process is repeated to obtain t 128-dimensional feature vectors for the voxel. Finally, max pooling is used to obtain the 128-dimensional global features of the voxel.

[0021] (2-3) Sparse convolution of submanifolds:

[0022] The input and output of sparse convolution are both sparse feature maps. A hash table is constructed to store the positions of values ​​in the feature maps. Let the input feature map be F. in There are m elements with values ​​in F. in Encoded as an m×n feature matrix Q in and a hash table H in Where n represents the dimension of the feature vector, Q in F is stored in All non-zero eigenvectors, H in It is a hash table composed of key-value pairs, where each key-value pair represents the row index of the feature matrix and the corresponding element in F.in The row index is an integer in the range [0, m-1]; the sparse feature map F output after convolution is... out Also coded as Q out and H out H out =H in Q out It is an m×l matrix, where l refers to the dimension of the output feature vectors;

[0023] set up

[0024]

[0025] Among them, A i The dimension of the input, B i The dimension of the output, A i and B i Satisfy constraints

[0026]

[0027] Where f represents the kernel size, s represents the stride, and p represents zero-value padding.

[0028] Due to F in If the tensor is d-dimensional, then the receptive field size of the convolution kernel is a d-dimensional cube f with side length f. d Sparse convolution will have F in A certain element f in =F in (a1,a2,…,a d Centered on n0, the surrounding f d Zero-padding is performed at all positions of null values ​​in the space, and then convolution is performed with the convolution kernel. Let F... in The region to be zero-padding is D zero , will F out The corresponding area is forcibly cleared to zero.

[0029] In the preferred technical solution, in step (2-1), the reference threshold is an integer between 30 and 50.

[0030] In the above technical solution, step (3) specifically includes:

[0031] (3-1) Voxel coordinate transformation:

[0032] First, convert the non-zero voxel indices obtained in step (2) into coordinates in point cloud space. The features of each voxel are converted into point form, denoted as P = {(f i ,p i f | i = 1, ..., M}, where f i p is the eigenvector. iThis represents the coordinates of a point in the point cloud space, with a total of M voxel indices.

[0033] With {p i Centered on |i=1,…,M}, we aggregate all points within a certain radius around it and interpolate the voxel features. Specifically, let Ψ(p i () represents a spherical region. Different spherical radii are set in each layer of the convolutional downsampling network, and p is selected. i The reciprocal of the distance between all points within the radius is used as the weighted average. Let the weighting characteristic be... The feature vector of each point can then be obtained by the following formula.

[0034]

[0035] Finally, the weighted features obtained from different radii are concatenated and stacked.

[0036] (3-2) Foreground point segmentation:

[0037] The weighted features with different radii obtained in step (3-1) are input into a two-layer multilayer perceptron network to obtain the weights of the foreground points. The first layer of the multilayer perceptron network includes two convolution processes. The first convolution is performed with the input features using 128 convolution kernels of size 1×128×1, resulting in n 1×128 feature vectors. The second convolution is performed with 256 convolution kernels of size 1×1×128, resulting in n 1×256 feature vectors. Then, a dimension transformation is used to transform the feature vectors into n×256. The second layer of the multilayer perceptron network has two convolution processes. The first convolution is performed with the input features using 128 convolution kernels of size 1×256×1, resulting in n 1×128 feature vectors. The second convolution is performed with one convolution kernel of size 1×1×128, resulting in n 1×1 feature vectors. Finally, a dimension transformation is used to transform the foreground point prediction vectors into n×1 dimensions.

[0038] When the input data contains multiple points, each point is multiplied by the same weight, with the weight range being (0,1), and finally multiplied by the feature vector to obtain the final feature of that point;

[0039] (3-3) Center point learning:

[0040] During the training process, set Let Δp be the average distance between the center point of the ground truth bounding box and the points within the box, and let Δp be the average distance between the voxel coordinates mapped to point cloud coordinates and the center point. Smooth-l1 is used as the loss function to achieve convergence of positional bias.

[0041]

[0042] Where Npos The number of points is represented by χ[·]. The indicator function means that the position of the voxel coordinates mapped to the point cloud must be within the ground truth bounding box.

[0043] Further technical solutions include implementing step (3) during training to conduct auxiliary training, and shutting down the auxiliary training network during evaluation.

[0044] In the above technical solution, step (4) specifically includes:

[0045] The bird's-eye view features obtained in step (2) are input into the detection network. The generation of detection boxes is to score the feature information of the bird's-eye view. Each point in the image represents the feature obtained along the Z-axis in the original point cloud space. The bird's-eye view is processed using a 2D convolutional neural network to obtain a probability score map and a localization score map. Each point in the bird's-eye view generates a detection box with 6 angles in the localization score map, namely 0°, 30°, 60°, 90°, 120°, and 150°. Each detection box consists of seven parameters (x, y, z, h, w, l, r). The first three parameters represent the coordinates of the X, Y, and Z axes in 3D space, and the last four parameters represent the height, width, length, and skew angle of the detection box.

[0046] The location alignment module aligns the classification confidence score and the detection box by performing spatial transformation on the feature map. Specifically, for a 3D detection box, let the corresponding classification confidence score be R. p The 3D detection box can be compressed into a 2D detection box from the BEV perspective through dimensional transformation. K points are uniformly sampled within this 2D detection box, and the feature vectors corresponding to these sampling points in the BEV feature map are extracted. Let the coordinates of the kth sampling point in the BEV view be (x...). k ,y k If the sampling point is located in the BEV feature map, then the coordinates of that sampling point are:

[0047]

[0048] And at this time (m k ,n k The value (m) is not necessarily an integer, therefore, features of surrounding points are selected and interpolated to obtain (m). k ,n k The characteristics of )

[0049]

[0050] b(i,j,m,n)=max(1-|im|,0)×max(1-|jn|,0) (13)

[0051] in, This means rounding down, and we need to get (m)k ,n k The coordinates of the four points in the neighborhood, i.e. In the formula, the index ranges of i and j correspond to the coordinate ranges of the four coordinate points and the target point (m). k ,n k The weight of ) The feature vectors representing the four coordinate points in the BEV feature map are used to align classification confidence and location confidence by selecting sampling points in the detection box and mapping them to the feature map, and by using interpolation.

[0052] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:

[0053] 1. This invention adopts a single-stage detection framework. By setting up an auxiliary learning network, voxels are mapped to the point cloud space to perform foreground point segmentation and center point learning, thereby enhancing the perception of the point cloud structure and improving detection accuracy while ensuring real-time performance.

[0054] 2. This invention introduces submanifold sparse convolution, which improves convolution efficiency by reducing computation in null regions, thereby further improving detection speed.

[0055] 3. A position alignment module is provided in the detection network to perform sampling and interpolation operations in the bird's-eye view, and to fuse the features of surrounding points to further improve the accuracy of detection.

[0056] 4. The present invention was compared with other excellent methods on the Kitti, Waymo, and Nuscene datasets. The accuracy of the method for detecting 3D dynamic targets reached 92.12%. The ablation experiments and accuracy detection results on the physical vehicle platform show that the method of the present invention has good portability, strong generalization ability and high detection accuracy. Attached Figure Description

[0057] Figure 1 This is a method framework diagram of Embodiment 1 of the present invention;

[0058] Figure 2 This is a schematic diagram of the VFE feature extraction layer in Example 1;

[0059] Figure 3 This is a schematic diagram of the submanifold sparse convolutional network in Example 1;

[0060] Figure 4 This is a schematic diagram of feature interpolation and cascaded network in Example 1;

[0061] Figure 5 This is a schematic diagram of the foreground point segmentation network in Example 1;

[0062] Figure 6This is a schematic diagram of the position alignment module in Embodiment 1;

[0063] Figure 7 This is a schematic diagram of the weight selection for auxiliary tasks in Example 1;

[0064] Figure 8 This is the recall-precision curve from Example 1;

[0065] Figure 9 This is a visualization of the detection results of the Kitti dataset in Example 1;

[0066] Figure 10 This is a comparison of the results of the position alignment module with different numbers of sampling points in Example 1;

[0067] Figure 11 It is the loss function curve in Example 1;

[0068] Figure 12 This is a comparison of the target detection accuracy of the three methods in Example 1 for different categories;

[0069] Figure 13 This is a comparison of the target detection accuracy of the three methods at different distances in Example 1;

[0070] Figure 14 This is the visualization result of the physical platform in Example 1. Detailed Implementation

[0071] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0072] Example 1: A 3D target detection method based on point cloud structure perception, see appendix. Figure 1 This includes the following steps:

[0073] 1. Obtain raw point cloud data;

[0074] 2. A feature extraction network is used to extract voxel features to obtain bird's-eye view features; the feature extraction network consists of a voxel space partitioning layer, a VFE feature extraction layer, and a submanifold sparse convolutional layer;

[0075] 3. An auxiliary training network is provided to supervise the feature extraction network during training and to learn the foreground points and target center points; the auxiliary training network consists of three parts: voxel coordinate transformation, foreground point segmentation, and center coordinate prediction;

[0076] 4. Input the bird's-eye view features obtained in step 2 into the detection network to generate a 3D target detection box. The detection network consists of a 2D convolutional layer, a region proposal network, and a position alignment module.

[0077] The specific steps for voxel feature extraction are as follows:

[0078] The voxel-based feature extraction network consists of three parts: a voxel space partitioning layer, a VFE feature extraction layer, and a submanifold sparse convolutional layer.

[0079] (1) Divide the voxel space layer

[0080] Construct a 3D spatial coordinate system along the Z, X, and Y axes in the point cloud space, and let the ranges of the three directions be H, D, and W, respectively, and the sampling step size be d. z ,d x ,d y Each coordinate axis is divided into H′, D′, and W′, therefore the entire point cloud space is divided into N equally sized voxels, where

[0081]

[0082] Taking Kitti's 3D point cloud dataset as an example, in vehicle detection, h = [-3, 1], d = [-40, 40], w = [0, 70.4], and the size of each voxel is (d z ,d x ,d y = (0.4, 0.2, 0.2). In pedestrian and cyclist detection, h = [-3, 1], d = [-20, 20], w = [0.48], and the size of each voxel is (d z ,d x ,d y = (0.4, 0.2, 0.2).

[0083] Typically, high-resolution point clouds obtained from LiDAR scans exceed 100K points, and these points are not uniformly distributed in 3D space. Processing every point in a voxel within a dense point cloud not only increases computational costs but also introduces detection errors due to varying point cloud densities in different regions. To address this issue, a reference threshold T is introduced. If the number of points within a voxel exceeds T, a random sampling method is used to select T points and discard redundant points. For example, in the Kitti dataset, T=35 for vehicle detection and T=45 for pedestrian detection.

[0084] Encode each voxel using It means that p i This represents the X, Y, Z coordinates of the i-th point in the non-empty voxel and the reflectance r of that point. i Let t represent the number of points contained in each voxel, t∈(0,T). The mean of all points in each voxel is taken as the local centroid, denoted as V. c =(v x ,v y ,v z ), calculate p iThe offset relative to the centroid is denoted as V. off =(x i -v x ,y i -v y ,z i -v z This allows us to obtain the seven-dimensional input features of each voxel.

[0085] (2) VFE Feature Extraction Layer

[0086] VFE feature extraction layer, such as Figure 2 As shown, the purpose of this layer is to process the input features V obtained above. in This process involves processing to avoid potential information loss during subsequent operations and to better describe voxel feature information. First, fully connected neural layers (FCNs) are used to process the voxel features. in The mapping is 32-dimensional, so each voxel has t 32-dimensional feature vectors. These are then input into a max-pooling layer to obtain the global variables for that voxel, which are then concatenated into the feature vector V at each point. in Then, each point now has 64-dimensional features. The above process is repeated to obtain t 128-dimensional feature vectors for that voxel. Finally, max pooling is used to obtain the 128-dimensional global features of that voxel. Taking the Kitti dataset as an example, based on the point cloud space size and voxel size, the space can be divided along the X, Y, and Z axes into (1-(-3)) / 0.4 = 10, (40-(-40)) / 0.2 = 400, and (70.4-0) / 0.2 = 352, resulting in a total of 10 × 400 × 352 voxels. Therefore, the feature dimensions of the entire point cloud space are (128, 10, 400, 352).

[0087] (3) Submanifold 3D sparse convolutional layer

[0088] If you want to extract dense features from sparse data, ordinary convolution is prone to distortion. Furthermore, many locations in sparse data are empty, yet ordinary convolution still calculates them one by one, undoubtedly increasing computational costs. For example, the sparsity of the Kitti point cloud data is close to 0.005, making ordinary convolution computationally very expensive. Therefore, submanifold sparse convolution is introduced. The principle is that if there are no corresponding activation points in the space, no convolution operation is performed. This method avoids the slowdown in subsequent convolutions caused by too many activation points in the point cloud. Only when the input point cloud is active will the corresponding location in the output point cloud be active.

[0089] This method consists of two parts: zero-padding for null values ​​and forced zeroing. The former is to maintain normal convolution operations, while the latter is to ensure the sparsity of features and avoid distortion. Since the input and output of 3D sparse convolution are both sparse feature maps, a hash table is constructed to store the positions of values ​​in the feature maps. Let the input feature map be F. in There are m elements with values ​​in F. in Encoded as an m×n feature matrix Q in and a hash table H in Where n represents the dimension of the eigenvectors, Qin stores all non-zero eigenvectors of Fin, and H in It is a hash table composed of key-value pairs, where each key-value pair represents the row index of the feature matrix and the corresponding element in F. in The row index is an integer in the range [0, m-1]. The sparse feature map F output after convolution is... out Also coded as Q out and H out Considering that the sparsity of the feature map should remain unchanged after sparse convolution of the submanifold, therefore H in =H out Q out It is an m×l matrix, where l refers to the dimension of the output feature vector.

[0090] Similar to regular convolution, sparse convolution requires defining parameters such as kernel size and stride.

[0091]

[0092] Among them, A i The dimension of the input, B i The dimension of the output, A i and B i Satisfy constraints

[0093]

[0094] Where f represents the kernel size, s represents the stride, and p represents zero-padded values. Since F in If the tensor is d-dimensional, then the receptive field size of the convolution kernel is f. d (Refers to a d-dimensional cube with side length f). Sparse convolution will use F... in A certain element f in =F in (a1,a2,…,a d Centered on n0, the surrounding f d Zero-padding is performed at all positions of null values ​​in the input space before convolution with the kernel. To maintain consistency between input and output sizes, it is necessary to... inZero-value padding is performed on the top layer. Furthermore, due to the convolution operation, positions that were originally zero-valued will become non-zero-valued after convolution, leading to denser feature regions and thus distortion. To maintain the sparsity of the original features, let F... in The region where empty values ​​are padded with zeros is D. zero , will F out The corresponding zero value region is forcibly cleared to zero.

[0095] Traditional convolution requires f d The computational cost of d-dimensional sparse convolution is s = m × n × l, which increases geometrically with increasing dimension. However, by introducing submanifold sparse convolution, the computational cost of convolution in the zero-value region can be ignored. Therefore, the computational cost of d-dimensional sparse convolution is s = m × n × l. Compared with the former, submanifold sparse convolution greatly reduces the computational cost.

[0096] Taking the Kitti dataset as an example, such as Figure 3 The image shows a sparse convolutional feature extraction network. Blue boxes represent submanifold sparse convolutional layers, green boxes represent ordinary convolutional layers, and orange boxes represent dimensionality reconstruction layers. The 3D convolution parameters are defined as Conv(C...). in C out The 3D feature map is converted into a 2D bird's-eye view feature map (400×352 voxels in the XY plane, each represented by a 16-dimensional feature vector) through 8x downsampling and a dimension reconstruction layer. The input to the detection network is then processed by a 2D convolutional neural network to predict location confidence and classification confidence, generating bounding boxes.

[0097] The auxiliary training network is as follows:

[0098] This invention proposes a supervised auxiliary network during training, which helps the backbone framework gain a deeper understanding of the spatial structure information of point clouds. This network can be shut down during evaluation without increasing computational cost. The network mainly consists of three parts: voxel coordinate transformation, foreground point segmentation, and center coordinate prediction.

[0099] (1) Auxiliary network function

[0100] As mentioned earlier, the features obtained by downsampling convolution operations on point cloud data inevitably lose structural details of the point cloud, and these structural features play a crucial role in subsequent target localization. Initially, the point cloud contains a large number of densely distributed points. As convolutional downsampling progresses, the resolution of the point cloud space continuously decreases, and some foreground points are integrated into the background. Therefore, in the low-resolution feature map, these points are misclassified, resulting in inaccurate detection boxes. This invention proposes an auxiliary network to supervise the feature maps obtained from different convolutional layers in the backbone network to learn the spatial structural information of the point cloud. Therefore, the voxel coordinates in space must first be converted to point cloud coordinates.

[0101] (2) Voxel coordinate transformation

[0102] First, the non-zero voxel indices in the backbone network are converted into coordinates in the point cloud space. Therefore, the features of each voxel can be transformed into point form, denoted as P = {(f i ,p i f | i = 1, ..., M}, where f i p is the eigenvector. i Let {p} represent the coordinates of a point in the point cloud space, with a total of M voxel indices. Since downsampling at different scales reduces spatial resolution, to maintain the spatial resolution characteristics of the point cloud, {p} is used as the indices. i Centered on |i=1,…,M}, we aggregate all points within a certain radius around it and interpolate the voxel features. For example... Figure 4 As shown, the point at the center of the circle represents the coordinates of the voxel in the point cloud, and the other points represent points existing in the original point cloud. The four aggregations of points with different radii in the figure correspond to the four downsampling processes in the backbone network. Specifically, let Ψ(p i Let pi be a spherical region. In each layer of the convolutional downsampling network, different spherical radii are set. The reciprocal of the distance between all points within the radius pi is selected as the weighted average. Let the weighted feature be... The feature vector of each point can then be obtained by the following formula.

[0103]

[0104] Taking the Kitti dataset as an example, the convolutional network extracts features by downsampling at 1, 2, 4, and 8 times, resulting in sphere radii of 0.05m, 0.1m, 0.2m, and 0.4m in the corresponding auxiliary network layers, respectively. Finally, the weighted features obtained with different radii are concatenated and stacked, and then input into the foreground point prediction module and the center point learning module for processing.

[0105] (3) Foreground Segmentation Network

[0106] Points representing the target in the point cloud are called foreground points. Identifying these foreground points is crucial for predicting the target's location and orientation; one of the purposes of auxiliary networks is foreground point segmentation. For example... Figure 5 As shown, specifically, the multi-scale features obtained above are input into a two-layer multilayer perceptron (MLP) network to derive the weights of the foreground points. The first layer of the MLP (128, 256) network includes two convolution processes. The first convolution uses 128 kernels of size 1×128×1 to perform convolution with the input features, resulting in n 1×128 feature vectors. The second convolution uses 256 kernels of size 1×1×128 to perform convolution, resulting in n 1×256 feature vectors. Finally, a dimensionality transformation is used to transform the feature vectors to n×256. The second layer of the MLP(128,1) network also has two convolution processes. The first convolution uses 128 convolution kernels of size 1×256×1 to perform convolution calculation with the input features, resulting in n 1×128 feature vectors. The second convolution uses a single convolution kernel of size 1×1×128 to perform convolution calculation, resulting in n 1×1 feature vectors. The purpose of using the MLP network is to obtain the probability that each input point belongs to the foreground point. Finally, a dimensionality transformation is used to transform the foreground point prediction vector into an n×1 dimension.

[0107] Since the points in a point cloud are not independent but have a certain structural relationship, they cannot be regarded as independent samples. Figure 5 In this context, "Shared" means that when the input data contains multiple points, each point is multiplied by the same weight, with the weight range being (0,1). This weight is then multiplied by the feature vector to obtain the final feature of that point. Specifically, this refers to the feature of each point that passes through the foreground point prediction network. It can be represented as

[0108]

[0109] in, Input features The two-layer MLP network mentioned above is used to predict the confidence level of belonging to foreground points. It consists of two parts: forward propagation and backward weight update.

[0110] In positive propagation, The weights of the hidden layers in the convolutional network are set using the softmax function as the activation function. The weights of the output layer, For the output probability confidence, One-Hot encoding is used, and FocalLoss

[48] is used as the loss function.

[0111]

[0112] Where, N pos The number of foreground points is represented by α and γ, which are hyperparameters. Based on the parameter settings in the original paper and combined with practical experience, α = 0.25 and γ = 2 are set. The gradient calculation and weight update formulas for the hidden layer and output layer are as follows:

[0113]

[0114]

[0115] Among them, E n The result is calculated for Focal Loss, where η is the weight in the gradient descent process, which is only performed during training. The foreground point segmentation network allows the backbone network to detect the bounding boxes of objects more accurately.

[0116] (4) Centroid Learning Network

[0117] Even with foreground point segmentation networks, the scaling and shape of the bounding box cannot be accurately predicted. To further improve the target localization accuracy, this invention designs a center point learning network to learn the position of points in the target relative to the target center. By learning the internal relationships of the target, it further helps the method model to scale the bounding box boundary and predict the target shape based on the object shape.

[0118] Similar to foreground point segmentation networks, center point learning networks, during training, set... Let Δp be the average distance between the center point of the ground truth boxes and the points inside the boxes. Let Δp be the average distance between the center point and the voxel coordinates after mapping to the point cloud coordinates. Smooth-l1

[49] is used as the loss function to achieve positional deviation convergence.

[0119]

[0120] Where N pas The number of points is represented by χ[·]. χ[·] indicates that the position of the voxel coordinates mapped to the point cloud must be within the ground truth bounding box. By combining the foreground segmentation network and the center point learning network, the backbone network can learn the structural features of the point cloud. Enabling the auxiliary module during training can significantly improve the localization accuracy without increasing computational cost during the evaluation stage.

[0121] The specific network to be detected is as follows:

[0122] The bird's-eye view features obtained from the voxel feature extraction network are input into the detection network. The generation of detection boxes involves scoring the feature information of the bird's-eye view. Each point in the image represents a feature obtained along the Z-axis in the original point cloud space. A mature 2D convolutional neural network is used to process the bird's-eye view, resulting in a probability score map and a localization score map. The probability score map scores the voxel features. Each point in the bird's-eye view generates six detection boxes at six angles (0°, 30°, 60°, 90°, 120°, and 150°) in the localization score map. Each detection box consists of seven parameters (x, y, z, h, w, l, r). The first three parameters represent the coordinates of the X, Y, and Z axes in 3D space, and the last four parameters represent the height, width, length, and azimuth angle of the detection box.

[0123] To address the mismatch between predicted bounding boxes and classification confidence maps, this invention proposes a Position-Alignment Module. This module aligns the classification confidence and the detection box by performing spatial transformations on the feature map. Specifically, for a 3D detection box, let the corresponding classification confidence be R. p The 3D detection box can be compressed into a 2D detection box from the BEV viewpoint through dimensionality transformation. K points are uniformly sampled within this 2D detection box. Taking the Kitti dataset as an example, K = 28, which is equivalent to uniformly sampling 4 × 7 points. Next, the feature vectors corresponding to these sampling points in the BEV feature map are extracted. Since the size of the BEV viewpoint is D × W = 400 × 352, and after an 8x sparse convolution downsampling process, the size of the BEV feature map is d × w = 50 × 44, which is 8 times smaller than the size of the BEV viewpoint, the coordinates of the sampling points need to be reduced by a factor of 8 to find the corresponding feature vectors in the BEV feature map. Let the coordinates of the kth sampling point in the BEV viewpoint be (x...). k ,y k If the sampling point is located in the BEV feature map, then the coordinates of that sampling point are:

[0124]

[0125] And at this time (m k ,n k The value (m) is not necessarily an integer, therefore, features of surrounding points are selected and interpolated to obtain (m). k ,n k The characteristics of )

[0126]

[0127] b(i,j,m,n)=max(1-|im|,0)×max(1-|jn|,0) (13)

[0128] in, This means rounding down, and we need to get (m) k ,n k The coordinates of the four points in the neighborhood, i.e. This corresponds to the index range of i and j in the formula. b(i,j,m,n) represents the distance from the four coordinate points to the target point (m). k ,n k The weight of ) The feature vectors represent the four coordinate points in the BEV feature map. At this point, by selecting sampling points in the detection box and mapping them to the feature map, an interpolation method is used to align the classification confidence and location confidence. Compared to other methods that use non-maximum suppression (NMS) to process dense feature maps to obtain regions of interest (ROIs), the alignment module proposed in this invention directly samples the detection box, resulting in lower computational cost and significantly better performance.

[0129] For the flow of the position alignment module, please refer to [link / reference]. Figure 6 The following is the pseudocode for the position alignment module:

[0130]

[0131]

[0132] In this invention, the loss function consists of four parts: location confidence loss L box Classification confidence loss L cls Foreground site segmentation loss L seg The center-point learning network loss L ctr Gradient descent is used to simultaneously optimize the backbone network detection task and the auxiliary network task, denoted as:

[0133] L = L cls +L box +βL seg +μL etr (14)

[0134] Where L seg and L ctr As mentioned above, β and μ are used to balance the hyperparameters of the auxiliary task and the backbone network. This invention will determine the values ​​of these two hyperparameters through analysis in the experimental section. box Using the Smooth-l1 function, L cls Using the Focal Loss function, it is expressed as

[0135] L cls =-α(1-c t ) γ logc t (15)

[0136]

[0137] The hyperparameter α represents the balancing factor, empirically set at 0.25, used to balance the imbalance between positive and negative samples. γ is used to reduce the loss of easily classified samples, with a value of 2.0. c t The output represents the probability that the value belongs to the true value. Δra represents the predicted residual, and Δra represents the regression objective.

[0138] The proposed method was validated using the Kitti, Waymo, and Nuscene datasets. On the Kitti dataset, several superior methods were compared with the proposed method to determine the weights of the foreground point segmentation module and the center point learning module in the loss function, and PR curves were plotted to verify the effectiveness of the proposed method. Simultaneously, comparative validation was also performed on the large datasets Waymo and Nuscene, demonstrating the strong generalization ability of the proposed method, and loss function curves were plotted. Finally, point cloud data was collected on a physical platform, and a small test dataset was created to conduct ablation experiments and accuracy verification on the proposed method. The final visualization results show that the proposed method has high detection accuracy, high time efficiency, and strong portability.

[0139] Kitti dataset

[0140] This invention evaluates the proposed method on the Kitti 3D point cloud dataset, which contains 7481 training samples and 7518 test samples. The mean average precision (mAP) is used as the evaluation metric, and the recall is 40%. This invention detects three target classes: cars, pedestrians, and cyclists. The IoU threshold for cars is 0.7, and the IoU threshold for pedestrians and cyclists is 0.5. The method is compared with other excellent methods, including classic single-stage and two-stage methods. Single-stage methods include VoxelNet, Second, PointPillars, and ContFuse. These methods extract features by dividing the voxel space and then use different detection methods for processing, demonstrating the strong real-time performance of voxel-based methods. Two-stage methods include F-PointNet, AVOD, PointRCNN, and MV3D. These methods extract ROI regions using PointNet as the main framework in the first stage and perform fine processing in the second stage, demonstrating the advantage of point-based methods in extracting point cloud structural information. As shown in Table 1, the method proposed in this invention has high detection accuracy, and is significantly superior to other methods in vehicle detection, improving mAP by 0.14%, 2.69%, and 0.21% in the simple, medium, and difficult difficulty levels, respectively. Furthermore, it employs a unique training model.

[0141] Table 1. Accuracy comparison of different methods on the Kitti 3D dataset.

[0142]

[0143] It is necessary to determine the weights β and μ of the foreground point segmentation task and the center point learning task in the final loss function of the auxiliary network. These are key hyperparameters in the auxiliary network of this invention. To obtain the optimal weights, this invention first sets μ to zero, adjusts β to obtain the optimal detection accuracy, then fixes β and adjusts μ for experimentation, such as... Figure 7 As shown, the weight β of the current scenic spot segmentation task is in the range of [0.8-1.2], which significantly improves the detection accuracy for medium-difficulty tasks. However, if the weight is too small or too large, the detection accuracy will decrease. A similar trend can be observed in the center point learning task. Therefore, through experiments, β = 1.0 and μ = 2.0 were determined.

[0144] like Figure 8 As shown, recall-precision curves (PR curves) are used to evaluate various methods based on the Kitti dataset. Recall is the number of correctly classified samples divided by the total number of positive samples at a given threshold, while precision is calculated as the number of predicted positive samples divided by the number of samples predicted as positive (including negative samples that were incorrectly predicted as positive) at a given threshold. As the curves show, under different thresholds, the proposed method balances precision and recall, exhibiting better convergence.

[0145] The Kitti dataset includes several common scenarios such as cities, roads, and campuses. The visualization results of the method in this invention are as follows: Figure 9 As shown, the upper part of each sub-image is an RGB image, the middle part is the 3D point cloud detection result, and the lower part is the detection result from the BEV perspective. Each bounding box represents a vehicle, pedestrian, and cyclist, respectively, and the confidence score is labeled next to the box. The results demonstrate that the method proposed in this invention has high detection accuracy, precise localization, and can detect occluded targets in RGB images.

[0146] To verify the time efficiency of the method of this invention, the detection time of each frame of point cloud was tested, and the two-stage methods PointRCNN and F-PointNet, and the single-stage methods Second and PointPillars were compared with the method of this invention. As shown in Table 2, the method of this invention has a significant time efficiency advantage compared with the other four methods, reaching 0.04s. The two-stage method first needs to select the region of interest and then perform fine processing on the ROI region. Although the accuracy is guaranteed, it undoubtedly takes more time. Compared with other single-stage methods, this invention adopts an innovative approach to the convolution method, introducing a submanifold sparse convolution method, which can greatly save computational costs and improve time efficiency. Furthermore, the addition of an auxiliary network enables foreground point segmentation and center point learning, which provides a guarantee for detection accuracy.

[0147] Table 2 Comparison of time efficiency based on the Kitti dataset

[0148] PointRCNN LiDAR 0.16 F-PointNet LiDAR+RGB 0.32 Second LiDAR 0.10 PointPillars LiDAR 0.12 This invention LiDAR 0.04

[0149] Finally, experiments were conducted on the Kitti medium-difficulty dataset to test the detection results and required detection time of the proposed position alignment module with different numbers of sampling points. Figure 10 As shown, sampling methods of 2×3, 3×5, 4×7 and 5×9 are performed in the BEV bird's-eye view. The most suitable result is selected based on the final detection accuracy and the required detection time. Although the detection result of the 5×9 scale is better than that of the 4×7 scale, the 4×7 scale sampling method is finally selected in this invention considering the time efficiency of the method.

[0150] Waymo dataset:

[0151] To further verify the effectiveness of the proposed method, it was evaluated on the Waymo dataset and compared with other excellent methods. The effective range of the X and Y axes in the detection space was set to [-75.2, 75.2] m, the detection range of the Z axis to [-2, 2] m, the voxel size to [0.1 m, 0.1 m, 0.15 m], and the IoU threshold to 0.7. Using the official evaluation tool, the point cloud space was divided along the X axis into [0, 30] m, [30, 50] m, and [50, 75.2] m based on the distance from the sensor center to the target, as shown in Table 3. The proposed method shows a significant improvement compared to other excellent methods, with an average accuracy improvement of 6.42% based on 3D objects and a 3.66% improvement in average accuracy under BEV view. This demonstrates that the proposed single-stage target detection method is comparable to traditional two-stage methods in detection accuracy and exhibits higher accuracy at greater distances, indicating that the proposed position alignment module effectively addresses the inconsistency between positional confidence and classification confidence.

[0152] Table 3. Comparison of mAP between 3D and BEV in the Waymo dataset.

[0153]

[0154] The proposed method model is trained using four GTX 1080ti GPUs with a learning rate of 0.01, a batch size of 16, and 80 training iterations, yielding the following results: Figure 11 As shown in the loss function curve, the gradient descent is rapid at the beginning and then tends to flatten out, without a significant decrease, which is consistent with the training pattern of the model.

[0155] Nuscene dataset:

[0156] The method proposed in this invention was also tested on the Nuscene dataset to verify its generalization ability. This dataset contains various scenes, as shown in Table 4. The detection accuracy of PointRCNN and Second method compared with the method proposed in this invention in detecting three categories: cars, pedestrians, and cyclists. It can be seen that the method proposed in this invention performs well on the Nuscene dataset, with detection accuracy of 90.86%, 89.56%, and 61.57% for cars, pedestrians, and cyclists, respectively, all of which are better than the other two methods.

[0157] Table 4. Detection results of different methods on the Nuscene dataset.

[0158] PointRCNN 86.22 89.03 59.53 78.26 Second 87.26 88.92 55.79 77.32 This invention 90.86 89.56 61.57 80.66

[0159] Physical platform testing:

[0160] Finally, the method proposed in this invention was ported to a real vehicle platform for testing to verify its effectiveness. The real vehicle platform includes various sensors. Since the method of this invention processes point clouds, a LiDAR sensor was used for testing. First, data was collected using the sensor to obtain point cloud files based on the Rosbag format. The test consisted of three parts: real-time dynamic multi-target detection, ablation comparison experiment, and accuracy detection. For the ablation comparison experiment and accuracy detection, a small test dataset based on LiDAR point clouds was created. By playing the Rosbag file and extracting keyframes, the size information of the point cloud space was referenced from the Kitti dataset. Then, annotation software was used to outline the actual data labels and location information of the targets in the point cloud. The targets were mainly divided into three categories: cars, pedestrians, and cyclists.

[0161] The ablation experiments focused on the foreground point segmentation module, center point learning module, and location alignment module, analyzing them separately. The trained sub-modules were ported to the ROS platform, and the collected point cloud data was used for detection. The results were compared with real data labels to obtain the detection accuracy. As shown in Table 5, using the auxiliary module proposed in this invention significantly improves the detection accuracy, increasing it by 2.37% compared to not using the foreground point segmentation and center point learning modules. Furthermore, the location alignment module of this invention can further improve the consistency between location confidence and classification confidence based on the auxiliary network, achieving a detection accuracy of 84.30%.

[0162] Table 5 Sub-module Ablation Experiments

[0163]

[0164]

[0165] The detection accuracy was compared with the proposed method using PointRCNN and Second methods, and the results are shown in the figure. Figure 12 and Figure 13 The curves shown in the figure demonstrate that the method proposed in this invention achieves a high detection accuracy of 86.37% for the vehicle category, with an average accuracy of 83.75%, which is superior to the other two methods. This indicates that the method proposed in this invention is effective, has strong generalization ability, and performs well on various datasets.

[0166] Finally, the method of this invention was ported to a ROS-based operating platform, and the point cloud dataset was input into the pre-trained model for real-time dynamic multi-object detection, achieving a time efficiency of 0.04s. The visualization results are as follows... Figure 14 As shown, it can detect three types of targets: cars, pedestrians, and cyclists. The detection results show that the detection is accurate, further proving that the method proposed in this invention has strong generalization ability and fast detection speed.

[0167] This embodiment was tested on the Kitti, Waymo, and Nuscene datasets and on a physical platform, and compared with other excellent methods. The results show that the method proposed in this invention is effective, time-efficient, and has strong generalization ability.

Claims

1. A three-dimensional target detection method based on point cloud structure perception, characterized in that, Includes the following steps: (1) Obtain raw point cloud data; (2) A feature extraction network is used to extract voxel features to obtain bird's-eye view features; the feature extraction network consists of a voxel space partitioning layer, a VFE feature extraction layer and a submanifold sparse convolutional layer, and the submanifold sparse convolutional layer optimizes the computational efficiency through zero-padding and forced zero-clearing operations; (3) Provide an auxiliary training network to supervise the feature extraction network during training, learn the foreground points and target center points, and turn off the auxiliary training network during the evaluation process; the auxiliary training network is divided into three parts: voxel coordinate transformation, foreground point segmentation and center coordinate prediction, wherein the voxel coordinate transformation converts non-zero voxel indices into point cloud spatial coordinates, performs distance inverse weighted interpolation of multi-radius spherical regions with voxel points as the center, and then concatenates and stacks the weighted features of different radii; (4) Input the bird's-eye view features obtained in step (2) into the detection network to generate a three-dimensional target detection box. The detection network consists of a 2D convolutional layer, a region proposal network and a position alignment module. The position alignment module compresses the 3D detection box into a 2D detection box under the BEV view, uniformly samples K points in the box, and calculates the classification confidence by linear interpolation weighting of the four neighboring points to achieve the alignment of classification confidence and position confidence.

2. The three-dimensional target detection method based on point cloud structure perception according to claim 1, characterized in that: Step (2) specifically includes: (2-1) Divide the voxel space: Along in the point cloud space Z , X , Y Construct a 3D spatial coordinate system using axes, and define the ranges of the three directions as follows: H , D , W The sampling step sizes are respectively d z , d x , d y The number of divisions for each coordinate axis is H' , D' , W' The entire point cloud space is divided into N Voxels of equal size, among which (1) Introduce a reference threshold T Suppose that the number of points within a certain voxel is greater than T Then, a random sampling method is used to select... T One point, remove redundant points; Encode each voxel using It means that among them p i Indicates the first non-empty voxel i points X , Y , Z Coordinates and reflectivity of that point r i , t This indicates the number of points contained in each voxel. Let the mean value of all points in each voxel be taken as the local centroid, denoted as: ,calculate p i The offset relative to the centroid is expressed as This yields the seven-dimensional input features of each voxel. ; (2-2) VFE feature extraction: First, use a fully connected layer to... V in If mapped to 32 dimensions, then each voxel has t A 32-dimensional feature vector is generated, then input into a max-pooling layer to obtain the global variable for that voxel, which is then concatenated into the feature vector at each point. V in Then, each point now has 64-dimensional features. The above process is repeated to obtain the voxel. t Each voxel has a 128-dimensional feature vector, and finally, the global feature of the voxel is obtained by max pooling. (2-3) Sparse convolution of submanifolds: Sparse convolution's input and output are both sparse feature maps. A hash table is constructed to store the positions of values ​​in the feature maps. Let the input feature map be... F in Among them, the value elements are m One, will F in Encode as a m × n Feature matrix Q in and a hash table H in ,in n The dimension of the feature vector. Q in Stored F in All non-zero eigenvectors, H in It is a hash table composed of key-value pairs, where each key-value pair represents the row index and corresponding element of the feature matrix. F in The index in the data, the row index takes the value [0, ... m Integers within the range of -1; sparse feature map output after convolution. F out Also encoded as Q out and H out , H out = H in , Q out yes m × l The matrix, l This refers to the dimension of the output feature vector; set up (2) in, A i Refers to the dimension of the input. B i Refers to the dimension of the output. A i and B i Satisfy constraints (3) in f Represents the size of the convolution kernel. s Represents the convolution stride. p This represents zero-value filling; because F in yes d For a tensor of dimension , the receptive field size of the convolution kernel is with a side length of . f of d 3D cube f d Sparse convolution will use F in A certain element Centered around f d Zero-padding is performed at all positions of null values ​​in the space, and then convolution is performed with the convolution kernel. Let... F in The area to be padded with zeros is D zero ,Will F out The corresponding area is forcibly cleared to zero.

3. The three-dimensional target detection method based on point cloud structure perception according to claim 2, characterized in that: In step (2-1), the reference threshold is an integer between 30 and 50.

4. The three-dimensional target detection method based on point cloud structure perception according to claim 1, characterized in that: Step (3) specifically includes: (3-1) Voxel coordinate transformation: First, convert the non-zero voxel indices obtained in step (2) into coordinates in the point cloud space. The features of each voxel are converted into points, denoted as... ,in f i For feature vectors, p i Represents the coordinates of a point in the point cloud space, with a total of M Individual index, by Centered on a certain radius, all points within that radius are aggregated to interpolate the voxel features. Specifically, let... For a spherical region, different spherical radii are set in each layer of the convolutional downsampling network, and the selection is... p i The reciprocal of the distance between all points within the radius is used as the weighted average. Let the weighting characteristic be... Then the feature vector of each point can be obtained by the following formula. (4) Finally, the weighted features obtained from different radii are concatenated and stacked. (3-2) Foreground Segmentation: The weighted features with different radii obtained in step (3-1) are input into a two-layer multilayer perceptron network to obtain the weights of the foreground points. The first layer of the multilayer perceptron network includes two convolution processes. The first convolution is performed with the input features using 128 convolution kernels of size 1×128×1, resulting in n 1×128 feature vectors. The second convolution is performed with 256 convolution kernels of size 1×1×128, resulting in n 1×256 feature vectors. Then, a dimension transformation is used to transform the feature vectors into n×256. The second layer of the multilayer perceptron network has two convolution processes. The first convolution is performed with the input features using 128 convolution kernels of size 1×256×1, resulting in n 1×128 feature vectors. The second convolution is performed with one convolution kernel of size 1×1×128, resulting in n 1×1 feature vectors. Finally, a dimension transformation is used to transform the foreground point prediction vectors into n×1 dimensions. When the input data contains multiple points, each point is multiplied by the same weight, with the weight range being (0,1), and finally multiplied by the feature vector to obtain the final feature of that point; (3-3) Focused Learning: During the training process, set Let be the average distance between the center point of the ground truth bounding box and the points inside the box. The mean distance between the voxel coordinates and the center point after mapping from voxel coordinates to point cloud coordinates is used. As a loss function, positional deviation convergence is achieved. (10) in The number of representative points, The position of the indicator function, i.e., the position of the voxel coordinates mapped to the point cloud, must be within the ground truth bounding box.

5. The three-dimensional target detection method based on point cloud structure perception according to claim 1, characterized in that: Step (4) specifically includes: The bird's-eye view features obtained in step (2) are input into the detection network. The generation of the detection box involves scoring the feature information of the bird's-eye view. Each point in the image represents a point in the original point cloud space along... Z The features obtained from the axes are processed using a 2D convolutional neural network to obtain a probability score map and a localization score map. For each point in the bird's-eye view, a detection box with six angles (0°, 30°, 60°, 90°, 120°, and 150°) is generated in the localization score map. Each detection box consists of seven parameters (x, y, z, h, w, l, r). The first three parameters represent the coordinates of the X, Y, and Z axes in 3D space, and the last four parameters represent the height, width, length, and skew angle of the detection box. The location alignment module aligns the classification confidence score and the detection box by performing spatial transformation on the feature map. Specifically, for a 3D detection box, let the corresponding classification confidence score be... R p The 3D detection box can be compressed into a 2D detection box from the BEV's perspective through dimensional transformation, and data can be uniformly collected within this 2D detection box. K For each of the n points, extract the feature vectors corresponding to these sampling points in the BEV feature map. Let the nth point be... k The coordinates of the sampling points in the BEV view are: Then the coordinates of the sampling point in the BEV feature map are: (11) At this time The integer value is not necessarily an integer; therefore, features from surrounding area points are selected and interpolated to obtain the result. The characteristics of (12) (13) in, This means rounding down to the nearest integer, which requires obtaining... The coordinates of the four neighboring points, i.e. In the corresponding formula i and j The index range, Represents the distance from four coordinate points to the target point. The weights, The feature vectors representing the four coordinate points in the BEV feature map are used to align classification confidence and location confidence by selecting sampling points in the detection box and mapping them to the feature map, and by using interpolation.

Images