A single-stage point cloud density-aware focused convolutional 3D object detection method

Abstract

This invention discloses a single-stage point cloud density-aware focused convolutional 3D object detection method, belonging to the field of 3D object detection. The 3D object detection method includes the following steps: dynamically converting point cloud data in a dataset into voxel-shaped data; obtaining voxel features that fuse point cloud density and distance based on voxel information; inputting the voxel features into an 8x downsampled residual focused sparse convolution for high-dimensional feature extraction; converting the features obtained from the sparse convolution into dense data and compressing them along the Z-axis to obtain a BEV feature map; feeding the BEV feature map into a two-branch convolutional network for feature extraction to obtain the final BEV features; and feeding the final BEV features into an RPN detection head to obtain the final detection result. This invention enhances the sensitivity of the single-stage 3D object detection model to the sparse point cloud density category and strengthens the learning ability of sparse convolution for spatial information, making it applicable to 3D object detection in LiDAR sensor environments.

Description

Technical Field

[0001] This invention belongs to the field of 3D target detection, and specifically relates to spatial learning techniques for point cloud density feature extraction and 3D sparse convolution, which can be used for 3D target detection in the context of LiDAR sensor environments. Technical Background

[0002] Scene perception is an essential component of autonomous driving technology, and 3D object detection technology is the key core of the perception system. Its purpose is to predict the position, size, and category of 3D objects near autonomous vehicles.

[0003] Many types of sensors can provide raw data for 3D object detection, with cameras and LiDAR being two of the most widely used. Cameras are inexpensive and easy to use, but they only capture appearance information and cannot directly obtain 3D structural information (such as depth) of the scene. Therefore, detection methods from images are often susceptible to extreme weather and time conditions. As a superior option, LiDAR sensors can directly obtain fine-grained 3D structure of the scene by emitting a laser beam and then measuring its reflection information, resulting in more robust detection performance. Because of this characteristic, LiDAR has become one of the main 3D object detection sensors in autonomous driving. However, the point pattern of LiDAR diverges with increasing distance, resulting in an uneven sampling point cloud, which is unsuitable for discrete feature extraction. Current methods either rely on voxelized point clouds or use inefficient sampling of the farthest point to mitigate the adverse effects of density variations, but largely ignore point density as a feature and its predictable relationship with the distance to the LiDAR sensor.

[0004] Point density also affects the detection of smaller objects such as pedestrians and cyclists. These objects have a smaller surface area where they intersect with the LiDAR laser beam, resulting in poorer object localization. Current state-of-the-art methods largely ignore the detection performance of pedestrians and cyclists, focusing only on cars or vehicle categories. As autonomous driving scenarios move towards datasets with higher environmental coverage, it is necessary to extend the architecture to a larger input space and serve as a multi-class solution for 3D object detection.

[0005] Meanwhile, current single-stage 3D object detection methods based on voxelization use sparse convolutional networks proposed in the SECOND network for the downsampling convolution stage. 3D sparse convolutional networks consist of regularized and submanifold sparse convolutional layers. While both types of sparse convolution have been widely used, they each have their limitations. Regularized sparse convolution extends all sparse features, inevitably adding a significant computational burden to the model. Furthermore, the detector aims to distinguish target objects from a large number of background features, but the characteristics of regularized sparse convolution greatly reduce sparsity, blurring feature distinctions. On the other hand, submanifold sparse convolution avoids computational problems by restricting the output feature positions of the input, but it ignores necessary information flow, especially when dealing with spatially disjoint features. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the prior art by proposing a single-stage point cloud density-aware focused convolutional 3D target detection method. This method solves the problems of ignoring point cloud density features and the inability of sparse convolution to dynamically process input features in the prior art, thereby improving detection accuracy.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A single-stage point cloud density-aware focused convolutional 3D object detection method includes the following steps:

[0009] Step 1, voxel feature extraction.

[0010] The point cloud data in the dataset is dynamically allocated to voxels at corresponding locations. The distances from the point cloud to the physical center of the voxel and from the point cloud to the centroid of the voxel are calculated as additional features. The features are then fused through a feature fusion network based on a feedforward network to obtain voxel features.

[0011] Step 2, 3D sparse convolution.

[0012] The voxel features are input into a 3D residual focusing sparse convolutional network for downsampling to obtain sparse 3D high-dimensional features that have learned the importance of spatial location.

[0013] Step 3, BEV feature transformation.

[0014] The sparse 3D high-dimensional features are transformed into dense tensors and compressed in the Z-axis direction. The feature z-axis is multiplied by the number of feature channels to obtain the BEV 2D features.

[0015] Step 4, BEV convolution.

[0016] The BEV 2D features are fed into a dual-branch 2D convolutional backbone network for downsampling, upsampling, and channel concatenation operations to extract deeper semantic features, resulting in the final BEV feature map.

[0017] Step 5, RPN network prediction training.

[0018] Prediction is performed using the final BEV feature map.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] 1) This invention, by employing dynamic voxelization and additional point cloud density features, enables the network to extract more useful features from sparse point cloud information and learn more semantic information. The improved object detection network has higher object detection accuracy, especially in point cloud density-sensitive categories such as Pedestrian and Cyclist, where the improvement is more significant.

[0021] 2) In the 3D sparse convolution operation, the present invention performs residual connections on the regular sparse convolution layer and the submanifold sparse convolution layer, which improves the training efficiency and stability of the network, avoids the gradient vanishing problem in deep networks, and ensures the stability and effectiveness of feature extraction.

[0022] 3) In the 3D sparse convolution operation, the present invention uses focusable sparse convolution to replace submanifold convolution, which solves the problem that sparse convolution cannot dynamically process input features, enabling sparse convolution to learn more spatial information and improving network performance. Attached Figure Description

[0023] Figure 1 This is a network structure diagram of a single-stage point cloud density-aware focused convolutional 3D target detection method according to the present invention.

[0024] Figure 2 This is a network diagram illustrating the voxel feature extraction step in this invention;

[0025] Figure 3 This is a schematic diagram of the 3D sparse convolution step in this invention;

[0026] Figure 4 A schematic diagram of the aggregateable sparse convolution module in this invention;

[0027] Figure 5 Comparison of the effects of this invention on target detection of people, bicycles, and cars using point cloud data. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other achievements obtained by those skilled in the art based on the present invention without inventive effort should fall within the scope of protection of the present invention.

[0029] Existing single-stage 3D object detection techniques have shortcomings in feature extraction and sparse convolution processing, resulting in poor detection performance on objects that are far away or have sparse point cloud information. To address these technical problems, this invention provides a single-stage point cloud density-aware focused convolutional 3D object detection method.

[0030] Combination Figure 1 As shown, the method of the present invention relies on a voxel feature extraction network, a downsampling module based on 3D sparse convolution, a Z-axis compression module for generating BEV features, a BEV feature extraction module, and an RPN network prediction module, etc., which constitute the main part of the present invention.

[0031] Reference Figure 1 The main implementation steps of the single-stage point cloud density-aware focused convolutional 3D target detection method of the present invention are as follows:

[0032] Step 1: Voxel feature extraction, that is, converting point cloud information into voxel features.

[0033] refer to Figure 2 This step mainly involves dynamically allocating point cloud data from the dataset to voxels at corresponding locations, calculating the distances from the point cloud to the physical center of the voxel and the point cloud to the centroid of the voxel as additional features, and then fusing these features through a feature fusion network based on a feedforward network to obtain voxel features.

[0034] Specifically, in this embodiment of the invention, the voxel feature extraction step includes:

[0035] Step 11, obtain point cloud data from the original dataset, in the format [x p y p , z p i p ], representing the 3D coordinates of point cloud p and the laser intensity, respectively. A unique voxel code is assigned to each point cloud based on its 3D coordinates and mesh size:

[0036] MC p =B p ×SC_xyz+PCx p ×SC_yz+PCy p ×SC_z+PCz p

[0037] Among them MC p B represents the unique voxel code of the point cloud p. p PCx represents the batch containing point cloud p. p PCy p PCz p Let be the grid coordinates of point cloud p, and SC_xyz, SC_yz, and SC_z be different grid scaling factors used to construct a unique code for each voxel.

[0038] Step 12, MC is encoded based on the unique voxel code of each point cloud. p The position coordinates VC of v in each voxel can be obtained through methods such as deduplication functions. v and the mean PM of the point cloud coordinates within voxel v v The value is the centroid of the voxel. The distance from the point cloud p to the centroid of the voxel is calculated based on the centroid of the voxel and is used as an additional feature f_core. p .

[0039]

[0040] S v ={p|p∈v}

[0041] f_core p =[x p y p , z p -PM v

[0042] Where mean(·) represents the mean operation, S v It is a set of point clouds within voxel v. This dynamic voxelization method based on unique voxel encoding effectively solves the problems of traditional voxelization methods, which fix the maximum number of voxels and the maximum number of point clouds within each voxel. Fixed voxelization not only wastes computational space but also randomly discards point cloud information exceeding the maximum number, which is not conducive to feature extraction of sparse point cloud information. Point cloud unique voxel encoding MC p After deduplication, the remaining MC p The unique voxel positions (VCs) are calculated by restoring the original values.

[0043] Step 13: Determine the physical center of each voxel based on its size and the range of the point cloud. Then, calculate the distance from the point cloud p to the physical center of the voxel based on the voxel's physical center, and use this distance as an additional feature f_center. p .

[0044] Step 14, extract the original features of point cloud p, f_core p f_center p Cascading yields new features f_core p f_center p Two additional features reflect the density information of the point cloud, The input feature fusion network obtains the features f of voxel v. v .

[0045]

[0046] S v ={p|p∈v}

[0047] Where max(·) represents max pooling, and G(·) represents a multilayer perceptron network for feature extraction. In this embodiment, the number of input feature channels of the multilayer perceptron is... The feature length is 11, and the number of output feature channels is 32, meaning that this step yields voxel features with 32 channels.

[0048] Step 2: 3D sparse convolution.

[0049] refer to Figure 3 The voxel features are input into a 3D residual focusing sparse convolutional network for downsampling, resulting in sparse 3D high-dimensional features that have learned the importance of spatial location. In this embodiment, 8× downsampling is performed, and the specific process can be described as follows:

[0050] Step 21: First, based on the voxel features f, voxel position coordinates, and voxel size obtained in Step 1, generate a sparse voxel feature tensor f. initnv Its spatial dimensions are [32, 41, 1600, 1408], where [41, 1600, 1408] is the spatial size of the input voxel features.

[0051] Step 22, f initial Feature input conv input Convolutional blocks unify the input feature dimensions to obtain f input Convolutional blocks (conv) input It consists of a residual submanifold sparse convolutional layer, a batch normalization layer, and a ReLU activation function layer. Its function is to unify the number of input feature channels to 16 to facilitate subsequent convolution operations. input The spatial dimensions are [16, 41, 1600, 1408].

[0052] Step 23, f input The input is fed into the conv1 convolution block for convolution operation to obtain f. conv1 The convolutional block conv1 consists of a residual submanifold sparse convolutional module and an aggregateable sparse convolutional module. This operation does not change the spatial dimension of the features.

[0053] Step 24, f conv The input is fed into the conv2 convolutional block for the first downsampling convolution operation, resulting in f. conv2 The convolutional block conv2 consists of a residual regularized sparse convolutional module, two residual submanifold sparse convolutional modules, and a convergent sparse convolutional module. The downsampling stride is 2, and the number of feature channels changes from 16 to 32. Therefore, f conv The spatial dimensions become [32, 800, 704, 21].

[0054] Step 25, f conv The input is fed into the conv3 convolutional block for a second downsampling convolution operation to obtain f. conv3 The convolutional block conv3 consists of a residual regularized sparse convolutional module, two residual submanifold sparse convolutional modules, and a convergent sparse convolutional module. The downsampling stride is 2, and the number of feature channels changes from 32 to 64. Therefore, f conv3 The spatial dimensions become [64, 400, 352, 11].

[0055] Step 26, f conv3 The input is fed into the conv4 convolutional block for a third downsampling convolution operation, resulting in f. co The convolutional block conv4 consists of one residual regularized sparse convolutional module and two residual submanifold sparse convolutional modules. The downsampling stride is 2, and the number of feature channels remains unchanged. Therefore, f conv4 The spatial dimensions become [64, 200, 176, 5].

[0056] Step 27, f conv4 Input to conv out The convolution block undergoes a convolution operation to obtain f. out Convolutional blocks (conv) out It consists of a regular sparse convolutional layer, a batch normalization layer, and a ReLU activation function layer. The stride of the regular sparse convolutional layer is [2, 1, 1], the kernel size is [3, 1, 1], and the padding is [0, 0, 0]. The number of feature channels changes from 64 to 128, therefore f out The spatial dimensions become [128, 200, 176, 2]. The purpose of this step is to further expand the number of feature channels and change the size of the z-axis to 2 to facilitate processing by the subsequent detection head.

[0057] In the above description, one of the residual submanifold sparse convolution modules consists of a residual submanifold sparse convolution layer, a batch normalization layer, and a ReLU activation function layer. The specific operation of the residual submanifold sparse convolution layer is as follows:

[0058]

[0059] Where f in1 The features of the sparse convolutional layer representing the input residual submanifold, conv subm (·) represents the sparse convolution operation of a submanifold. Represents residual connection, f out1 This represents the features obtained after passing through a sparse convolutional layer of a residual submanifold.

[0060] One of the residual regularized sparse convolution modules consists of a residual regularized sparse convolution layer, a batch normalization layer, and a ReLU activation function layer. The specific operations of the residual regularized sparse convolution layer are as follows:

[0061]

[0062] Where f in2 The features of the input residual regularized sparse convolutional layer, conv reg (·) represents a regular sparse convolution operation. Represents residual connection, f out2 This represents the features obtained after passing through a residual regularized sparse convolutional layer.

[0063] It is important to note that the residual connection operation of residual submanifold sparse convolution differs from that of residual regular sparse convolution. Due to the sparsity of voxel features, the number of input features and the spatial size are not actually equal. Submanifold sparse convolution does not change the sparsity of the space, so the number of input features and the number of output features are the same and can be directly added. However, regular sparse convolution changes the sparsity of the data, and the number of input features is less than the number of output features, so the sparse_add() API needs to be used for the addition operation.

[0064] refer to Figure 4 The specific operation of one of the aggregated sparse convolution modules is as follows:

[0065] (a) Using a submanifold sparse convolution conv imp Based on the voxel features f_in of the input voxel v v The effective location, predicting the effective feature unit and its surrounding kernel_size. 3 The importance of each feature unit is represented by the importance value mask_v. v and mask_k v Where kernel_size refers to the size of the convolution kernel.

[0066] (b) Input voxel features f_in v With mask_v v Multiplication allows voxel features to learn the importance of spatial information and also facilitates subsequent backpropagation calculations.

[0067] f_ori v =mask_v v ×f_in v

[0068] Where f_ori v Voxel features learned after spatial importance.

[0069] (c) Obtain the distinguishing mask_v based on the threshold. v The index mask for the foreground and background points in indices_v fore and indices_v back And according to indices_v fore Get indices_k fore :

[0070] indices_v fore =mask_v v >thresho ld

[0071] indices_v back =mask_v v ≤threshold

[0072] indices_k fore =mask_k v [indices_v fore threshold

[0073] Where [] represents Boolean indexing operations, indices_v fore Indices_k represents the index mask of valid feature units identified as foreground points. fore The index mask represents the unit whose importance value is greater than a threshold among the extended feature units that are judged to be valid feature units of the foreground.

[0074] (d) via indices_v fore and indices_v back Get f_ori v The foreground feature f_ori_fore v and background point features f_ori_back v And filter out mask_k v The feature units mask_k_fore that belong to the foreground and have an importance value greater than the threshold. v The feature units it represents are the important locations where additional convolution calculations will be added later:

[0075] f_ori_fore v =f_ori v [indices_v fore ]

[0076] f_ori_backv=f_ori v [indices_v back ]

[0077] mask_k_fore v =mask_k v [indices_v fore ][indices_k fore ]

[0078] (e) in f_ori_fore v Features with a pre-connection value of 0 represent new, important locations that need to be added to the convolutional computation, enabling the convolutional network, which has learned spatial information, to output dynamically. Finally, the f_ori_back connection is established. v Obtain the complete voxel feature output f_out v , is represented as:

[0079] f_out v ={[(0,...,0),...,(0,...,0)]},f_ori_fore v f_ori_back v ]

[0080] Furthermore, f_out can also be made v With mask_k_fore v Multiplication, thereby enabling all feature units, including the newly extended ones, to learn the spatial importance of the feature, is expressed as:

[0081] f_out′ v =[{(0,...,0),...,(0,...,0)},f_ori_fore v f_ori_back v [×mask_k_fore] v

[0082] The number of (0, ..., 0) is mask_k_fore v The number of positions, i.e., the number of new positions that need to be added to the convolution calculation, is f_in. v Dimensions of voxel features.

[0083] (f) f_out v (or f_out′)v The input is a submanifold sparse convolutional layer, followed by a batch normalization layer and a ReLU activation function layer, to obtain the output of the final aggregateable sparse convolutional module.

[0084] Furthermore, additional super-supervision can be added based on more valuable prior knowledge of the foreground points, and this step can be described as: (g) for mask_v v and mask_k v Calculate the Focal loss between the predicted importance value and the true value of the antecedent. Add the Focal loss to the overall network loss to facilitate backpropagation.

[0085] Step 3: BEV feature transformation.

[0086] In step 2, we finally obtain f. out Since the sparse 3D features are sparse tensors and cannot participate in subsequent 2D convolution calculations, they need to be converted into dense tensors first. Considering that objects in 3D space do not overlap along the Z-axis, the 3D feature tensors are compressed along the Z-axis. The feature z-axis is multiplied by the number of feature channels to obtain the BEV 2D features. In this embodiment, the spatial dimension information of the obtained BEV 2D features is [128×2, 200, 176]. This operation simplifies the design of the network detection head, increases the receptive field in the height direction, and accelerates the training and inference speed of the network.

[0087] Step 4: BEV convolution.

[0088] The BEV 2D features are fed into a dual-branch 2D convolutional backbone network for downsampling, upsampling, and channel concatenation operations to extract deeper semantic features, resulting in the final BEV feature map. The implementation steps are described as follows:

[0089] Step 41: Input the BEV feature f_bev into downsampling convolution branch 1 to obtain f_bev down1 .

[0090] The downsampling convolution conv2d in branch 1 dow By only changing the number of feature channels, the BEV feature dimension becomes [128, 200, 176] after convolution.

[0091] Step 42: Input the BEV feature f_bev into downsampling convolution branch 2 to obtain f_bev down2 .

[0092] Branch 2 downsampling convolution cony2d dow By simply changing the size of the feature map, the BEV feature dimension becomes [128, 100, 88] after the convolution operation.

[0093] Step 43, f_bev down1 Upsampling is performed on the input convolution branch 1 to obtain f_bev up1 .

[0094] f_bev dawn1 The upsampled convolution conv2d fed into branch 1 up After convolution, the BEV feature dimensions are restored to [256, 200, 176].

[0095] Step 44, f_bev down Upsampling is performed on the input convolution branch 2 to obtain f_bev up .

[0096] f_bev dawn The upsampled convolution conv2d fed into branch 2 uv2 After convolution, the BEV feature dimensions are restored to [256, 200, 176].

[0097] Step 45, f_bev up and f_bev up Perform a concatenation operation along the channel dimension to obtain the final BEV feature map f_spa.

[0098] Since the feature maps are of the same size, the computation of the two branches is concatenated in the 0th dimension to obtain the final BEV feature f_spa, whose spatial dimensions are [256×2, 200, 176].

[0099] Step 5: RPN network prediction training.

[0100] In this step, the final BEV feature map is fed into the RPN prediction head network for class prediction, bounding box regression, and orientation classification tasks to obtain the final prediction result. The prediction step can be described as follows:

[0101] Step 51: Generate all anchor boxes based on the feature map size and the types of categories, sizes, and orientations to be detected.

[0102] Step 52: Input the final BEV feature map f_spa into the classification convolution conv. cls Boundary regression convolution (conv) box Convolutional directional classification dir_cls This yields the predicted classification result, predicted bounding box regression result, and predicted orientation classification result for each anchor box.

[0103] Step 53: Calculate the loss function based on the assigned true values, and train the network through backpropagation. Specifically:

[0104] First, based on the positive and negative sample thresholds for each category, a ground truth class, orientation class, and ground truth box are assigned to each anchor box. Then, the bounding box encoding is calculated based on each positive anchor box 'a' and its assigned ground truth box. Used for subsequent loss function calculation. Finally, based on the predicted classification result, predicted orientation classification result, and bounding box encoding of each anchor box. Calculate the loss function and train the network through backpropagation.

[0105] Furthermore, the anchor frame generation method in step 51 is as follows: Since the size of individuals in each category is relatively fixed and the differences are not significant in the 3D world, the average length, width, and height of the Car, Pedestrian, and Cyclist categories in the KITTI dataset are directly used as the anchor frame size. Additionally, each category's anchor frame has two directional angles: 0° and 90°. The anchor frame size is in meters. The anchor frame size for the Car category is [3.9, 1.6, 1.56], with its center at -1 meter on the Z-axis; the anchor frame size for the Pedestrian category is [0.8, 0.6, 1.73], with its center at -0.6 meters on the Z-axis; and the anchor frame size for the Cyclist category is [1.76, 0.6, 1.73], with its center at -0.6 meters on the Z-axis.

[0106] To prevent excessive loss due to the anchor box's predicted direction being exactly opposite to the ground truth box, each anchor box is assigned two one-hot vectors: one for orientation classification (whether the object is facing forward or backward) and one for class classification. In addition, a vector of length 7 is used for anchor box bounding regression, with values ​​(x, y, z, l, w, h, θ), where x, y, z are the coordinates of the anchor box's center point, l, w, h are the anchor box's length, width, and height, and θ represents the object's orientation in the coordinate system.

[0107] Ultimately, we can obtain anchor boxes for three categories, each with spatial dimensions [z, y, x, num]. size ,num rot ,7], where num size The number of scales for each category anchor box (1), num rot For each category of anchor boxes, there are 2 directions, and the seven-dimensional vector is (x, y, z, l, w, h, θ), which represents the information of the anchor box border. Finally, we get a list of anchor boxes for the three analogies, with spatial dimensions [1, 200, 176, 1, 2, 7].

[0108] Further, the prediction head in step 52 is implemented as follows: the prediction head is used to predict the corresponding category, orientation category, and seven parameters of the bounding box for each anchor box on the feature map. Each prediction or regression function is implemented through a convolution with a kernel size of 1×1. The convolution used for category classification is... cls The input channel count is 512, and the output channel count is 2×3×3, where 2×3 represents generating 6 anchor boxes per pixel, and 3 represents the expected value of each anchor box for the Car, Pedestrian, or Cyclist category. The convolutional layer used for orientation classification is conv. dir_cls The input channel count is 512, and the output channel count is 2×3×2, where 2×3 represents generating 6 anchor boxes per pixel, and 2 represents the expected value of each anchor box in the forward or backward direction; the convolutional function conv used for bounding box regression... box The input channel count is 512, and the output channel count is 2×3×7, where 2×3 represents the generation of 6 anchor boxes for each pixel, and 7 represents the regression values ​​of 7 positional parameters for each anchor box. The final spatial dimensions of the category prediction are [200, 176, 18], the spatial dimensions of the orientation classification are [200, 176, 12], and the spatial dimensions of the bounding box regression are [200, 176, 42].

[0109] The strategy for assigning a corresponding truth box to each anchor box is as follows: first, set the truth box for each anchor box to the one with the highest IOU value; then, calculate the anchor box with the highest IOU value for each truth box, and replace the truth box assigned to this set of anchor boxes with the current truth box. The category label and truth box of the anchor box are updated with the assigned values.

[0110] In addition, anchor boxes need to be categorized into positive and negative samples based on a threshold. Anchor boxes with a threshold greater than or equal to the positive sample threshold are assigned as foreground anchor boxes, while those with a threshold less than the negative sample threshold are assigned as background anchor boxes and do not participate in the loss calculation. To accommodate targets of various scales and shapes and to normalize changes in center coordinates and size, the parameters of the anchor boxes assigned as positive samples and their corresponding ground truth boxes are encoded.

[0111]

[0112] Where x a y a , z a w a , l a h a θ a These are the position parameters of the anchor frame, d a x is the diagonal distance between the length and width; p y g , z gw g , l g h g θ g These are the position parameters of the truth boxes.

[0113] The loss function is calculated based on the truth box and category assigned to each anchor box. The total loss function is:

[0114] L total =β1L cls +β2(L reg-θ +L reg-other )+β3L air

[0115] Where β1, β2, and β3 are 1.0, 2.0, 0.2, and L, respectively. cls L represents the object classification loss. reg-θ L represents the regression loss of the anchor frame angle. reg-other L represents the regression loss for anchor frame position and size. dir This represents the directional classification loss.

[0116] L cls The loss is calculated using the Focal loss method, and its formula is as follows:

[0117] FL(p t )=-α t (1-p t ) γ log(p t )

[0118] Where p t α represents the class probability predicted by the model. t γ is the class weight, used to balance class imbalance, with a value of 0.25. γ is an adjustment factor used to adjust the impact of easy and difficult samples on the loss function, with a value of 2.

[0119] L reg-other The loss is calculated using the SmoothL1 function, L reg-θ The following adjustments were made, and the calculation formula is as follows:

[0120] L reg-θ =SmoothL1(sin(θ) p -θ t ))

[0121] θ p θ is the predicted direction angle. t For the true direction angle, when θ p and θ t When the phase difference is 180°, the loss also approaches zero. To address the issue of reversal, an additional L is introduced. dirThe loss is calculated by using a one-hot vector for each anchor box to determine the orientation classification. dir The loss is calculated using the cross-entropy function.

[0122] Step 6: Reasoning Steps.

[0123] During inference, after predicting the classification result and bounding box regression for each anchor box, a decoding operation is performed based on the anchor box position and the prediction result to obtain the predicted bounding box. Finally, NMS is performed to remove redundant predicted boxes to obtain the prediction result. Specifically, if the orientation classification result is opposite, the angle of the predicted bounding box regression is increased by 180°.

[0124] The effectiveness of the algorithm of this invention can be further illustrated by the following comparative experiments:

[0125] I. Experimental Environment:

[0126] Operating System: Ubuntu 20.4.0

[0127] CPU: Intel Core i7-10700

[0128] GPU: RTX 3060 Ti

[0129] Deep learning framework: PyTorch

[0130] II. Experiment Content

[0131] Comparative experiments show that the method of this invention, compared with existing single-stage 3D target detection networks, can significantly improve the utilization rate of sparse point cloud information, thereby improving the accuracy of target recognition results.

[0132] The dataset used is KITTI.

[0133] The KITTI dataset, jointly created in 2012 by the Karlsruhe Institute of Technology (KIT) in Germany and the Toyota Technological Institute at Chicago (TTI-C) in the United States, is one of the most widely used international datasets for evaluating computer vision algorithms in autonomous driving scenarios. This dataset is used to evaluate the performance of computer vision technologies such as stereo imaging, optical flow, visual odometry, 3D object detection, and 3D tracking in in-vehicle environments. The KITTI dataset contains real-world image data collected from urban, rural, and highway scenes. Each image contains up to 15 vehicles and 30 pedestrians, with varying degrees of occlusion and truncation. For the 3D object detection task, the KITTI dataset provides 14,999 images and corresponding point clouds, of which 7,481 sets are used for training and 7,518 sets are used for testing. Annotations are provided for three object classes in the scene: cars, pedestrians, and bicycles, totaling 80,256 labeled objects.

[0134] Table 1 compares existing single-stage 3D object detection networks with the AP|R network of this invention under Moderate difficulty 3D perspective. 40 Value comparison

[0135] PointPillar 77.2846 47.1352 60.7532 SECOND 77.7459 47.7311 64.5856 CenterPoint-Pillar 77.3763 48.2386 65.3561 CenterPoint-Dynamic-Pillar 77.2397 48.3362 65.7378 CenterPoint 78.2481 48.3782 66.0331 Target detection network of the present invention 78.5145 50.9161 67.9398

[0136] Measurement criteria:

[0137] Average precision is the average of the precision values ​​on the PR curve, and this metric is commonly used in the field of object detection. Its formula is as follows:

[0138] AP=∫0 1 P(r)dr

[0139] Where P(r) is the precision-recall curve. The main difference between AP and 2D object detection lies in the matching criteria between ground truths and predictions when calculating precision and recall. KITTI proposed two widely used AP metrics: AP3D and APBEV. AP3D matches the predicted object with its respective ground truths when the 3D intersection (3D IoU) of two cuboids exceeds a certain threshold, while APBEV matches based on the IoU (BEV IoU) of the two cuboids from a bird's-eye view. However, due to the complexity of integral calculations, interpolation methods are often used in practical applications to calculate AP values, R. 40To uniformly select 40 values ​​between 0 and 1:

[0140]

[0141] p interp (r) = maxP(r′) r′>r

[0142] The experimental results are shown in Table 1. This invention, under the Moderate difficulty of the KITTI dataset, achieves the following AP|R in 3D view for the Car category with an IOU threshold of 0.7: 40 The metric is 78.5145. For the Pedestrian category, with an IOU threshold of 0.5, the AP|R in 3D view is... 40 The metric is 50.9161. For the Cyclist category, with an IOU threshold of 0.5, the AP|R in the 3D view is... 40 The index is 67.9398, and the prediction accuracy of each category is improved compared with the existing single-stage invention method. This is due to the learning of sparse spatial information and the sensitivity to point cloud density. The improvement is more significant for the Pedestrian and Cyclist categories.

[0143] Figure 5 (a) Visualization of the prediction results of the SECOND network in the BEV perspective; (b) Visualization of the prediction results of the present invention in the BEV perspective; (c) Visualization of the prediction results of the SECOND network in the 3D perspective; (d) Visualization of the prediction results of the present invention in the 3D perspective. Blue represents the ground truth bounding box, green represents the predicted bounding box for the Car category, yellow represents the predicted bounding box for the Pedestrian category, and light blue represents the predicted bounding box for the Cyclist category. It can be concluded that the detection results provided by the present invention significantly reduce the number of false positive predicted bounding boxes, and the Pedestrian and Cyclist categories are almost all accurately predicted true positive bounding boxes. Furthermore, the quality of the true positive predicted bounding boxes provided by the present invention is significantly better than that of the SECOND network, proving that the method of the present invention can fully utilize point cloud information, thereby improving the detection effect.

Claims

1. A single-stage point cloud density-aware focused convolutional 3D target detection method, characterized in that, Includes the following steps: Step 1: Dynamically allocate point cloud data from the dataset to voxels at corresponding locations. Calculate the distances from the point cloud to the physical center of the voxel and the distances from the point cloud to the centroid of the voxel as additional features. Then, fuse these features using a feedforward-based feature fusion network to obtain voxel features. Specifically, this includes: Step 11: Obtain point cloud data from the original dataset, and assign a unique voxel code to each point cloud based on its 3D coordinates and mesh size. in Representative point cloud The unique voxel code, Representative point cloud The batch it belongs to, , , For point clouds Grid coordinates, , , Different grid scaling factors are used to construct a unique code for each voxel; Step 12: Calculate each voxel based on the unique voxel code of each point cloud. Position coordinates and voxels Mean of interior point cloud coordinates Its value is the centroid of the voxel, and the point cloud is calculated. The distance to the center of gravity of the voxel, as an additional feature. ; Step 13: Determine the physical center of each voxel based on its size and the extent of the point cloud, and calculate the point cloud. Distance to the voxel physical center, as an additional feature ; Step 14, convert the point cloud The original characteristics, , Cascading yields new features , , Two additional features reflect the density information of the point cloud, Input feature fusion network to obtain voxels Features ; in This represents the maximum pooling operation. This represents a multilayer perceptron network used for feature extraction. It is a voxel A collection of interior point clouds; Step 2: Input the voxel features into a 3D residual focusing sparse convolutional network for downsampling to obtain sparse 3D high-dimensional features that have learned the importance of spatial location. Step 3: The sparse 3D high-dimensional features are transformed into dense tensors and compressed in the Z-axis direction. The feature z-axis is multiplied by the number of feature channels to obtain the BEV 2D features. Step 4: The BEV 2D features are fed into a dual-branch 2D convolutional backbone network for downsampling, upsampling and channel connection operations to extract deeper semantic features, and the final BEV feature map is obtained. Step 5: Use the final BEV feature map for prediction.

2. The single-stage point cloud density-aware focused convolutional 3D target detection method according to claim 1, characterized in that, Step 2, the step of obtaining sparse 3D high-dimensional features, includes: Step 21: Generate a sparse voxel feature tensor based on the voxel features, voxel position coordinates, and voxel size. ; Step 22, Feature Input Convolutional blocks, unifying the input feature dimensions to obtain convolutional blocks It consists of a residual submanifold sparse convolutional layer, a batch normalization layer, and a ReLU activation function layer; Step 23, Input to The convolution block is subjected to a convolution operation to obtain... convolutional blocks It consists of a residual submanifold sparse convolution module and an aggregateable sparse convolution module; Step 24, Input to The convolutional block undergoes its first downsampling convolution operation to obtain... convolutional blocks It consists of a residual regular sparse convolution module, two residual submanifold sparse convolution modules, and a clusterable sparse convolution module; Step 25, Input to The convolutional block undergoes a second downsampling convolution operation to obtain... convolutional blocks It consists of a residual regular sparse convolution module, two residual submanifold sparse convolution modules, and a clusterable sparse convolution module; Step 26, Input to The convolutional block undergoes a third downsampling convolution operation to obtain... convolutional blocks It consists of one residual regular sparse convolution module and two residual submanifold sparse convolution modules; Step 27, Input to The convolution block is subjected to a convolution operation to obtain... convolutional blocks It consists of a regular sparse convolutional layer, a batch normalization layer, and a ReLU activation function layer.

3. The single-stage point cloud density-aware focused convolutional 3D target detection method according to claim 2, characterized in that, The residual submanifold sparse convolution module consists of a residual submanifold sparse convolution layer, a batch normalization layer, and a ReLU activation function layer; the residual regularized sparse convolution module consists of a residual regularized sparse convolution layer, a batch normalization layer, and a ReLU activation function layer. The operation of the residual submanifold sparse convolutional layer is as follows: in Features representing sparse convolutional layers of the input residual submanifold. Represents sparse convolution operations on submanifolds. Represents residual connectivity. This represents the features obtained after passing through a sparse convolutional layer of a residual submanifold; The operation of the residual regularized sparse convolutional layer is as follows: in Features representing the input residuals of a regularized sparse convolutional layer. This represents a regular sparse convolution operation. Represents residual connectivity. This represents the features obtained after passing through a residual regularized sparse convolutional layer.

4. The single-stage point cloud density-aware focused convolutional 3D target detection method according to claim 2 or 3, characterized in that, The operations of the aggregateable sparse convolution module include: (a) Using a submanifold sparse convolution Based on the input voxels voxel characteristics The effective location, predicting the effective feature unit and its surroundings. The importance of each feature unit, with importance values ​​of respectively. and , Refers to the kernel size; (b) Input voxel features and Multiplication allows voxel features to learn the importance of spatial information and also facilitates subsequent backpropagation calculations. in Voxel features learned after spatial importance; (c) Based on the threshold Distinguishing Indexed masks of foreground and background points in and and according to get : Where [] represents Boolean indexing operations, The index mask of the units whose importance value is greater than the threshold among the extended feature units that are judged to be valid feature units of the foreground point; (d) Through and get Foreground features and background point features And filter out Feature units that are foreground points and have importance values ​​greater than the threshold The feature units it represents are the important locations where additional convolution calculations will be added later: (e) in Features with pre-connection values ​​of 0 represent new, important locations that need to be added to the convolutional computation, enabling the convolutional network, which has learned spatial information, to output dynamically. Finally, the connections... Obtain complete voxel feature output : in The number of The number of positions, i.e., the number of new positions that need to be added to the convolution calculation. The number of zeros in the middle is Dimensions of voxel features; (f) will Inputting a submanifold sparse convolutional layer, followed by a batch normalization layer and a ReLU activation function layer, yields the output of the final aggregateable sparse convolutional module. .

5. The single-stage point cloud density-aware focused convolutional 3D target detection method according to claim 4, characterized in that, In (e), let and Multiplication, thereby enabling all feature units, including the newly extended ones, to learn the spatial importance of the feature, is expressed as: Accordingly, in (f), let Alternative .

6. The single-stage point cloud density-aware focused convolutional 3D target detection method according to claim 5, characterized in that, Following (f), it also includes: (g) Based on the more valuable prior knowledge of the foreground points, additional super-supervision is added for... and Calculate the Focal loss between the predicted importance value and the true value of the foreground point, and add the Focal loss to the loss of the entire network to facilitate backpropagation.

7. The single-stage point cloud density-aware focused convolutional 3D target detection method according to claim 1, characterized in that, Step 4, which involves obtaining the final BEV feature map, includes: Step 41, feature Input downsampling convolution branch 1, and get ; Step 42, feature Input downsampling convolution branch 2, and get ; Step 43, Upsampling is performed on the input convolution branch 1 to obtain... ; Step 44, will Input upsampling convolution branch 2 is used for upsampling, resulting in ; Step 45, will and Perform a concatenation operation along the channel dimension to obtain the final BEV feature map. .

8. The single-stage point cloud density-aware focused convolutional 3D target detection method according to claim 1, characterized in that, In step 5, the final BEV feature map is fed into the RPN prediction head network for class prediction, bounding box regression, and orientation classification tasks to obtain the final prediction result.

9. The single-stage point cloud density-aware focused convolutional 3D target detection method according to claim 8, characterized in that, The steps to obtain the final prediction result include: Step 51: Generate all anchor boxes according to the required categories, sizes, and orientations to be detected; Step 52, generate the final BEV feature map. Input classification convolution Boundary regression convolution Convolutional directional classification This yields the predicted classification result, predicted bounding box regression result, and predicted orientation classification result for each anchor box. Step 53: Assign the truth class, orientation class, and truth box to each anchor box based on the positive sample threshold and negative sample threshold for each category; Step 54, based on each positive sample anchor box And the bounding box encoding of the truth box it is assigned. This is used for subsequent loss function calculation; Step 55: Based on the predicted classification result, predicted orientation classification result, and bounding box encoding of each anchor box. Calculate the loss function and train the network through backpropagation; Step 56: During inference, after predicting the classification result and bounding box regression for each anchor box, a decoding operation is performed based on the anchor box position and prediction result. Finally, NMS is performed to remove redundant prediction boxes to obtain the prediction result.

Images