Single-stage 3D point cloud target detection method based on deep learning

Abstract

The invention discloses a single-stage 3D point cloud target detection method based on deep learning, and aims to improve the accuracy and time cost of point cloud target detection. Firstly, point cloud data are processed into voxel data, a six-stage form is adopted in a 3D backbone network, Sparse Bottleneck is applied to perform step-by-step dimensionality reduction, convolution extraction and expansion on features in each stage, and then multi-scale features are applied to fuse the features of the last three stages to obtain output 3D features; in a 2D backbone network, two-stage processing of dynamic overlapping processing and space-time attention is adopted, and point cloud features, especially small target features, are refined. And finally, in a post-processing stage, a regression prediction frame is calibrated by applying a distance-based NMS, and finally prediction is performed to obtain a final prediction frame. According to the method, an effective tool is provided for point cloud target detection, valuable reference is provided for related researchers in the field, and further development of the point cloud target detection technology is promoted.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision and artificial intelligence, and mainly aims at a large-scale outdoor point cloud object detection method, and particularly relates to a single-stage 3D point cloud object detection method based on deep learning. Background Art

[0002] Point cloud object detection is an important task in the fields of computer vision and autonomous driving. Compared with camera images, the 3D point cloud of LIDAR can provide accurate depth information and robust environmental information under different levels of light. Early 3D object detection mainly relied on lidar to achieve object recognition and positioning through direct processing of point cloud data. Traditional 3D point cloud object detection algorithms mostly rely on clustering algorithms, such as the RANSAC clustering algorithm and the Mean Shift clustering algorithm. However, the clustering algorithm has serious deficiencies in judging object categories and has very low accuracy.

[0003] With the rapid development of computer vision and deep learning technologies, to solve this problem, research on 3D point cloud object detection based on deep learning has been proposed, aiming to overcome the deficiencies of traditional methods in detection accuracy and inference efficiency. PointPillars efficiently performs 3D object detection by learning point cloud columnar features; VoteNet is an advanced framework for directly performing 3D object detection on point clouds, which uses a voting mechanism to aggregate feature information, thereby improving detection accuracy; Point-GNN uses a graph neural network (GNN) to model the structural information in point cloud data and improve detection performance; PV-RCNN combines Voxel and Point-based methods to improve object detection accuracy through a two-stage strategy.

[0004] These deep learning-based methods have achieved good results in detection accuracy, but most methods adopt a two-stage strategy, resulting in a high computational complexity and being difficult to meet the real-time detection requirements. At the same time, there are still deficiencies in multi-scale feature fusion, which limits the detection ability for objects of different scales. The present invention aims to solve the limitations of existing methods and proposes an improved single-stage 3D point cloud object detection method. This method introduces an efficient feature extraction module, adopts a cross-scale feature fusion strategy, and normalizes information with different spatial resolutions at the same time, so as to improve detection accuracy while reducing computational costs, thereby achieving efficient and accurate 3D object detection. Summary of the Invention

[0005] The purpose of the present invention is to provide a single-stage 3D point cloud object detection method based on deep learning. Based on the ordinary SCECOND-IOU model, by introducing 3D backbones and 2D backbones with different structures, the final features obtained are made more abundant and accurate, and the accuracy and robustness of object detection are improved.

[0006] To achieve the above technical objectives, the technical solution of the present invention is as follows:

[0007] A single-stage 3D point cloud object detection method based on deep learning, the detection method includes the following steps:

[0008] Preprocess the input 3D point cloud data and voxelize the processed point cloud data;

[0009] In the 3D backbone network, use a sparse convolutional backbone network containing a channel attention mechanism to extract features from the voxelized point cloud data; construct a feature pyramid structure, and use an atrous convolution fusion module to achieve cross-scale feature aggregation;

[0010] Compress the 3D voxel features along the height dimension in a concatenated form to form a BEV feature map and obtain a BEV feature map;

[0011] In the 2D backbone network, use a dynamic overlapping processing module and a spatio-temporal hybrid attention module to extract BEV features. After upsampling processing, input them into the lower-layer single-stage detection head;

[0012] Use the single-stage detection head to directly regress and predict the three-dimensional bounding box parameters and target categories of the objects in the point cloud data, post-process the prediction results, and obtain the final output result by optimizing the loss function;

[0013] The processing process of the dynamic overlapping processing module is: taking the BEV feature map as the input, generating an overlapping block sequence P according to the sliding step, and performing shared feature processing on each block. The shared feature processing is expressed as:

[0014] Γ share (Pi) = W2 * GELU(W1 * (DWConv(P i )))

[0015] where DWConv represents depth convolution, W1 and W2 are the weight matrices of point convolution, and Γ share is the shared convolution, and P i is a certain block in P;

[0016] After the shared processing, perform dynamic routing gating processing to obtain the feature P i ′, perform weighted combination reconstruction on P i ′ to obtain the enhanced BEV feature.

[0017] Furthermore, the 3D backbone network is divided into six stages with feature strides of {1, 2, 4, 8, 16, 32}. Each stage consists of a downsampling layer and several Sparse Bottleneck blocks. Inside each stage, the Sparse Bottleneck block uses a 1×1-3×3-1×1 convolution structure to gradually reduce the dimension of the features, perform convolution extraction and expansion, and at the same time use residual connections and channel attention modules to enhance feature expression;

[0018] For the deep features X5 and X6, upsampling is achieved through sparse transposed convolutions with different dilation rates, and after aligning to the feature scale of the X4 layer, they are fused through the Multi-Scale Fusion module.

[0019] Furthermore, the output of the dynamic overlapping processing module is downsampled by a ConvNeXt block. The ConvNeXt block includes a depthwise convolution layer, two pointwise convolution layers, a BN layer is attached to the depthwise convolution layer, a linear unit ReLU is applied to the first pointwise convolution layer. In addition, there is a residual connection between the input and the output; the first pointwise convolution layer expands the number of channels to twice the original size, while the second pointwise convolution layer reduces the number of channels to the original size.

[0020] Further, after being processed by the ConvNeXt block, it enters the spatio-temporal hybrid attention module to fuse three features of spatial enhancement, temporal fusion, and channel enhancement respectively;

[0021] Spatial attention uses a 3×3 convolution layer to map the input features to a single channel, and then through Sigmoid activation to obtain the spatial attention weight, and then uses this weight for spatial weighting; temporal fusion is to use 2D convolution to extract time information, and then add the extracted temporal features to the current features to achieve the fusion of time information; channel attention is to first perform global average pooling on the fused features to converge the information of each channel to a scalar, and then through two 1×1 convolutions and Sigmoid activation to generate the attention weight of each channel.

[0022] Furthermore, non-maximum suppression (NMS) is adopted in the post-processing to eliminate redundant bounding boxes generated during the detection process and only retain the detection result with the highest confidence, and finally obtain the detection result and detection accuracy.

[0023] The present invention also protects a computer program product for executing the above method. The product includes a storage medium and computer-executable instructions stored on the storage medium. The instructions are configured to implement the steps of the method when executed.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] In the method of the present invention, in terms of the 3D backbone network, from input preprocessing, phased feature extraction to upsampling and multi-scale fusion, the entire process maintains the sparse tensor structure while achieving efficient information transfer through residual and attention mechanisms, providing a solid feature foundation for subsequent detection tasks. Additionally, for the 2D backbone, a feature extraction network introducing dynamic overlapping processing and spatio-temporal attention mechanisms is adopted to improve the feature expression ability, which can largely retain the local details of features and also enhance the robustness of the model. This method has important value in tasks such as augmented reality and autonomous driving that require high precision and high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic flowchart of the single-stage 3D point cloud object detection method based on deep learning of the present invention;

[0027] Figure 2 is a schematic structural diagram of the 3D backbone network in the present invention;

[0028] Figure 3 is a schematic structural diagram of the 2D backbone network in the present invention;

[0029] Figure 4 is a schematic structural diagram of the ConvNext convolution in the present invention;

[0030] Figure 5 is a schematic structural diagram of the dynamic overlapping processing module in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] The present invention will be further described below in conjunction with the accompanying drawings and embodiments, but this is not intended to limit the protection scope of the present application.

[0032] Embodiment 1

[0033] As Figure 1 shown, the single-stage 3D point cloud object detection method based on deep learning of the present invention includes the following steps:

[0034] S1: Point cloud processing and voxelization;

[0035] First, noise removal processing is performed on the input point cloud data to eliminate isolated points and outliers; subsequently, the MeanVFE method is used to discretize the point cloud data into fixed voxel units, with a voxel grid of size 0.1m × 0.1m × 0.2m, and the point cloud density and position information are statistically calculated within each voxel; finally, the data is normalized.

[0036] S2: 3D backbone network;

[0037] As Figure 2 shown, for the original voxel feature X inputAfter SubM convolution, normalization, and activation, the preliminary feature X0 is obtained:

[0038] X0 = ReLU(BN(Conv SubM (X input )))

[0039] The 3D backbone network is divided into six stages, and the feature stride is {1, 2, 4, 8, 16, 32}. The features of the i-th stage can be described as:

[0040] X i = Stage i (X i-1 )

[0041] Each stage consists of a downsampling layer (when the stride > 1) and several Sparse Bottleneck blocks, which are hereinafter simply referred to as Bottleneck blocks. Inside each stage, the Bottleneck block uses a 1×1-3×3-1×1 convolution structure to gradually reduce the dimension of the features, perform convolution extraction and expansion, and at the same time use residual connections and channel attention modules to enhance the feature representation. Each Stage i can be described as:

[0042]

[0043] where n i represents the number of Bottleneck blocks in the i-th stage, and Downsample represents downsampling.

[0044] For the deep features X5 and X6, upsampling is achieved through sparse transposed convolutions with different dilation rates, and after aligning to the feature scale of the X4 layer, they are fused through the Multi-Scale Fusion module to intelligently fuse features at various scales, significantly improving the robustness to target scale changes. The described sparse convolutional backbone network incorporates a channel attention mechanism, and the preliminary features contain rich local information for subsequent attention operations in feature processing.

[0045] The multi-scale feature pyramid structure constructs a hierarchical feature representation, making the extracted features more abundant.

[0046] The dilated convolution fusion module corresponds to different spatial resolutions to achieve cross-scale context awareness, that is, cross-scale enhancement

[0047] S3: High-dimensional compression operation Height-Compression;

[0048] Through the view(N, C*D, H, W) operation, the height dimension D is merged into the channel dimension C to form a new 2D feature map (N, C×D, H, W), denoted as the BEV feature map, which can reduce the dimension and improve the computational efficiency. Compress the 3D feature along the height dimension to generate the BEV feature map for subsequent processing.

[0049] S4: 2D Network Backbone - Dynamic Overlap Processing;

[0050] As Figure 3 shown, the 2D network backbone includes four modules: dynamic overlap processing, downsampling, spatio-temporal attention module, and upsampling. In part S4, only the dynamic overlap processing part is introduced here. To overcome the limitations of the fixed receptive field, the input of the dynamic overlap processing module (DOFM) is the BEV feature map Take the sliding stride s t :

[0051]

[0052] where s is the base block size and ρ is the overlap rate. Generate an overlapping block sequence P according to the sliding stride:

[0053]

[0054] where P i is a certain block, h and w are the length and width of each block respectively, and N is the number of output blocks.

[0055] Then each block undergoes shared feature processing.

[0056] Apply the dynamic routing gating architecture to the overlapping block sequence P produced by the sliding stride, and perform dynamic weighted fusion between the original processed features and the features after pooling operation, so as to achieve the adaptive adjustment and enhancement of features. The specific process is:

[0057] P i ' = Γ route (Γ share (P i ))

[0058] where Γ share is the shared convolution, Γ route is the routing gate,

[0059] Γ share can be further expressed as:

[0060] Γ share (Pi) = W2 * GELU(W1 * (DWConv(P i ))) where DWConv represents depth convolution, and W1 and W2 are the weight matrices of point convolution; the shared convolution

[0061] The processed feature is denoted as P is Routing gating Γ route can be described as:

[0062] g i = σ(W g *P is )(routing weight)

[0063] P i ′ = g i ☉P is +(1 - g i )⊙AvgPool(P is )

[0064] where σ is the Softmax function, and W g is the weight.

[0065] After that, the feature map F enh (x, y) is reconstructed through weighted fusion to obtain the enhanced BEV feature:

[0066]

[0067] where, P i ′(x, y) is the feature of a certain pixel point in P i ′, wi is the hyperparameter weight, and ε is the hyperparameter.

[0068] The dynamic overlapping processing module uses overlapping patch segmentation and adaptive dynamic routing to effectively extract and fuse local and global information.

[0069] S5: 2D network backbone - spatio-temporal hybrid attention;

[0070] After dynamic processing, the ConvNeXt block is used to continue extracting the enhanced BEV feature and perform downsampling. The structure of the ConvNeXt block is as Figure 4 shown. The ConvNeXt block includes a depth convolution layer, followed by two pointwise convolution layers. A BN layer is attached to the depth convolution layer, and a linear unit (ReLU) is applied to the first pointwise convolution layer. In addition, there is a residual connection between the input and the output. The first pointwise convolution layer expands the number of channels to twice the original size, while the second pointwise convolution layer reduces the number of channels to the original size.

[0071] Spatio-temporal hybrid attention is a parallel process that separately fuses three features: spatial enhancement, temporal fusion, and channel enhancement. Spatial attention uses a 3×3 convolutional layer to map the input features to a single channel, and then through Sigmoid activation to obtain the spatial attention weights, and then uses these weights for spatial weighting; temporal fusion is to use 2D convolution to extract time information, and then add the extracted temporal features to the current features to achieve the fusion of time information; channel attention is to first perform global average pooling on the fused features to converge the information of each channel into a scalar, and then through two 1×1 convolutions (with ReLU activation in the middle) and Sigmoid activation to generate the attention weights for each channel. Overall process:

[0072] F temp = T-Conv(Concat[F t-2 , F t-1 , F t )

[0073] A space = σ(W s ★ F temp )

[0074] A channel = σ(W c (GAP(F temp )))

[0075] F out = A channel ☉(A space ☉)F temp )

[0076] Where T-Conv is the temporal convolution kernel and GAP is the global average pooling.

[0077] Finally, after concatenation and upsampling, the final 2D features are obtained.

[0078] S6: Multi-task head;

[0079] The misalignment between localization accuracy and classification confidence is a common problem encountered in single-stage detectors. To solve this problem, an IOU prediction branch is adopted on the multi-task head. More specifically, the output feature map of the 2D backbone is parallelly passed through four 1×1 convolutional layers to generate separate predictions for each task.

[0080] S7: Post-processing;

[0081] To better generate prediction boxes, Distance-varied IOU-weighted NMS (DI-NMS) is introduced for post-processing. DI-NMS corrects the confidence based on the distance between the prediction box and the anchor box. Then, during non-maximum suppression (NMS), it not only considers the IoU overlap between candidate boxes but also combines the IoU prediction values for weighted fusion, and the loss function is involved in this process. Specifically, by calculating the cumulative weights for the candidate box population and using weighted averaging with a decay factor to generate new bounding boxes, the final output result becomes more robust and has higher precision.

[0082] S8: Output result;

[0083] Finally, by matching the detection results with the ground truth, metrics such as average precision (AP) and average recall (AR) are calculated to evaluate the detection performance of the model.

[0084] Performance evaluation is carried out after all processes are completed. It is mainly evaluated using the metrics Miou (mean intersection over union) and OA (overall accuracy) on the large-scale outdoor dataset KITTI.

[0085] The KITTI dataset is a large-scale dataset for autonomous driving scenarios and is widely used in multiple research directions such as object detection, semantic segmentation, instance segmentation, tracking, stereo vision, optical flow, and visual odometry. This dataset is equipped with 7,481 training samples and 7,518 test samples. The training samples are further divided into a training set (3,712 samples) and a validation set (3,769 samples). The dataset mainly collects data by installing high-resolution binocular cameras, Velodyne lidars, and GPS / IMU systems on moving vehicles, covering various driving environments such as urban scenes, highways, and rural roads, and obtaining rich image and 3D point cloud information. Each captured frame provides detailed 2D and 3D object detection annotations, covering object categories such as vehicles, pedestrians, and bicycles, and provides 2D bounding boxes, 3D bounding boxes, and corresponding class labels for each object. At the same time, it also supports tasks such as stereo vision, optical flow, tracking, and semantic segmentation, and is an important benchmark dataset in the field of autonomous driving and computer vision.

[0086] In the experiment, the 3D average precision of vehicle detection at a recall rate of 40 on the KITTI test set and the 3D average precision calculated based on 11 sampling recall points are compared between the present invention and other networks.

[0087] Table 1 Comparison of experimental results between the present invention and other networks on the KITTI test set (40)

[0088]

[0089]

[0090] Table 2 Comparison of experimental results of the present invention and other single-stage networks on KITTI (11)

[0091]

[0092] It can be seen that, regardless of the recall rate, the results of the model of the present invention on the KITTI dataset are higher than those of many other methods. Moreover, compared with the current popular single-stage object detection model, its time cost is only half of that of the other model, and the results are more accurate than that of the other model. This shows that both the accuracy and robustness of the present invention are relatively good, indicating that it has good effects when completing the object detection task.

[0093] Through these evaluations, the method of the present invention can clearly identify the advantages of the new method and potential improvement directions, providing guidance for future research. At the same time, these results also provide valuable references for researchers in related fields, promoting the further development of point cloud object detection technology.

[0094] Example 2

[0095] The method for single-stage 3D point cloud object detection based on deep learning in this embodiment includes point cloud processing, voxelization, 3D backbone network, 2D backbone network, detection head with iou, loss function optimization module, etc.; combining multi-scale feature fusion and spatio-temporal attention mechanism to achieve efficient detection of 3D objects in complex scenes.

[0096] The preliminary point cloud voxelization process adopts a voxel grid filtering method to discretize the continuous 3D point cloud data that has been enhanced, rotated, and cropped into voxels of a fixed size, and summarize the point information in each voxel, converting it into a format suitable for deep network processing to improve the efficiency and robustness of subsequent feature extraction.

[0097] The 3D backbone network module is a new type of three-dimensional backbone network based on sparse convolution. This network makes full use of the sparsity of lidar point cloud data. Through multi-scale feature extraction and fusion, channel attention mechanism and residual learning, after multi-layer feature extraction, each layer contains a downsampling and multiple Sparse Bottleneck modules, so that the voxelized point cloud data can obtain efficient and strong expressive feature data after passing through the backbone network.

[0098] The 2D backbone network module includes three core modules: a dynamic overlapping processor that divides the input feature map into dynamically overlapping patches, performs feature extraction and fusion respectively, and improves the ability to retain local details; a BEV feature extraction backbone network that constructs the backbone network for BEV perception, uses ConvNeXt blocks as feature extraction units, and realizes feature enhancement and extraction; a spatio-temporal attention mechanism that enhances the feature expression ability and suppresses redundant information through attention in the channel, space, and time dimensions. The three modules complement each other to achieve efficient 2D backbone network processing.

[0099] In the post-processing and loss function optimization module, the IOU prediction branch is trained in the post-processing stage simultaneously with the bounding box regression and classification branches. The predicted bounding boxes are separated from the computational graph in the IOU prediction and are processed with DI-NMS. In terms of the loss function, the focal loss, smooth-L1 loss, and bounding box classification, box regression, and orientation classification losses are used for object detection, and the weights of each loss are optimized.

[0100] The overall process is divided into five stages: point cloud processing, 3D backbone network, 2D backbone network, IOU post-processing, and regression prediction evaluation. From feature extraction to 3D depth processing, and then to 2D BEV processing, it enters the detection head regression prediction, and finally, each part performs its own functions and complements each other, jointly building this 3D point cloud object detection method based on deep learning.

[0101] The commonly used voxel processing network is adopted to first process the point cloud data into voxel data. The process of deep learning is reflected in two parts: the 3D backbone network and the 2D backbone network. In the 3D backbone network, a six-stage form is adopted. In each stage, Sparse Bottleneck is used to gradually reduce the dimension of the features, perform convolution extraction and expansion, and then multi-scale feature fusion is applied to fuse the features of the last three stages to obtain the output 3D features. In the 2D backbone network, a two-stage processing of dynamic overlapping processing and spatio-temporal attention is proposed to refine the point cloud features, especially the small target features. Finally, in the post-processing stage, distance-based NMS is applied to calibrate the bounding boxes predicted by regression, and finally, predictions are made to obtain the final predicted bounding boxes. This method provides an effective tool for point cloud object detection and also provides valuable reference for relevant researchers in this field, promoting the further development of point cloud object detection technology.

[0102] Matters not described in this invention are applicable to the prior art.

Claims

1. A single-stage 3D point cloud object detection method based on deep learning, characterized in that, The detection method comprises the following steps: Preprocess the input 3D point cloud data and voxelize the processed point cloud data; In the 3D backbone network, a sparse convolution backbone network with a channel attention mechanism is used to extract features from voxelized point cloud data; a feature pyramid structure is constructed, and a dilated convolution fusion module is used to achieve cross-scale feature aggregation; The 3D voxel features are compressed along the height dimension in a serial manner to form a BEV feature map, thereby obtaining a BEV feature map; In the 2D backbone network, the dynamic overlap processing module and the spatiotemporal hybrid attention module are used to extract BEV features, which are then input into the lower-layer single-stage detection head after upsampling. The single-stage detection head is used to directly regress and predict the 3D bounding box parameters and target category of the target in the point cloud data, and the prediction results are post-processed to obtain the final output results by optimizing the loss function. The processing process of the dynamic overlap processing module is: taking the BEV feature map as input, generating an overlapping block sequence P according to the sliding step length, and performing shared feature processing on each block. The shared feature processing is expressed as: Gamma share (Pi) = W2 * GELU(W1 * (DWConv(P i ))) Among them, DWConv represents depthwise convolution, W1 and W2 are the weight matrices of pointwise convolution, and Γ share is a shared convolution, and P i is a certain block in P; After shared processing, it undergoes dynamic routing gating processing to obtain feature P i ′, and for P i ′, weighted coincidence reconstruction is performed to obtain the enhanced BEV feature.

2. The method according to claim 1, wherein The 3D backbone network is divided into six stages with feature step sizes of {1, 2, 4, 8, 16, 32}. Each stage consists of a downsampling layer and several Sparse Bottleneck blocks. Within each stage, the Sparse Bottleneck block uses a 1×1-3×3-1×1 convolution structure to gradually reduce the dimension, extract convolutions, and expand the features. At the same time, residual connections and channel attention modules are used to improve feature expression. For the deep features X5 and X6, upsampling is achieved through sparse inverse convolution with different expansion rates, and they are aligned to the feature scale of the X4 layer and then fused through the Multi-Scale Fusion module.

3. The method according to claim 1, wherein The output of the dynamic overlap processing module is downsampled by the ConvNeXt block, which includes a deep convolutional layer, two point-by-point convolutional layers, a BN layer attached to the deep convolutional layer, and a linear unit ReLU applied to the first point-by-point convolutional layer. In addition, there is a residual connection between the input and output; the first point-by-point convolutional layer expands the number of channels to twice the original size, while the second point-by-point convolutional layer reduces the number of channels to the original size.

4. The method according to claim 3, characterized in that, After being processed by the ConvNeXt block, it enters the spatiotemporal hybrid attention module to fuse the three features of spatial enhancement, temporal fusion, and channel enhancement respectively; Spatial attention uses a 3×3 convolutional layer to map the input features into a single channel, and then activates it through Sigmoid to obtain the spatial attention weight, which is then used to perform spatial weighting. Temporal fusion uses 2D convolution to extract temporal information, and then adds the extracted temporal features to the current features to achieve the fusion of temporal information; channel attention first performs global average pooling on the fused features, aggregates the information of each channel into a scalar, and then generates the attention weight of each channel through two 1×1 convolutions and Sigmoid activation.

5. The method according to claim 1, wherein In the post-processing, non-maximum suppression (NMS) is used to eliminate redundant bounding boxes generated during the detection process and only retain the detection result with the highest confidence, finally obtaining the detection result and the detection accuracy rate.

6. A computer program product for performing the method according to any one of claims 1 to 5, characterized in that, The product includes a storage medium and computer-executable instructions stored on the storage medium, and the instructions are configured to implement the steps of the method when executed.

Images