A 3D point cloud target detection method based on a state space model

Abstract

The application belongs to the field of automatic driving environment perception, and particularly relates to a 3D point cloud target detection method based on a state space model; comprising: collecting road scene point cloud data and labeling a target to be detected; dividing the point cloud data into a regular voxel grid, using a voxel coding module to perform feature coding on the voxel grid to obtain voxel features; inputting the voxel features into a 3D backbone network for processing to obtain multi-scale fusion features; projecting the multi-scale fusion features to a BEV space, and using a 2D backbone network to extract BEV features; using a head network to process the BEV features to obtain a target detection result; calculating a total loss of the model according to the target detection result and a point cloud label, and adjusting model parameters according to the total loss of the model to obtain a trained model; and using the trained model to perform target detection; the application not only improves model detection precision but also speeds up inference.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving environmental perception, specifically involving a 3D point cloud target detection method based on a state space model. Background Technology

[0002] With the rapid development of autonomous driving technology in recent years, 3D point cloud object detection has become a research field of great interest. Point cloud data, with its rich spatial information and accurate 3D positioning capabilities, has become key to achieving high-precision environmental perception and object recognition. However, the sparse, uneven, and irregular distribution of point cloud data makes efficient and effective 3D object detection a very challenging task.

[0003] To address these challenges, researchers have proposed several mainstream methods: Voxelization-based methods discretize point cloud data into regular 3D meshes, enabling them to be processed by traditional convolutional operations. Their advantage lies in their fast processing speed, meeting real-time requirements; however, information loss is unavoidable during voxelization. Point-based methods achieve efficient point cloud processing through point-by-point feature extraction and global feature aggregation. They avoid the information loss caused by voxelization, improving the detection accuracy and efficiency of the model to some extent. However, directly processing the features of each point may face challenges such as high computational complexity and insufficient real-time performance.

[0004] Against this backdrop, state-space models, as an emerging technology, have been introduced into 3D point cloud object detection. By modeling the dynamic characteristics of the system, state-space models can not only effectively process temporal information but also play a crucial role in spatial data processing. Combining state-space models with the U-Net architecture can fully leverage their advantages in feature extraction and multi-scale information fusion. The U-Net structure, with its encoder-decoder form, can extract deep features during downsampling and restore spatial resolution during upsampling, while preserving low-level features through skip connections. Compared to traditional 3D convolutional neural networks, the U-Net architecture based on state-space models has significant advantages in computational efficiency and feature extraction capabilities. Its use of sparse convolutions effectively reduces computational load, adapting to the needs of large-scale data processing. Simultaneously, this method better preserves the integrity of spatial information during feature extraction, improving object detection accuracy. Compared to the PointNet series of methods, state-space models exhibit better scalability and efficiency when processing large-scale data and have a stronger ability to integrate multi-scale features, improving detection performance without significantly increasing computational overhead. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a 3D point cloud target detection method based on a state-space model. The method includes: acquiring point cloud data of a road scene to be detected and inputting it into a trained point cloud target detection model for processing to obtain target detection results.

[0006] The training process of a point cloud object detection model includes:

[0007] S1: Collect point cloud data of the road scene and label the targets to be detected;

[0008] S2: Divide the point cloud data into a regular voxel grid, and use a voxel encoding module to encode the features of the voxel grid to obtain voxel features;

[0009] S3: Input the voxel features into the 3D backbone network for processing to obtain multi-scale fused features;

[0010] S4: Project the multi-scale fused features onto the BEV space and extract BEV features using a 2D backbone network;

[0011] S5: Use the head network to process the BEV features to obtain the target detection results;

[0012] S6: Calculate the total loss of the model based on the target detection results and point cloud labels; use the optimizer to adjust the model parameters based on the total loss of the model to obtain the trained point cloud target detection model.

[0013] Preferably, the process of the voxel encoding module performing feature encoding on the voxel grid includes: obtaining the coordinates of each point in each voxel grid and calculating the mean and covariance of the voxel grid based on the coordinate vector of each point; merging the mean and covariance to obtain the voxel features.

[0014] Preferably, the 3D backbone network includes four encoding blocks, three decoding blocks, and three feature enhancement modules; both the encoding and decoding blocks consist of multiple ZOMB modules and sparse convolutional layers alternately connected; the output features of the four encoding blocks are {f d1 ,f d2 ,f d3 ,f bottom}, will {f d1 ,f d2 ,f d3 The inputs are fed into three feature enhancement modules for lateral feature enhancement, resulting in the enhanced features {f}. m1 ,f m2 ,f m3};f bottom After upsampling, and with f m3 As the input to the third decoding block; the output features of the three decoding blocks are respectively {f u1 ,fu2 ,f u3}, f u3 After upsampling, and with f m2 As the input to the second decoding block, f u2 After upsampling, and with f m1 As the input to the first decoded block; {f u1 ,f u2 ,f u3 ,f bottom The sums are then normalized to obtain the multi-scale fused feature f. 3d_out .

[0015] Furthermore, the ZOMB module consists of location embedding, Z-Order curve sorting, and a state-space model.

[0016] Furthermore, the feature enhancement module's processing of the input features includes: performing a sparse convolution with a kernel size of (1,1,1) on the input features to obtain intermediate features; feeding the intermediate features into three branches: the first branch performs a sparse convolution with a kernel size of (3,3,3) on the intermediate features; the second branch compresses the intermediate features and then performs a residual connection with the input features; the third branch does not perform any processing; and the outputs of the three branches are weighted and summed to obtain the output features of the feature enhancement module.

[0017] Preferably, the 2D backbone network is a 2D convolutional neural network.

[0018] Preferably, the head network uses a central detector head to process the BEV features and generate classification information and 3D BBOX information of the predicted target.

[0019] Preferably, the total model loss is a weighted sum of heatmap loss, target size loss, object orientation regression loss, and center point offset loss.

[0020] The beneficial effects of this invention are as follows: This invention provides a 3D point cloud target detection method based on a state-space model. The 3D backbone network uses the ZOMB module composed of the Mamba state-space model for feature learning. The Mamba model, which has a faster inference speed and stronger ability to capture global information, is used for 3D feature extraction. The overall UNet network structure is adopted to obtain multi-scale features, and a lateral feature enhancement module is proposed to strengthen the intermediate features in the downsampling process. Finally, global 3D features containing multi-scale information are obtained for the input of the next stage, which improves the model inference speed and detection accuracy. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the implementation of the point cloud target detection model in this invention.

[0022] Figure 2 This is a schematic diagram of the 3D backbone network structure in this invention;

[0023] Figure 3 This is a schematic diagram of the Z-Order curve traversal method of the present invention;

[0024] Figure 4 This is a schematic diagram of the feature enhancement module structure of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Autonomous driving systems typically consist of three layers: perception, decision-making, and execution. Sensors acquire environmental information about the vehicle's surroundings, extracting richer and more accurate features, which are then sent to the decision-making layer for judgment and to formulate corresponding control strategies. These strategies are then sent to the execution layer for motion control of the vehicle. The perception layer is particularly crucial, as the accuracy of its perception of the vehicle's surroundings determines subsequent decisions and motion control. Roads are generally complex environments, containing various vehicles, pedestrians, and traffic signs. Therefore, this invention provides a 3D point cloud target detection method based on a state-space model. This method primarily addresses the problem of long-distance feature dependence after point cloud voxelization, utilizing multi-scale features to improve detection accuracy. Figure 1 As shown, the method includes the following:

[0027] The point cloud data of the road scene to be detected is acquired and input into the trained point cloud object detection model for processing to obtain the object detection result.

[0028] The training process of a point cloud object detection model includes:

[0029] S1: Collect point cloud data of the road scene and label the targets to be detected.

[0030] The vehicle captures real-time video of the road scene using cameras and LiDAR, and processes and saves the image data and corresponding point cloud information. Labeling software is used to categorize vehicles, pedestrians, and other targets in the point cloud data, generating label files. Each point cloud data point is paired with a corresponding label and divided into training, testing, and validation sets.

[0031] S2: Divide the point cloud data into a regular voxel grid, and use a voxel encoding module to encode the features of the voxel grid to obtain voxel features.

[0032] The point cloud data of the training set is divided into a regular voxel grid. Assume the point cloud data set is P = {p1, p2, ..., p...} N}, where p i =(x i ,y i ,z i Let p represent a point in three-dimensional space, and N be the total number of points. First, the voxels are divided according to the density of the point cloud: Let point p... i The local density estimate is ρ i , in As an indicator function, when bar point p j In p i The value is 1 when the object is inside a sphere centered at a radius of r, and 0 otherwise. Based on these local density estimates, the voxel grid can be dynamically divided. Points with similar local densities can be grouped into the same voxel grid, or the size of the voxel grid can be adaptively adjusted according to the local density, making the voxel grid larger in sparse regions and smaller in dense regions.

[0033] The voxel features are obtained by feature augmentation of the original voxels using a voxel encoding module. voxel Specifically:

[0034] Dynamic voxel encoding is used, meaning that the pre-divided voxel grid needs to be encoded. Assume there are M voxel grids, each voxel grid... m (m = 1, 2, ..., M) contains a set of points Where n m It is a voxel grid v m The number of points contained therein.

[0035] The mean, covariance, and other statistical properties of points within the voxel grid are calculated as the voxel encoding. Calculating the mean: Let the voxel grid be v... m The encoding vector is e m =(e m1 ,e m2 ,…,e md ), where d is the dimension of the encoding vector. Taking e as an example... m1 For example:

[0036] Where x mk It is a voxel grid v m The x-coordinate of the k-th point.

[0037] Calculate covariance: Let v be the voxel grid. m The coordinate matrix of the midpoint is Then the covariance matrix Σ m It can be calculated as follows: in It is voxel v m The mean vector of the midpoint coordinates.

[0038] e m and Σ m By combining them, we can obtain the encoding vectors of all voxel grids, i.e., voxel features. voxel ={f1,f2,…,f M These encoded vectors can serve as a feature representation of point cloud data for subsequent tasks such as object detection and classification.

[0039] S3: Input the voxel features into the 3D backbone network for processing to obtain multi-scale fused features.

[0040] feat voxel The coords are input into the 3D backbone network to obtain multi-scale fused features. For example... Figure 2 As shown, the 3D backbone network includes four encoding blocks, three decoding blocks, and three feature enhancement modules; both the encoding and decoding blocks consist of multiple ZOMB modules and sparse convolutional layers connected alternately; the output features of the four encoding blocks are {f d1 ,f d2 ,f d3 ,f bottom}, will {f d1 ,f d2 ,f d3 The inputs are fed into three feature enhancement modules for lateral feature enhancement, resulting in the enhanced features {f}. m1 ,f m2 ,f m3};f bottom After upsampling, and with f m3 As the input to the third decoding block; the output features of the three decoding blocks are respectively {f u1 ,f u2 ,f u3}, f u3 After upsampling, and with f m2 As the input to the second decoding block, f u2 After upsampling, and with f m1 As the input to the first decoded block; {f u1 ,f u2 ,f u3 ,f bottom The sums are then normalized to obtain the multi-scale fused feature f. 3d_out .

[0041] The ZOMB module consists of location embedding, Z-Order curve sorting, and a state-space model.

[0042] The Mamba model takes the input sequence Mapping to output sequence For a linear time-invariant system, the continuous state can be described by differential equations as follows:

[0043]

[0044] y(t)=Ch(t) (2)

[0045] In the formula, It is in a hidden state. It's a new hidden state. These are the parameters that the model needs to learn.

[0046] For object detection tasks, continuous differential equation models are not applicable. Therefore, a zero-order hold is used to discretize the continuous state-space model into a discrete state-space model:

[0047]

[0048] In the formula, Δ is the sampling period. These are the parameters after discretization. Therefore, the discrete state-space model can be described as:

[0049]

[0050] In the formula h k-1 This is the current state, h k It is the predicted state at the next moment, y k This is the output result.

[0051] According to equation (6), the state at the next moment is derived from the state at the current moment. Therefore, the state-space model is a causal pattern model, which requires that the input sequence of the model be a causal sequence and not unordered. Therefore, in the ZOMB module, the position of each voxel is first embedded, and then the voxel blocks in the 3D space are arranged according to... Figure 3 The Z-Order curves shown are sorted in a specific way to help the Mamba state-space model learn the spatial features of voxels.

[0052] like Figure 4As shown, the feature enhancement module's processing of the input features includes: performing a sparse convolution with a kernel size of (1,1,1) on the input features to obtain intermediate features; feeding the intermediate features into three branches: the first branch performs a sparse convolution with a kernel size of (3,3,3) on the intermediate features; the second branch compresses the intermediate features and then performs a residual connection with the input features; the third branch does not perform any processing; the outputs of the three branches are weighted and summed to obtain the output features of the feature enhancement module.

[0053] S4: Project the multi-scale fused features onto the BEV space and extract BEV features using a 2D backbone network.

[0054] BEV, or Bird's Eye View, is a perspective or coordinate system (3D) used to describe the perceived world. Multi-scale fused features are projected onto the BEV space, and BEV features are extracted using a 2D backbone network. Preferably, the 2D backbone network is a 2D convolutional neural network.

[0055] S5: Use the head network to process the BEV features to obtain the target detection results.

[0056] Preferably, the head network uses a CenterHead to process the BEV features, generating classification information and 3D BBOX information for the predicted target. This head performs target localization and classification by predicting the heatmap of the object's center point, as well as other attributes related to the center point such as size and orientation. The object center heatmap H(x,y) is modeled using a Gaussian distribution.

[0057]

[0058] In the formula, (x c ,y c ) is the true target center, and σ is the standard deviation of the Gaussian distribution, used to control the width of the distribution.

[0059] Since the center point will not fall precisely at the voxel center every time, it is necessary to predict the offset to improve accuracy. The regression offset for each center point is:

[0060]

[0061] For each predicted center point, the size (w,l,h) and orientation λ of the regressed object can be learned by the regression module and optimized using L1 loss or smoothed L1 loss.

[0062] S6: Calculate the total loss of the model based on the target detection results and image labels, and adjust the model parameters according to the total loss to obtain the trained point cloud target detection model.

[0063] In this invention, the loss function of the point cloud target detection model is a weighted sum of multiple losses:

[0064] L=λ1L heatm +λ2L size +λ3L direction +λ4L offset (11)

[0065] In the formula, L heatmap For heatmap loss, L size For the size loss of the target, L direction L is the object's orientation regression loss. offset The center point offset loss is represented by λ1, λ2, λ3, and λ4, which are the weights of the four types of losses mentioned above.

[0066] The heatmap loss is expressed as:

[0067]

[0068] In the formula, N is the normalization factor. It is the heatmap value of category c at location (x,y) predicted by the model, where Y is... xyc α is the corresponding true heatmap value (usually 0 or 1, indicating whether it is the target center), and α and β are hyperparameters used to adjust the attention given to positive and negative samples.

[0069] The size loss of the target is expressed as:

[0070]

[0071] In the formula, N is the number of detected targets, s i It is the actual size of the i-th target. It is the size of the i-th target predicted by the model, and SmoothL1(x) is defined as...

[0072] The orientation regression loss of an object is expressed as:

[0073]

[0074] In the formula, N is the number of detected targets, and d j d is the true orientation angle of the j-th target. j It is the orientation angle of the j-th target predicted by the model.

[0075] The center point offset loss is expressed as:

[0076]

[0077] In the formula, o jIt is the actual offset of the i-th center point. It is the offset of the predicted center point by the model.

[0078] The optimizer continuously adjusts the model parameters based on the total model loss. Training stops when the total loss function converges or reaches the maximum preset number of iterations, and the current optimal model parameters are saved, resulting in a trained point cloud object detection model. After training, the trained point cloud object detection model is deployed to a real vehicle for real-time 3D road object detection.

[0079] In summary, this invention improves the 3D backbone network based on the current advanced 3D detector CenterNet, utilizes the Mamba model with faster inference speed and stronger ability to capture global information for 3D feature extraction, and adopts the UNet structure to downsample and then upsample the input features to obtain multi-scale features for detecting 3D targets of different sizes. This not only improves the detection accuracy of the model but also speeds up the inference speed.

[0080] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A 3D point cloud target detection method based on a state-space model, characterized in that, include: The point cloud data of the road scene to be detected is acquired and input into a trained point cloud object detection model for processing to obtain the object detection result. The training process of a point cloud object detection model includes: S1: Collect point cloud data of the road scene and label the targets to be detected; S2: Divide the point cloud data into a regular voxel grid, and use a voxel encoding module to encode the features of the voxel grid to obtain voxel features; S3: The voxel features are input into a 3D backbone network for processing to obtain multi-scale fused features; the 3D backbone network includes four encoding blocks, three decoding blocks, and three feature enhancement modules; both the encoding and decoding blocks are composed of multiple ZOMB modules and sparse convolutional layers connected alternately; the output features of the four encoding blocks are {f d1 ,f d2 ,f d3 ,f bottom }, will {f d1 ,f d2 ,f d3 The inputs are fed into three feature enhancement modules for lateral feature enhancement, resulting in the enhanced features {f}. m1 ,f m2 ,f m3 };f bottom After upsampling, and with f m3 As the input to the third decoding block; the output features of the three decoding blocks are respectively {f u1 ,f u2 ,f u3 }, f u3 After upsampling, and with f m2 As the input to the second decoding block, f u2 After upsampling, and with f m1 As the input to the first decoded block; {f u1 ,f u2 ,f u3 ,f bottom The sums are then normalized to obtain the multi-scale fused feature f. 3d_out The ZOMB module consists of location embedding, Z-Order curve sorting, and a state-space model. The feature enhancement module processes the input features as follows: It performs a sparse convolution with a kernel size of (1,1,1) on the input features to obtain intermediate features; the intermediate features are then fed into three branches: the first branch performs a sparse convolution with a kernel size of (3,3,3); the second branch compresses the intermediate features and then performs a residual connection with the input features; the third branch performs no processing; and the outputs of the three branches are weighted and summed to obtain the output features of the feature enhancement module. S4: Project the multi-scale fused features onto the BEV space and extract BEV features using a 2D backbone network; S5: Use the head network to process the BEV features to obtain the target detection results; S6: Calculate the total loss of the model based on the target detection results and point cloud labels; use the optimizer to adjust the model parameters based on the total loss of the model to obtain the trained point cloud target detection model.

2. The 3D point cloud target detection method based on a state-space model according to claim 1, characterized in that, The process of feature encoding of the voxel grid by the voxel encoding module includes: obtaining the coordinates of each point in each voxel grid and calculating the mean and covariance of the voxel grid based on the coordinate vector of each point; merging the mean and covariance to obtain the voxel features.

3. The 3D point cloud target detection method based on a state-space model according to claim 1, characterized in that, The 2D backbone network is a 2D convolutional neural network.

4. The 3D point cloud target detection method based on a state-space model according to claim 1, characterized in that, The head network uses a central detector head to process BEV features, generating classification information and 3D BBOX information for the predicted target.

5. The 3D point cloud target detection method based on a state-space model according to claim 1, characterized in that, The total loss of the model is a weighted sum of heatmap loss, target size loss, object orientation regression loss, and center point offset loss.

Images