3D point cloud target detection method based on state space model

Abstract

The invention belongs to the field of automatic driving environment perception, and particularly relates to a 3D point cloud target detection method based on a state space model. Comprising the steps of collecting road scene point cloud data and labeling a to-be-detected target; dividing the point cloud data into regular voxel grids, and performing feature coding on the voxel grids by adopting a voxel coding module to obtain voxel features; inputting the voxel features into a 3D backbone network for processing to obtain multi-scale fusion features; projecting the multi-scale fusion features to a BEV space, and extracting BEV features by using a 2D backbone network; processing the BEV features by using a head network to obtain a target detection result; calculating the total loss of the model according to the target detection result and the point cloud label, and adjusting model parameters according to the total loss of the model to obtain a trained model; performing target detection by using the trained model; according to the method, the model detection precision is improved, and the reasoning speed is increased.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving environment perception, and specifically relates to a 3D point cloud target detection method based on a state space model. Background Art

[0002] With the rapid development of autonomous driving technology in recent years, 3D point cloud object detection has become a research field that has attracted much attention. Point cloud data, with its rich spatial information and accurate 3D positioning capabilities, has become the key to achieving high-precision environmental perception and object recognition. However, the sparse, uneven and irregular distribution of point cloud data makes efficient and effective 3D object detection a very challenging task.

[0003] In order to solve these problems, researchers have proposed several mainstream methods: The voxelization-based method discretizes point cloud data into regular three-dimensional grids so that it can be processed by traditional convolution operations. Its advantage is that the processing speed is fast and can meet the real-time requirements, but information loss is inevitable in the voxelization process. The point-based method achieves efficient processing of point clouds through point-by-point feature extraction and global feature aggregation. It avoids the information loss caused by voxelization and improves the detection accuracy and efficiency of the model to a certain extent. However, due to its direct processing of the features of each point, it may face the challenges of high computational complexity and lack of real-time performance.

[0004] In this context, the state space model is introduced into 3D point cloud target detection as an emerging technology. The state space model can not only effectively process time series information by modeling the dynamic characteristics of the system, but also play an important role in spatial data processing. The state space model combined with the U-Net architecture can give full play to its advantages in feature extraction and multi-scale information fusion. The U-Net structure, in its encoder-decoder form, can extract deep features in the downsampling process and restore spatial resolution in the upsampling process, while retaining low-level features through jump connections. Compared with traditional 3D convolutional neural networks, the U-Net architecture based on the state space model has significant advantages in computational efficiency and feature extraction capabilities. The use of sparse convolution can effectively reduce the amount of calculation and adapt to the needs of large-scale data processing. At the same time, this method can better maintain the integrity of spatial information in the feature extraction process and improve the accuracy of target detection. Compared with the PointNet series of methods, the state space model shows better scalability and efficiency in processing large-scale data, and has stronger capabilities in the integration of multi-scale features, which can improve detection performance without increasing too much computational overhead. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a 3D point cloud target detection method based on a state space model, the method comprising: obtaining point cloud data of a road scene to be detected and inputting the point cloud data into a trained point cloud target detection model for processing to obtain a target detection result;

[0006] The training process of the point cloud object detection model includes:

[0007] S1: Collect road scene point cloud data and mark the targets to be detected;

[0008] S2: Divide the point cloud data into regular voxel grids, and use the voxel encoding module to encode the voxel grids to obtain voxel features;

[0009] S3: Input the voxel features into the 3D backbone network for processing to obtain multi-scale fusion features;

[0010] S4: Project the multi-scale fusion features into the BEV space and use the 2D backbone network to extract BEV features;

[0011] S5: Use the head network to process the BEV features and obtain the target detection results;

[0012] S6: Calculate the total model loss based on the target detection results and point cloud labels; use the optimizer to adjust the model parameters according to the total model loss to obtain a trained point cloud target detection model.

[0013] Preferably, the process of feature encoding of the voxel grid by the voxel encoding module includes: obtaining the coordinates of each point in each voxel grid and calculating the mean and covariance of the voxel grid according to the coordinate vector of each point; combining the mean and covariance to obtain the voxel feature.

[0014] Preferably, the 3D backbone network includes four encoding blocks, three decoding blocks and three feature enhancement modules; the encoding blocks and the decoding blocks are composed of multiple ZOMB modules and sparse convolutional layers alternately connected; the output features of the four encoding blocks are {f d1 ,f d2 ,f d3 ,f bottom}, {f d1 ,f d2 ,f d3} are respectively input into three feature enhancement modules for lateral feature enhancement, and the enhanced features {f m1 ,f m2 ,f m3}; f bottom After upsampling, f m3 As the input of the third decoding block; the output features of the three decoding blocks are {f u1 ,fu2 ,f u3}, f u3 After upsampling, f m2 As the input of the second decoding block, f u2 After upsampling, f m1 As the input of the first decoding block; {f u1 ,f u2 ,f u3 ,f bottom} are added and normalized to obtain the multi-scale fusion feature f 3d_out .

[0015] Furthermore, the ZOMB module consists of position embedding, Z-Order curve sorting and state space model.

[0016] Furthermore, the feature enhancement module processes the input features as follows: performing a sparse convolution with a convolution kernel size of (1,1,1) on the input features to obtain intermediate features; sending the intermediate features to three branches, the first branch performs a sparse convolution with a convolution kernel size of (3,3,3) on the intermediate features; the second branch compresses the intermediate features and then performs a residual connection with the input features; the third branch does not perform any processing; the outputs of the three branches are weighted summed to obtain the output features of the feature enhancement module.

[0017] Preferably, the 2D backbone network is a 2D convolutional neural network.

[0018] Preferably, the head network uses a central detection head to process the BEV features to generate classification information and 3D BBOX information of the predicted target.

[0019] Preferably, the total model loss is a weighted sum of the heat map loss, the target size loss, the object orientation regression loss, and the center point offset loss.

[0020] The beneficial effects of the present invention are as follows: the present invention provides a 3D point cloud target detection method based on a state-space model, the 3D backbone network uses a ZOMB module composed of a state-space model Mamba for feature learning, and uses a Mamba model with faster reasoning speed and stronger ability to capture global information to extract 3D features, and adopts a UNet network structure as a whole to obtain multi-scale features, and proposes a lateral feature enhancement module to enhance the intermediate features in the downsampling process, and finally obtains global 3D features containing multi-scale information for input to the next stage, thereby improving the model reasoning speed and detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart for implementing the point cloud target detection model in the present invention;

[0022] Figure 2 It is a schematic diagram of the 3D backbone network structure in the present invention;

[0023] Figure 3 It is a schematic diagram of the Z-Order curve traversal method of the present invention;

[0024] Figure 4 It is a schematic diagram of the structure of the feature enhancement module of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] An autonomous driving system can usually be divided into three parts: a perception layer, a decision layer, and an execution layer. Sensors are used to obtain environmental information around the vehicle, and richer and more accurate features are extracted. The features are sent to the decision layer for judgment, and corresponding control strategies are formulated. The features are sent to the execution layer to perform corresponding motion control on the vehicle. Among them, the perception layer is particularly important. Whether the environmental information around the vehicle is accurately perceived determines subsequent decision-making and motion control. However, the roads on which vehicles travel are generally more complex scenes, such as vehicles, pedestrians, traffic signs, and so on. Therefore, the present invention provides a 3D point cloud target detection method based on a state-space model, which can mainly solve the problem of long-distance feature dependency after point cloud voxelization, and use multi-scale features to improve detection accuracy. Figure 1 As shown, the method includes the following contents:

[0027] The point cloud data of the road scene to be detected is obtained and input into the trained point cloud target detection model for processing to obtain the target detection result.

[0028] The training process of the point cloud object detection model includes:

[0029] S1: Collect road scene point cloud data and mark the targets to be detected.

[0030] The driving vehicle collects road scene videos in real time through cameras and lidar, and processes and saves the image data and the corresponding point cloud information. The vehicles and pedestrians waiting for detection in the point cloud data are labeled by the annotation software to generate a label file. The point cloud data corresponds to the point cloud labels one by one and is divided into training set, test set, and validation set.

[0031] S2: Divide the point cloud data into regular voxel grids, and use the voxel encoding module to encode the voxel grid features to obtain voxel features.

[0032] The point cloud data of the training set is divided into regular voxel grids. Assume that the point cloud data set is P = {p1, p2, ..., p N}, where p i =(x i ,y i ,z i ) represents a point in three-dimensional space, and N is the total number of points. First, the voxels are divided according to the density of the point cloud: let point p i The local density estimate is ρ i , in is the indicator function, when the point p j In p i The value is 1 when the point is in a sphere with a radius of r and the center is centered, and 0 otherwise. Based on these local density estimates, the voxel grid can be dynamically divided. Points with similar local density can be divided into the same voxel grid, or the size of the voxel grid can be adaptively adjusted according to the local density, so that the voxel grid is larger in sparse areas and smaller in dense areas.

[0033] Use the voxel encoding module to expand the original voxel features to obtain the voxel features feat voxel . Specifically:

[0034] Use dynamic voxel coding, that is, the divided voxel grid needs to be encoded. Assume that M voxel grids are divided, each voxel grid v m (m=1,2,…,M) contains a set of points where n m is the voxel grid v m The number of points contained in .

[0035] Calculate the mean, covariance and other statistics of the points in the voxel grid as the encoding of the voxel. Calculate the mean: Let the voxel grid v m The encoding vector is e m =(e m1 ,e m2 ,…,e md ), d is the dimension of the encoding vector. m1 For example:

[0036] where x mk is the voxel grid v m The x-coordinate of the kth point in .

[0037] Calculate the covariance: Let the voxel grid v m The coordinate matrix of the midpoint is Then the covariance matrix Σ m It can be calculated as: in is the voxel v m Mean vector of midpoint coordinates.

[0038] E m and Σ m By combining, we can get the encoding vector of all voxel grids, namely the voxel feature feat voxel ={f1,f2,…,f M These encoded vectors can be used as a feature representation of point cloud data for subsequent target detection, classification and other tasks.

[0039] S3: Input the voxel features into the 3D backbone network for processing to obtain multi-scale fusion features.

[0040] Will feature voxel And coords are input into the 3D backbone network to obtain multi-scale fusion features. Figure 2 As shown in the figure, the 3D backbone network includes four encoding blocks, three decoding blocks and three feature enhancement modules; both the encoding blocks and the decoding blocks are composed of multiple ZOMB modules and sparse convolutional layers connected alternately; the output features of the four encoding blocks are {f d1 ,f d2 ,f d3 ,f bottom}, {f d1 ,f d2 ,f d3} are respectively input into three feature enhancement modules for lateral feature enhancement, and the enhanced features {f m1 ,f m2 ,f m3}; f bottom After upsampling, f m3 As the input of the third decoding block; the output features of the three decoding blocks are {f u1 ,f u2 ,f u3}, f u3 After upsampling, f m2 As the input of the second decoding block, f u2 After upsampling, f m1 As the input of the first decoding block; {f u1 ,f u2 ,f u3 ,f bottom} are added and normalized to obtain the multi-scale fusion feature f 3d_out .

[0041] The ZOMB module consists of position embedding, Z-Order curve sorting and state space model.

[0042] The Mamba model takes as input the sequence Mapping to output sequence The linear time-invariant system can be described by differential equations in the continuous state:

[0043]

[0044] y(t)=Ch(t) (2)

[0045] In the formula, is the hidden state, is the new hidden state, are the parameters that the model needs to learn.

[0046] For the target detection task, the continuous differential equation model is not applicable, so the continuous state space model is discretized into a discrete state space model through a zero-order holder:

[0047]

[0048] Where Δ is the sampling period, is the discretized parameter. Therefore, the discrete state space model can be described as:

[0049]

[0050] Where h k-1 is the current state, h k is the predicted state at the next moment, y k is the output result.

[0051] According to formula (6), it can be found that the state at the next moment is derived from the state at the current moment. Therefore, the state space model is a causal model, which requires that the sequence of model input is a causal sequence and cannot be disordered. Therefore, in the ZOMB module, each voxel is first embedded in position, and then the voxel blocks in the 3D space are arranged according to Figure 3 The Z-Order curve traversal method shown is sorted, and this specific arrangement helps the state-space model Mamba learn the spatial characteristics of voxels.

[0052] like Figure 4As shown in the figure, the feature enhancement module processes the input features as follows: the input features are sparsely convolved with a convolution kernel size of (1,1,1) to obtain intermediate features; the intermediate features are sent to three branches, the first branch performs sparse convolution processing with a convolution kernel size of (3,3,3) on the intermediate features; the second branch compresses the intermediate features and then performs residual connection with the input features; the third branch does not perform any processing; the outputs of the three branches are weighted summed to obtain the output features of the feature enhancement module.

[0053] S4: Project the multi-scale fusion features into the BEV space and use the 2D backbone network to extract BEV features.

[0054] BEV, or Bird's Eye View, is a perspective or coordinate system (3D) used to describe the perceived world. Multi-scale fusion features are projected into the BEV space, and a 2D backbone network is used to extract BEV features. Preferably, the 2D backbone network is a 2D convolutional neural network.

[0055] S5: Use the head network to process the BEV features to obtain the target detection results.

[0056] Preferably, the head network uses a center detection head CenterHead to process the BEV features and generate classification information and 3D BBOX information of the predicted target. The detection head locates and classifies the target by predicting the heat map of the center point of the object and other attributes such as size and direction related to the center point. The object center heat map H(x,y) is modeled using a Gaussian distribution:

[0057]

[0058] In the formula, (x c ,y c ) is the true target center, and σ is the standard deviation of the Gaussian distribution, which is used to control the width of the distribution.

[0059] Since the center point will not fall exactly at the center of the voxel every time, it is necessary to predict the offset to improve the accuracy. The regression offset for each center point is:

[0060]

[0061] For each predicted center point, the size (w, l, h) and orientation λ of the regressed object can be learned by the regression module and optimized using L1 loss or smoothed L1 loss.

[0062] S6: Calculate the total model loss based on the target detection results and image labels and adjust the model parameters based on the total model loss to obtain a trained point cloud target detection model.

[0063] The loss function of the point cloud target detection model in this invention is the weighted sum of multiple losses:

[0064] L=λ1L heatm +λ2L size +λ3L direction +λ4L offset (11)

[0065] Where, L heatmap is the heat map loss, L size is the target size loss, L direction is the object orientation regression loss, L offset is the center point offset loss, λ1,λ2,λ3,λ4 are the weights of the above four losses respectively.

[0066] Among them, the heat map loss is expressed as:

[0067]

[0068] Where N is the normalization factor, is the heat map value of category c at position (x, y) predicted by the model, Y xyc is the corresponding true heat map value (usually 0 or 1, indicating whether it is the target center), α and β are hyperparameters used to adjust the attention to positive and negative samples.

[0069] The target size loss is expressed as:

[0070]

[0071] Where N is the number of detected targets, s i is the true size of the i-th target, is the size of the i-th target predicted by the model, and SmoothL1(x) is defined as

[0072] The object orientation regression loss is expressed as:

[0073]

[0074] Where N is the number of detected targets, d j is the actual heading angle of the jth target, d j is the orientation angle of the jth target predicted by the model.

[0075] The center point shift loss is expressed as:

[0076]

[0077] In the formula, o jis the true offset of the center point, is the offset of the nth center point predicted by the model.

[0078] Use the optimizer to continuously adjust the model parameters according to the total loss of the model. When the total loss function converges or reaches the maximum preset number of iterations, stop training, save the current optimal model parameters, and obtain a trained point cloud target detection model. After the model training is completed, the trained point cloud target detection model is deployed to the actual vehicle for real-time road 3D target detection.

[0079] In summary, the present invention improves the 3D backbone network based on the current advanced 3D detector CenterNet, uses the Mamba model with faster reasoning speed and stronger ability to capture global information to extract 3D features, and adopts the UNet structure to downsample the input features first and then upsample them to restore them, and obtain multi-scale features for detecting 3D objects of different sizes, which not only improves the model detection accuracy but also speeds up the reasoning speed.

[0080] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D point cloud target detection method based on a state space model, characterized in that: include: Obtain the road scene point cloud data to be detected and input it into the trained point cloud target detection model for processing to obtain the target detection result; The training process of the point cloud object detection model includes: S1: Collect road scene point cloud data and mark the targets to be detected; S2: Divide the point cloud data into regular voxel grids, and use the voxel encoding module to encode the voxel grids to obtain voxel features; S3: Input the voxel features into the 3D backbone network for processing to obtain multi-scale fusion features; S4: Project the multi-scale fusion features into the BEV space and use the 2D backbone network to extract BEV features; S5: Use the head network to process the BEV features and obtain the target detection results; S6: Calculate the total model loss based on the target detection results and point cloud labels; use the optimizer to adjust the model parameters according to the total model loss to obtain a trained point cloud target detection model.

2. The 3D point cloud target detection method based on the state space model according to claim 1, characterized in that: The process of feature encoding of the voxel grid by the voxel encoding module includes: obtaining the coordinates of each point in each voxel grid and calculating the mean and covariance of the voxel grid according to the coordinate vector of each point; merging the mean and covariance to obtain the voxel feature.

3. The 3D point cloud target detection method based on the state space model according to claim 1, characterized in that: The 3D backbone network includes four encoding blocks, three decoding blocks and three feature enhancement modules; the encoding blocks and decoding blocks are composed of multiple ZOMB modules and sparse convolutional layers connected alternately; the output features of the four encoding blocks are {f d1 ,f d2 ,f d3 ,f bottom }, replace {f d1 ,f d2 ,f d3 } are respectively input into three feature enhancement modules for lateral feature enhancement, and the enhanced features {f m1 ,f m2 ,f m3 }; f bottom After upsampling, f m3 As the input of the third decoding block; the output features of the three decoding blocks are {f u1 ,f u2 ,f u3 }, f u3 After upsampling, f m2 As the input of the second decoding block, f u2 After upsampling, f m1 As the input of the first decoding block; {f u1 ,f u2 ,f u3 ,f bottom } are added and normalized to obtain the multi-scale fusion feature f 3d_out .

4. The 3D point cloud target detection method based on the state space model according to claim 3 is characterized in that: The ZOMB module consists of position embedding, Z-Order curve sorting and state space model.

5. The 3D point cloud target detection method based on the state space model according to claim 3, characterized in that: The feature enhancement module processes the input features by: performing a sparse convolution with a convolution kernel size of (1,1,1) on the input features to obtain intermediate features; sending the intermediate features to three branches, wherein the first branch performs a sparse convolution with a convolution kernel size of (3,3,3) on the intermediate features; the second branch compresses the intermediate features and then performs a residual connection with the input features; the third branch does not perform any processing; and the outputs of the three branches are weighted summed to obtain the output features of the feature enhancement module.

6. The 3D point cloud target detection method based on the state space model according to claim 1, characterized in that: The 2D backbone network is a 2D convolutional neural network.

7. The 3D point cloud target detection method based on the state space model according to claim 1, characterized in that: The head network uses a central detection head to process the BEV features and generate classification information and 3D BBOX information of the predicted target.

8. The 3D point cloud target detection method based on the state space model according to claim 1, characterized in that: The total loss of the model is the weighted sum of the heat map loss, the target size loss, the object orientation regression loss and the center point offset loss.

Images