Point cloud target detection method based on recursive gated convolution and sparse Transform

Abstract

The invention discloses a point cloud target detection method based on recursive gated convolution and sparse Transform. The method comprises the following steps: firstly, collecting point cloud data in a driving scene and preprocessing the point cloud data; secondly, voxelization processing is carried out to obtain a 3D sparse voxel feature grid, downsampling is carried out by using a 3D feature extraction network, and multi-scale voxel features are extracted; then, the voxel features are highly compressed, features in a 2D space are extracted, and a first-stage detection head is input to generate a first-stage target detection frame; finally, dividing the target detection frame of the first stage into a plurality of uniform grid points by adopting grid point feature aggregation; voxel search is adopted to obtain feature representation of the candidate box; and predicting to obtain a final target detection frame by using a two-stage detection head through the aggregated grid point features and combining the target detection frame of the first stage. According to the method, high-order spatial information interaction is constructed, and the method has good performance on long-distance target detection and small target detection.

Description

Technical Field

[0001] The present invention relates to point cloud three-dimensional target detection technology, specifically to a point cloud 3D target detection method based on recursive gated convolution and sparse Transformer. Background Art

[0002] Point cloud data consists of a large number of points, each of which contains its three-dimensional coordinates and reflection intensity in space. Point cloud three-dimensional object detection technology aims to use point cloud data to identify and locate objects, such as vehicles, pedestrians, buildings, etc., which is an important task in the fields of autonomous driving, robot navigation, augmented reality (AR), etc. Compared with traditional two-dimensional image object detection, three-dimensional object detection can provide richer spatial information and help improve the accuracy and robustness of perception. Therefore, point cloud object detection technology has become an important research direction in the field of computer vision and three-dimensional perception.

[0003] At present, point cloud target detection methods are mainly divided into two methods: point-based and voxel-based. The point-based method directly inputs the point cloud data into the target detection model without additional voxelization processing. It uses MLP (multi-layer perceptron) to extract features from the point cloud data and then outputs the target detection box. In the voxel-based method, the point cloud space is first divided into a uniform voxel network, and voxel features are extracted using pooling operations and sparse convolutions. The voxel features are then compressed into 2D space to extract features again and output the target detection box. The point-based method is prone to missing local features of the point cloud, while the voxel-based method faces the problem of limited receptive field. And due to the sparsity of the point cloud, there are great challenges in detecting distant targets. Summary of the invention

[0004] The present invention aims to overcome the problem of low detection accuracy of long-distance targets and small targets in environmental perception of LiDAR, and provides a point cloud target detection method based on recursive gated convolution and sparse Transformer. A high-order point cloud information interaction is constructed through the recursive gated convolution module, which effectively solves the problem of limited receptive field of traditional detection models and improves the detection capability of small targets. Through the sparse Transformer module, the association between long-distance point clouds and short-distance point clouds and local large-range point clouds is established, which improves the detection capability of long-distance targets and small targets. In terms of improving the detection accuracy of vehicles and pedestrians, the method of the present invention shows significant advantages.

[0005] The point cloud 3D object detection method based on recursive gated convolution and sparse Transformer specifically includes steps S1 to S6:

[0006] S1: Collect point cloud data in real driving scenarios through LiDAR and perform data preprocessing on the collected point cloud data.

[0007] S2: voxelize the preprocessed point cloud data and divide the preprocessed point cloud data into a 3D sparse voxel feature grid with a fixed dimension.

[0008] S3: Use a 3D feature extraction network consisting of a 3D sparse convolution module and a sparse Transformer module to downsample the voxelized 3D sparse voxel feature grid and extract multi-scale voxel features.

[0009] S4: The downsampled voxel features are highly compressed to obtain a bird's-eye view feature map in 2D space. A 2D feature extraction network consisting of 2D convolution and recursive gated convolution is used to further extract features in 2D space.

[0010] S5: The features in the 2D space extracted in S4 are input into the first-stage detection head to generate the first-stage target detection box.

[0011] S6: Using the grid point feature aggregation method, the first-stage target detection box is divided into several uniform grid points. The voxel search method is used to aggregate the multi-scale voxel features to the grid points, and further aggregate all the grid point features in the candidate box to obtain the feature representation of the candidate box. Using the two-stage detection head, the final target detection box is predicted by combining the aggregated grid point features with the first-stage target detection box.

[0012] In step S1 of the present invention, the specific steps of pre-processing include S1-1 to S1-4:

[0013] S1-1: Filter out the distant point clouds that are beyond the valid range of the dataset and only retain the valid point clouds within the range.

[0014] S1-2: Perform random flipping, random rotation, and random scaling operations on the point cloud filtered by S1-1 for data enhancement.

[0015] S1-3: Resample the target. After data enhancement, for the current frame data that needs to be trained, extract the point cloud in the real bounding box of other frames in the training set and place it in the empty position where there is no target in the current frame, increase the number of targets in the current frame, and dynamically generate the target data of the current frame, so as to achieve the purpose of training sample data enhancement.

[0016] S1-4: Randomize the point cloud order of the point cloud data obtained in S1-3 to reduce the model's dependence on the input point cloud order.

[0017] In step S2 of the present invention, the voxelization method is: for each point P in the point cloud i ={(x, y, z, r)}, P i ∈R 3, where x, y, z represent the coordinates of the point in 3D space, and r represents the radar reflection intensity. First, the preprocessed point cloud is grouped into voxels according to the coordinates, and the average value of the point cloud data in the voxel group is calculated as the feature of the voxel to obtain the feature v of voxel j j .

[0018] After the point cloud is voxelized, a j The 3D sparse voxel feature features are composed of dimensions (L, W, H, C), where L and W represent the dimensions of the 3D sparse voxel feature grid along the x and y directions, H represents the dimension of the 3D sparse voxel feature grid along the z direction, and C represents the feature dimension of the 3D sparse voxel. In this case, the feature dimension is 4, namely (x, y, z, r).

[0019] In step S3 of the present invention, the specific implementation steps of the 3D feature extraction network composed of the 3D sparse convolution module and the sparse Transformer module are S3-1 to S3-4:

[0020] S3-1: The voxelized 3D sparse voxel features (L, W, H, C) are downsampled in the x, y, and z directions through a sub-manifold sparse convolution block to increase the feature dimension to 16, and then the voxel features are extracted again through 4 sub-manifold sparse convolution modules to obtain the first layer of voxel features F 1 =(L, W, H, 16).

[0021] S3-2: The output F of S3-1 1 , after a sparse convolution module, it is downsampled twice in the x, y, and z directions, and the feature dimension is increased to 32, and then it is extracted through 4 sub-flow sparse convolution modules again to obtain the second layer of voxel features F 2 =(L / 2, W / 2, H / 2, 32).

[0022] S3-3: According to F 1 The midvoxel and F 2 The downsampling correspondence of the voxels in F 1 The centroid coordinates of the corresponding voxels in the averaging are used to calculate F 2 The centroid coordinates of the voxels in the image are input into the sparse Transformer module to extract the voxel features and the centroid coordinates of the voxel. Then, a sparse convolution module is used to perform two-fold downsampling in the x, y, and z directions, and the feature dimension is increased to 64. The voxel features are extracted again through four sub-flow sparse convolution modules to obtain the third-layer voxel features F 3 =(L / 4, W / 4, H / 4, 64).

[0023] S3-4: According to F 2 The midvoxel and F3 The downsampling correspondence of the voxels in F 2 The centroid coordinates of the corresponding voxels in the averaging are used to calculate F 3 The centroid coordinates of the voxels in the image are input into the sparse Transformer module to extract the voxel features and the centroid coordinates of the voxels. Then, a sparse convolution module is used to perform two-fold downsampling in the x, y, and z directions, and the feature dimension is increased to 128. The voxel features are extracted again through four sub-flow sparse convolution modules and an additional sparse convolution module to obtain the fourth layer of voxel features F 4 =(L / 8, W / 8, H / 8, 128).

[0024] In S3 of the present invention, the sparse Transformer module is composed of a sparse Transformer module in a Cartesian coordinate system and a sparse Transformer module in a polar coordinate system, and the specific implementation steps include S3-5 to S3-8:

[0025] S3-5: For voxel features of dimension (L, W, H, C), they are divided into two parts along the feature dimension, both of which have dimensions (L, W, H, C / 2), and are input into the sparse Transformer module in the Cartesian coordinate system and the sparse Transformer module in the polar coordinate system respectively.

[0026] S3-6: In the sparse Transformer module in the Cartesian coordinate system, the input voxel features are first divided into windows in the Cartesian coordinate system, the relative positions of the voxel features in the same window are encoded, and then multi-head attention calculations are performed.

[0027] S3-7: In the sparse Transformer module in the polar coordinate system, the input voxel features are first divided into windows in the polar coordinate system, the relative positions of the voxel features in the same polar coordinate window are encoded, and then multi-head attention calculations are performed.

[0028] S3-8: After the features output by the sparse Transformer module in the Cartesian coordinate system and the sparse Transformer module in the polar coordinate system are concatenated, they are input into the MLP for feature fusion. First, the first layer of MLP is entered for feature fusion, and then the output of the first layer of MLP is input into the second layer of MLP for fusion again. The output results of the two layers of MLP are merged using residual connection to obtain the final output features.

[0029] In S4 of the present invention, the high compression is the output feature F of the 3D feature extraction network 4=(L / 8, W / 8, H / 8, 128), passes through a sparse convolution module with a convolution kernel size of (1, 1, 3), extracts features in the height direction, and performs two-fold downsampling in the z direction. Then, the obtained voxel feature space is highly compressed, and the features are spliced ​​along the z direction, finally obtaining BEVmap = (L / 8, W / 8, 256).

[0030] In S4 of the present invention, the 2D feature extraction network composed of 2D convolution and recursive gated convolution, the specific implementation steps include S4-1 to S4-3:

[0031] S4-1: Input the 2D feature map BEVmap into a convolution module with a convolution kernel size of (3,3) for one-fold downsampling, and convert the feature dimension to 128. Then, it passes through 5 convolution modules with a convolution kernel size of (3,3) for 2D feature extraction and inputs it into the recursive gated convolution module to obtain the 2D feature map BEVmap. 1 .

[0032] S4-2: 2D feature map BEVmap 1 Input a convolution module with a convolution kernel size of (3,3) and perform 2x downsampling. At the same time, the feature dimension is converted to 256. Then, it passes through 5 convolution modules with a convolution kernel size of (3,3) for 2D feature extraction and inputs the recursive gated convolution module to obtain the 2D feature map BEVmap. 2 .

[0033] S4-3: 2D feature map BEVmap 1 、BEVmap 2 Input deconvolution module, upsample by 1x and 2x in x and y directions respectively, BEVmap 1 The feature dimension of BEVmap is converted from 128 to 256. 2 The feature dimension of remains unchanged, and the two are fused at multiple scales along the feature dimension to obtain the final 2D feature map BEVmap final , whose feature dimension is 512.

[0034] In S4-2 of the present invention, the recursive gated convolution module specifically includes steps S4-2-1 to S4-2-3:

[0035] S4-2-1: For an input feature map of dimension (L, W, C), first perform linear projection to double its feature dimension to obtain a feature map of dimension (L, W, 2C), and then divide the feature into 4 parts: p 1 ,q 1 ,q 2 ,q 3The feature dimensions of each part are: C / 4, C / 4, C / 2, C.

[0036] S4-2-2: To q 1 ,q 2 ,q 3 Perform depth-wise convolution to obtain q′ 1 , q′ 2 , q′ 3 , p 1 ,q 1 ′,q′ 2 , q′ 3 The calculation is done according to the following formula:

[0037] p k+1 =q′ k *mlp(p k ),k=1,2,3

[0038] S4-2-3: Put p 4 Input MLP for feature extraction and obtain the final output feature map.

[0039] In S6 of the present invention, a two-stage detection frame refinement is performed, and the specific implementation steps include S6-1 to S6-5:

[0040] S6-1: Divide the target detection box generated in the first stage into uniform grids and obtain the coordinates of the center point of each grid.

[0041] S6-2: Using the voxel search method, the third and fourth layers of the 3D feature extraction network are respectively 3 , F 4 In the above example, voxels with a Manhattan distance between 2 and 4 from the center of the grid are searched, and the Manhattan distance between two voxels is determined by the index of the voxel.

[0042] S6-3: For the voxels found in S6-2, the voxel features and their centroid coordinates are respectively input into MLP for feature extraction, and their respective outputs are spliced ​​and then input into MLP (multi-layer perceptron) to fuse multi-scale features to obtain all voxel features within the set distance of the grid point. The maximum pooling method is used to aggregate the features of the surrounding voxels to the grid point.

[0043] S6-4: For each target detection box, multiple grid points with aggregated multi-scale 3D voxel features can be obtained. All grid point features are concatenated to obtain a one-dimensional feature vector, which is passed into the MLP for feature fusion to obtain the feature representation of the candidate box.

[0044] S6-5: Input the obtained feature representation of the candidate frame into the classification head and the regression head. The classification head obtains the classification prediction of the final detection frame. The regression head obtains the coordinate deviation, size deviation, and rotation angle deviation based on the first-stage detection frame. The information of the first-stage target detection frame is added with the corresponding deviation to obtain the corrected target detection frame, where the target detection frame information includes: the coordinates of the center point of the target detection frame on the x, y, and z axes, the length, width, and height of the detection frame, the rotation angle of the detection frame, and the category of the detection frame; then non-maximum suppression is performed on the corrected detection frame to obtain the final target detection frame.

[0045] The present invention implements 3D target detection through an integrated processing flow. First, a laser radar is used to scan the target and obtain point cloud data. Then, the point cloud data is preprocessed to ensure the accuracy and high availability of the data. Subsequently, the point cloud voxels are feature extracted through a 3D sparse convolutional network, and the sparse Transformer module therein is used to enhance the feature expression of the point cloud during the feature extraction process. After highly compressing the features of the 3D space, a 2D feature map from the BEV perspective is obtained, and features are extracted using a 2D feature extraction network. The recursive gated convolutional network therein improves the receptive field of the model while establishing high-order spatial interactions. After that, a first-stage 3D detection frame is generated through the target detection head, and the first-stage detection frame is divided into multiple grids. The voxel features in the 3D feature extraction network are aggregated to grid points, and all grid points are fused through MLP. The fused features are used to further refine the first-stage detection frame to obtain the final target detection frame.

[0046] Beneficial effects of the present invention:

[0047] The point cloud 3D target detection method proposed in the present invention establishes the association between long-distance point clouds, short-distance point clouds, and local large-range point clouds through the sparse Transformer module, and uses recursive gated convolution in the 2D feature space to construct high-order spatial information interaction and expand the receptive field of the model in the 2D space, so that the method has good performance in long-distance target detection and small target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a model framework diagram of the present invention;

[0049] Figure 2 A visualization image of the original input physical centroid in the 3D feature extraction network of step 2 of the present invention;

[0050] Figure 3 It is a visualization image of the centroid of the body after 8 times downsampling in the 3D feature extraction network of step 2 of the present invention;

[0051] Figure 4This is a visual comparison diagram of the detection results of the model of the present invention;

[0052] Figure 5 This is a visual comparison diagram of the detection results of the model of the present invention;

[0053] Figure 6 This is a visual comparison diagram of the detection results of the model of the present invention. DETAILED DESCRIPTION

[0054] like Figure 1 As shown, the point cloud target detection method based on recursive gated convolution and sparse Transformer proposed in the present invention specifically includes the following steps:

[0055] S1: Read the point cloud data in the KITTI dataset and preprocess each frame of point cloud data. Specifically, the preprocessing includes steps S1-1 to S1-4:

[0056] S1-1: Filter out distant points that are out of range and retain only valid points within the range. For each point P in the original point cloud i ={(x, y, z, r)}, P i ∈R 3 , where x, y, z represent the coordinates of the point in 3D space, and r represents the radar reflection intensity. The valid range of the KITTI dataset is specified as follows: 0m≤x≤70.2m, -40m≤y≤40m, -3m≤z≤1m.

[0057] S1-2: Perform data enhancement on the point cloud filtered by S1-1 by performing random flipping, random rotation, and random scaling operations on it.

[0058] S1-3: After resampling the target and enhancing the data, for the current frame data that needs to be trained, extract the point cloud within the real bounding box of other frames in the training set and place it in the empty position where there is no target in the current frame, increase the number of targets in the current frame, and dynamically generate target data for the current frame, so as to achieve the purpose of training sample data enhancement.

[0059] S1-4: Randomize the point cloud order of the point cloud data obtained in S1-3 to reduce the model's dependence on the input point cloud order.

[0060] S2: voxelize the preprocessed point cloud data and divide the collected sparse point cloud data into 3D voxel grids of fixed size. First, the preprocessed point cloud is grouped by voxel coordinates, and the voxel size is (0.05m, 0.05m, 0.1m). The average value of the point cloud data in the voxel group is calculated as the feature of the voxel. The calculation formula is as follows:

[0061] v j =avg(Pi ), P i ∈V j

[0062] v j represents the feature of voxel j, V j represents the jth voxel, P i Represents a point in point cloud space.

[0063] After the point cloud is voxelized, a j The 3D sparse voxel features are composed of dimensions (L, W, H, C), where L and W represent the dimensions of the 3D sparse voxel feature grid along the x and y directions, H represents the dimension of the 3D sparse voxel feature grid along the z direction, and C represents the feature dimension of the 3D sparse voxel feature grid. In this case, the feature dimension is 4, namely (x, y, z, r).

[0064] S3: Use a 3D feature extraction network composed of a 3D sparse convolution module and a sparse Transformer module to downsample the voxelized 3D sparse voxel feature grid. The original input volume centroid of the 3D feature extraction network is as follows: Figure 2 As shown in the figure, the physical centroid after 8 times sparse convolution downsampling is as follows Figure 3 The specific implementation steps of downsampling to extract multi-scale voxel features include S3-1 to S3-5:

[0065] S3-1: The voxelized 3D sparse voxel features (L, W, H, C) are downsampled in the x, y, and z directions through a sub-manifold sparse convolution block to increase the feature dimension to 16, and then the voxel features are extracted again through 4 sub-manifold sparse convolution modules to obtain the first layer of voxel features F 1 =(L, W, H, 16).

[0066] S3-2: The output F of S3-1 1 , after a sparse convolution module, it is downsampled twice in the x, y, and z directions, and the feature dimension is increased to 32, and then it is extracted through 4 sub-flow sparse convolution modules again to obtain the second layer of voxel features F 2 =(L / 2, W / 2, H / 2, 32).

[0067] S3-3: According to F 1 The midvoxel and F 2 The downsampling correspondence of the voxels in F 1 The centroid coordinates of the corresponding voxels in the averaging are used to calculate F 2The centroid coordinates of the voxels in the image are input into the sparse Transformer module for large-scale information interaction and feature extraction. Then, a sparse convolution module is used to perform two-fold downsampling in the x, y, and z directions, and the feature dimension is increased to 64. The voxel features are extracted again through four sub-flow sparse convolution modules to obtain the third-layer voxel features F 3 =(L / 4, W / 4, H / 4, 64).

[0068] S3-4: According to F 2 The midvoxel and F 3 The downsampling correspondence of the voxels in F 2 The centroid coordinates of the corresponding voxels in the averaging are used to calculate F 3 The centroid coordinates of the voxels in the image are input into the sparse Transformer module for large-scale information interaction and feature extraction. Then, a sparse convolution module is used to perform two-fold downsampling in the x, y, and z directions, and the feature dimension is increased to 128. The voxel features are extracted again through four sub-flow sparse convolution modules and an additional sparse convolution module to obtain the fourth-layer voxel feature F. 4 =(L / 8, W / 8, H / 8, 128).

[0069] The sparse Transformer module consists of a sparse Transformer module in a Cartesian coordinate system and a sparse Transformer module in a polar coordinate system. The specific implementation steps include S3-5 to S3-8:

[0070] S3-5: For voxel features of dimension (L, W, H, C), they are divided into two parts along the feature dimension, both of which have dimensions (L, W, H, C / 2), and are input into the sparse Transformer module in the Cartesian coordinate system and the sparse Transformer module in the polar coordinate system respectively.

[0071] S3-6: In the sparse Transformer module in the Cartesian coordinate system, the input voxel features are first divided into windows in the Cartesian coordinate system, the relative positions of the voxel features in the same window are encoded, and then multi-head attention calculations are performed to complete large-scale information interaction in the local space.

[0072] S3-7: In the sparse Transformer module in the polar coordinate system, the input voxel features are first divided into windows in the polar coordinate system, the relative positions of the voxel features in the same polar coordinate window are encoded, and then multi-head attention calculation is performed to complete the information interaction between long-distance and short-distance point clouds.

[0073] S3-8: The features output by the sparse Transformer module in the Cartesian coordinate system and the sparse Transformer module in the polar coordinate system are concatenated and input into the MLP for feature fusion. First, the first layer of MLP is input for feature fusion, and then the output of the first layer of MLP is input into the second layer of MLP for fusion again. The output results of the two layers of MLP are merged using residual connection to obtain the final output features.

[0074] In S4 of the present invention, the high compression is the output feature F of the 3D feature extraction network 4 =(L / 8, W / 8, H / 8, 128), after a sparse convolution module with a convolution kernel size of (1,1,3), feature extraction is performed in the height direction, and two times downsampling is performed in the z direction. Then the obtained voxel feature space is highly compressed, and the features are spliced ​​along the z direction. Finally, BEVmap = (L / 8, W / 8, 256) is obtained. The purpose of high compression is to project it into 2D space. In 2D space, 2D convolution in image processing can be used to process point clouds to reduce the complexity of operations.

[0075] S4: Use a 2D feature extraction network composed of 2D convolution and recursive gated convolution to further extract features in 2D space. The specific implementation steps include S4-1 to S4-3:

[0076] S4-1: The 2D feature map BEVmap is input into a convolution module with a convolution kernel size of (3,3) for one-fold downsampling. At the same time, the feature dimension is converted to 128. Then, it is extracted from 2D features through 5 convolution modules with a convolution kernel size of (3,3). It is input into the recursive gated convolution module for high-order spatial interaction in 2D space to obtain the 2D feature map BEVmap. 1 .

[0077] S4-2: 2D feature map BEVmap 1 Input a convolution module with a convolution kernel size of (3,3) and perform 2x downsampling. At the same time, the feature dimension is converted to 256. Then, it passes through 5 convolution modules for 2D feature extraction and inputs the recursive gated convolution module for high-order spatial interaction in 2D space to obtain a 2D feature map BEVmap. 2 .

[0078] S4-3: 2D feature map BEVmap 1 、BEVmap 2 Input deconvolution module, upsample by 1x and 2x in x and y directions respectively, BEVmap 1 The feature dimension of BEVmap is converted from 128 to 256. 2The feature dimension of remains unchanged, and the two are fused at multiple scales along the feature dimension to obtain the final 2D feature map BEVmap final , whose feature dimension is 512.

[0079] In the specific implementation step S4-2, the recursive gated convolution module specifically includes steps S4-2-1 to S4-2-3:

[0080] S4-2-1: For an input feature map of dimension (L, W, C), first perform linear projection to double its feature dimension to obtain a feature map of dimension (L, W, 2C), and then divide the feature into 4 parts: p 1 ,q 1 ,q 2 ,q 3 The feature dimensions of each part are: C / 4, C / 4, C / 2, C.

[0081] S4-2-3: To q 1 ,q 2 ,q 3 Perform depth-wise convolution to obtain q′ 1 , q′ 2 , q′ 3 , p 1 ,q 1 ′,q′ 2 , q′ 3 The calculation is done according to the following formula:

[0082] p k+1 =q′ k *mlp(p k ),k=1,2,3

[0083] S4-6: Put p 4 Input MLP for feature extraction and obtain the final output feature map.

[0084] S5: BEVmap final Input the detection head in the first stage of detection, and the generated target detection box expression is as follows:

[0085] (x 1 ,y 1 , z 1 , l 1 , w 1 ,h 1 ,θ 1 )

[0086] where x 1 ,y 1 , z 1 is the position of the center point of the candidate box in the first stage, l 1 , w1 ,h 1 is the length, width and height of the candidate box in one stage, θ 1 It is the rotation angle of the candidate box along the z-axis.

[0087] For the first-stage detection box, the loss function calculation formula is: Loss 1 =α 1 Loss cls +α 2 Loss loc +α 3 Loss dir , where Loss cls is the classification loss of the detection box, Loss loc is the coordinate loss of the detection box, Loss dir is the direction loss of the detection box, α 1 , α 2 , α 3 They are the loss weights for classification loss, coordinate loss, and direction loss respectively.

[0088] S6: In order to fully extract the local features of the first-stage detection frame, the present invention adopts a grid point feature aggregation method to divide the first-stage target detection frame into several uniform grid points, and adopts a voxel search method to aggregate the multi-scale voxel features in the 3D feature extraction network to the grid points, and further aggregate all grid point features in the candidate frame to obtain the feature representation of the candidate frame. Using the second-stage detection head, the final target detection frame is predicted by combining the aggregated grid point features with the first-stage target detection frame. The specific implementation steps include S6-1 to S6-5:

[0089] S6-1: Divide the target detection box generated in the first stage into 6×6×6 uniform grids and obtain the coordinates of the center point of each grid.

[0090] S6-2: Using the voxel search method, the third and fourth layers of the 3D feature extraction network are respectively 3 , F 4 In the above example, we search for voxels whose Manhattan distance from the center of the grid is between 2 and 4. The Manhattan distance between two voxels is determined by the index of the voxel. ab The calculation formula is as follows:

[0091] D ab =|x a -x b |+|y a -y b |+|z a -z b |

[0092] D abis the Manhattan distance between voxel a and voxel b,

[0093] (x a ,y a , z a ) is the index of voxel a in the voxel feature space, (x b ,y b , z b ) is the index of voxel b in the voxel feature space.

[0094] S6-3: For the voxels found in S6-2, the voxel features and coordinates (changed to centroid coordinates) are respectively input into MLP for feature extraction, and the extracted voxel features and coordinate features are input into MLP to further fuse the voxel features and coordinate information to obtain all voxel features within a certain distance of the grid point, and the maximum pooling is used to aggregate the features of the surrounding voxels to the grid point.

[0095] S6-4: For each target detection box, 216 grid points that aggregate multi-scale 3D voxel features can be obtained. All grid point features are concatenated to obtain a one-dimensional feature vector, which is passed into the MLP for feature fusion to obtain the feature representation of the candidate box.

[0096] S6-5: Input the obtained feature representation of the candidate frame into the classification head and the regression head. The classification head obtains the classification prediction of the final detection frame. The regression head obtains the coordinate deviation, size deviation, and rotation angle deviation based on the first-stage detection frame. The information of the first-stage target detection frame is added with the corresponding deviation to obtain the corrected target detection frame, where the target detection frame information includes: the coordinates of the center point of the target detection frame on the x, y, and z axes, the length, width, and height of the detection frame, the rotation angle of the detection frame, and the category of the detection frame; then non-maximum suppression is performed on the corrected detection frame to obtain the final target detection frame.

[0097] For the second-stage target detection frame, the loss function calculation formula is: Loss 2 =β 1 Loss cls +β 2 Loss reg +β 3 Loss corner +β 4 Loss iou3d , where Loss cls is the classification loss of the detection box, Loss reg is the regression loss of the detection box, Loss corner is the corner loss of the detection box, Loss iou3d is the 3D intersection-over-union loss of the detection box, β 1 , β 2 , β3 , β 4 They are the loss weights of classification loss, regression loss, corner loss, and 3D intersection-over-union loss respectively.

[0098] During training, the optimizer type selected by the present invention is Adam_onecycle, the learning rate is 0.007, the weight decay coefficient is 0.01, the momentum parameter is 0.9, the batch size is 4, and 40 rounds of parallel training are performed on two GeForce RTX 3090 graphics cards. Prediction is performed on the KITTI data validation set, and the effect is evaluated on the KITTI evaluation program. The evaluation indicator used is the average precision of the 3D detection box (AP3D) at medium difficulty. The specific results are shown in Table 1:

[0099] Table 1

[0100] Model Car-AP3D(%) Cyclist-AP3D (%) Pedestrain-AP3D (%) CenterPoint 75.87 62.40 48.70 PointPillar 77.28 62.68 52.29 SECOND 78.62 67.15 52.98 SECOND-IoU 79.09 71.31 55.74 PointRCNN 78.70 72.11 54.41 PointRCNN-IoU 78.75 71.34 58.32 Part-A2-Free 78.72 74.29 65.99 Part-A2-Anchor 79.40 69.90 60.05 GD-MAE 82.01 67.16 48.40 PV RCNN++ 82.75 72.01 51.59 VoxelRCNN 82.83 71.31 51.49 The present invention 83.11 75.62 57.20

[0101] The verification results of the method of the present invention and the prior art on the KITTI dataset are compared as shown in Table 1. The target detection accuracy of vehicles, cyclists and pedestrians reached 83.11, 75.62 and 57.20 respectively, which were improved by 0.28%, 4.31% and 5.71% respectively compared with the baseline method VoxelRCNN. The effect diagram of the model detection results is shown in Figure 4 , Figure 5 , Figure 6 As shown, Figure 4 , Figure 5 , Figure 6 The sub-images in the figure are respectively, from top to bottom, a visualization of the real detection box of the KITTI dataset, a visualization of the detection result of the VoxelRCNN model, and a visualization of the detection result of the model of the present invention. The detection accuracy of the method of the present invention exceeds that of most existing methods, and achieves high-precision 3D object detection.

Claims

1. A point cloud target detection method based on recursive gated convolution and sparse Transformer, characterized in that: The following steps are involved: S1: Collect point cloud data in real driving scenarios through LiDAR and perform data preprocessing on the collected point cloud data; S2: voxelize the preprocessed point cloud data and divide the preprocessed point cloud data into a 3D sparse voxel feature grid with a fixed dimension; S3: Use a 3D feature extraction network consisting of a 3D sparse convolution module and a sparse Transformer module to downsample the 3D sparse voxel feature grid and extract multi-scale voxel features; S4: The downsampled voxel features are highly compressed, and a 2D feature extraction network consisting of 2D convolution and recursive gated convolution is used to extract features in 2D space; S5: Input the features in the 2D space into the first-stage detection head to generate the first-stage target detection frame; S6: Grid point feature aggregation is used to divide the first-stage target detection box into several uniform grid points. Voxel search is used to aggregate multi-scale voxel features to grid points, and all grid point features in the candidate box are aggregated to obtain the feature representation of the candidate box. The second-stage detection head is used to predict the final target detection box by combining the aggregated grid point features with the first-stage target detection box.

2. The point cloud target detection method based on recursive gated convolution and sparse Transformer according to claim 1, characterized in that: The specific implementation process of the data preprocessing is as follows: S1-1: Filter out the point clouds that are beyond the valid range of the dataset and only retain the valid point clouds within the range; S1-2: Perform random flipping, random rotation, and random scaling operations on the filtered point cloud for data enhancement; S1-3: After data enhancement, for the current frame data that needs to be trained, point clouds in the real bounding boxes of other frames in the training set are extracted and placed in the empty positions where there are no targets in the current frame, increasing the number of targets in the current frame and dynamically generating target data for the current frame; S1-4: Randomize the point cloud order of the point cloud data obtained in S1-3.

3. The point cloud target detection method based on recursive gated convolution and sparse Transformer according to claim 1, characterized in that: The specific implementation process of the voxelization process is as follows: For each point P in the point cloud i ={(x, y, z, r)}, P i ∈R 3 , where x, y, z represent the coordinates of the point in 3D space, and r represents the radar reflection intensity; first, the preprocessed point cloud is grouped into voxels according to the coordinates, and the average value of the point cloud data in the voxel group is calculated as the feature of the voxel to obtain the feature v of voxel j j ; After the point cloud is voxelized, a j The composed dimensions are (L, W, H, C), where L and W represent the dimensions of the 3D sparse voxel feature grid along the x and y directions, H represents the dimension of the 3D sparse voxel feature grid along the z direction, and C represents the feature dimension of the 3D sparse voxel. In this case, the feature dimension is 4, i.e. (x, y, z, r).

4. The point cloud target detection method based on recursive gated convolution and sparse Transformer according to claim 3, characterized in that: The specific implementation process of step S3 is as follows: S3-1: The voxelized 3D sparse voxel features (L, W, H, C) are downsampled in the x, y, and z directions through a sub-manifold sparse convolution block to increase the feature dimension to 16, and then the voxel features are extracted again through 4 sub-manifold sparse convolution modules to obtain the first-layer voxel features F1 = (L, W, H, 16); S3-2: F1 is passed through a sparse convolution module, downsampled twice in the x, y, and z directions, and the feature dimension is increased to 32, and then passed through 4 sub-flow sparse convolution modules again to extract voxel features, and the second layer voxel features F2 = (L / 2, W / 2, H / 2, 32) are obtained; S3-3: According to the down-sampling correspondence between the voxels in F1 and the voxels in F2, the centroid coordinates of the corresponding voxels in F1 are averaged, the centroid coordinates of the voxels in F2 are calculated, and the voxel features and the centroid coordinates are input into the sparse Transformer module to extract features; Then, it passes through a sparse convolution module, performs two-fold downsampling in the x, y, and z directions, and increases the feature dimension to 64. It then passes through four sub-flow sparse convolution modules again to extract voxel features, and obtains the third-layer voxel feature F3 = (L / 4, W / 4, H / 4, 64); S3-4: According to the downsampling correspondence between the voxels in F2 and the voxels in F3, the center of mass coordinates of the corresponding voxels in F2 are averaged, the center of mass coordinates of the voxels in F3 are calculated, and the voxel features and the center of mass coordinates of the voxels are input into the sparse Transformer module to extract the features; then, a sparse convolution module is passed to perform two-fold downsampling in the x, y, and z directions, and the feature dimension is increased to 128, and then four sub-manifold sparse convolution modules and an additional sparse convolution module are passed again to extract the voxel features, and the fourth layer of voxel features F4 = (L / 8, W / 8, H / 8, 128) is obtained.

5. The point cloud target detection method based on recursive gated convolution and sparse Transformer according to claim 4, characterized in that: The sparse Transformer module is composed of a sparse Transformer module in a Cartesian coordinate system and a sparse Transformer module in a polar coordinate system. The specific implementation process is as follows: S3-4-1: For voxel features with dimensions (L, W, H, C), they are divided into two parts along the feature dimension, both of which have dimensions (L, W, H, C / 2), and are input into the sparse Transformer module in the Cartesian coordinate system and the sparse Transformer module in the polar coordinate system respectively; S3-4-2: In the sparse Transformer module in the Cartesian coordinate system, the input voxel features are first divided into windows in the Cartesian coordinate system, the relative positions of the voxel features in the same window are encoded, and then multi-head attention calculation is performed; S3-4-3: In the sparse Transformer module in the polar coordinate system, the input voxel features are first divided into windows in the polar coordinate system, the relative positions of the voxel features in the same polar coordinate window are encoded, and then multi-head attention calculation is performed; S3-4-4: After concatenating the features output by the sparse Transformer module in the Cartesian coordinate system and the sparse Transformer module in the polar coordinate system, they are input into two layers of MLP for feature fusion, and the output results of the two layers of MLP are merged using residual connection to obtain the final output features.

6. The point cloud target detection method based on recursive gated convolution and sparse Transformer according to claim 5, characterized in that: The height compression is to extract the features in the height direction of the output feature F4 = (L / 8, W / 8, H / 8, 128) of the 3D feature extraction network through a sparse convolution module with a convolution kernel size of (1, 1, 3), and perform two times downsampling in the z direction, and then splice the obtained voxel features along the z direction to finally obtain BEVmap = (L / 8, W / 8, 256).

7. The point cloud target detection method based on recursive gated convolution and sparse Transformer according to claim 6, characterized in that: The 2D feature extraction network composed of 2D convolution and recursive gated convolution is used to extract features in 2D space. The specific process is as follows: S4-1: Input the 2D feature map BEVmap into a convolution module with a convolution kernel size of (3,3) for one-fold downsampling, and convert the feature dimension to 128. Then, it passes through 5 convolution modules with a convolution kernel size of (3,3) for 2D feature extraction, and inputs the recursive gated convolution module to obtain the 2D feature map BEVmap1. S4-2: Input the 2D feature map BEVmap1 into a convolution module with a convolution kernel size of (3,3) for 2x downsampling, and convert the feature dimension to 256. Then, it passes through 5 convolution modules with a convolution kernel size of (3,3) for 2D feature extraction, and inputs the recursive gated convolution module to obtain the 2D feature map BEVmap2. S4-3: Input the 2D feature maps BEVmap1 and BEVmap2 into the deconvolution module, perform 1x and 2x upsampling along the x and y directions respectively, the feature dimension of BEVmap1 is converted from 128 to 256, the feature dimension of BEVmap2 remains unchanged, and the two are multi-scale fused along the feature dimension to obtain the final 2D feature map BEVmap final , whose feature dimension is 512.

8. The point cloud target detection method based on recursive gated convolution and sparse Transformer according to claim 7, characterized in that: The specific implementation process of the recursive gated convolution module is as follows: S4-2-1: For the input feature map of dimension (L, W, C), first linearly project it to double its feature dimension to obtain a feature map of dimension (L, W, 2C), and then divide the feature into 4 parts: p1, q1, q2, q3, and the feature dimensions of each part are: C / 4, C / 4, C / 2, C; S4-2-2: Perform depth-wise convolution on q1, q2, q3 to obtain q ′ 1,q ′ 2, q ′ 3. Change p1,q ′ 1,q ′ 2, q ′ 3 Calculate according to the following formula: p k+1 =q′ k *mlp(p k )k=1,2,3 S4-2-3: Input p4 into MLP for feature extraction to obtain the final output feature map.

9. The point cloud target detection method based on recursive gated convolution and sparse Transformer according to claim 8, characterized in that: The specific implementation process of step S6 is as follows: S6-1: Divide the target detection box generated in the first stage into uniform grids and obtain the coordinates of the center point of each grid; S6-2: Using the voxel search method, the voxels with a Manhattan distance of 2 and 4 from the center of the grid are searched in the third and fourth layer voxel features F3 and F4 of the 3D feature extraction network, respectively. The Manhattan distance between two voxels is determined by the voxel index; S6-3: For the voxels found in S6-2, the voxel features and their centroid coordinates are respectively input into the MLP for feature extraction, and the respective outputs are spliced ​​and then input into the multi-layer perceptron MLP to fuse multi-scale features to obtain all voxel features within the set distance of the grid point. The maximum pooling method is used to aggregate the features of the surrounding voxels to the grid point; S6-4: For each target detection box, multiple grid points of aggregated multi-scale 3D voxel features are obtained, all grid point features are concatenated to obtain a one-dimensional feature vector, which is passed into the MLP for feature fusion to obtain the feature representation of the candidate box; S6-5: Input the obtained feature representation of the candidate frame into the classification head and the regression head. The classification head obtains the classification prediction of the final detection frame. The regression head obtains the coordinate deviation, size deviation, and rotation angle deviation based on the first-stage detection frame. The information of the first-stage target detection frame is added with the corresponding deviation to obtain the corrected target detection frame, where the target detection frame information includes: the coordinates of the center point of the target detection frame on the x, y, and z axes, the length, width, and height of the detection frame, the rotation angle of the detection frame, and the category of the detection frame; then non-maximum suppression is performed on the corrected detection frame to obtain the final target detection frame.

Images