Three-dimensional target detection method based on full sparse hybrid network

Abstract

The invention relates to the technical field of computer vision and three-dimensional perception, and relates to a three-dimensional target detection method based on a full sparse hybrid network. The complementary advantages of a sparse convolution encoder and sparse attention encoder mixed architecture are utilized, the efficient characteristic of a convolution network is reserved, meanwhile, the receptive field of a feature extraction operator is greatly enlarged, and therefore an efficient and accurate three-dimensional target detector is obtained. A dynamic sparse detection head is adopted, and a high-quality balanced training sample can be provided in a training stage, so that a detection model can be better optimized; according to the method, the quality of feature expression is effectively improved, so that the accuracy and the stability of detection are remarkably improved, and the method is particularly excellent in complex scene and long-distance target detection; according to the method, effective information in sparse features can be more fully mined, so that the detection result is more accurate and robust.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and three-dimensional perception technology, and more specifically, to a three-dimensional target detection method based on a fully sparse hybrid network. Background Art

[0002] Three-dimensional object detection has attracted increasing research interest due to its growing applications in fields such as autonomous driving perception and robotic manipulation. Current LiDAR-based 3D object detection techniques typically convert 3D sparse features into 2D dense feature maps to leverage existing 2D object detection techniques. However, these detectors often waste significant computational and memory resources when processing unoccupied voxels. This problem becomes particularly prominent when performing long-range detection. Therefore, there is an urgent need to develop fully sparse 3D object detectors that can directly extract sparse features from sparse input without the need for dense operators.

[0003] To improve detection efficiency, existing fully sparse 3D object detectors only perform feature extraction on non-empty voxels. This makes it difficult to establish interaction between voxels that are far apart. Therefore, compared with dense 3D object detectors, existing fully sparse 3D object detectors often lack the ability to extract features at long distances. In addition, because lidar can typically only scan the surface of the target, non-empty voxels are usually only present on the target surface and are difficult to diffuse to the target center. This leads to the problem of missing target center features in fully sparse 3D object detectors. This problem affects the optimization of fully sparse 3D object detection networks. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology in low accuracy of three-dimensional target detection at long distances, and to provide a three-dimensional target detection method based on a fully sparse hybrid network, which can effectively improve the accuracy of three-dimensional target detection at long distances.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A 3D object detection method based on a fully sparse hybrid network is provided, comprising the following steps:

[0007] S1. Data Processing: Acquire point cloud data, divide and preprocess the point cloud data into datasets, and divide the point cloud dataset into training dataset and test dataset.

[0008] S2. Build a fully sparse hybrid detection network: The fully sparse hybrid detection network includes a voxel encoding layer for encoding point clouds in voxels, a sparse convolutional encoder for extracting local features, a sparse attention encoder for modeling long-range feature interactions, and a dynamic sparse detection head for selecting high-quality positive samples for training.

[0009] S3. Input the training dataset into the fully sparse hybrid detection network and iteratively train the fully sparse hybrid detection network;

[0010] S4. Use the trained model to perform inference and prediction on the test set point cloud data.

[0011] The present invention provides a three-dimensional target detection method based on a fully sparse hybrid network. The method utilizes the complementary advantages of the hybrid architecture of a sparse convolutional encoder and a sparse attention encoder, greatly increasing the receptive field of the feature extraction operator while retaining the high efficiency of the convolutional network, thereby obtaining an efficient and accurate three-dimensional target detector; the use of a dynamic sparse detection head can provide high-quality and balanced training samples during the training phase, thereby enabling the detection model to be better optimized; the present invention effectively improves the quality of feature expression, thereby significantly improving the accuracy and stability of detection, especially in complex scenes and long-distance target detection; the present invention can more fully mine the effective information in sparse features, making the detection results more accurate and robust.

[0012] Furthermore, in step S2, the point cloud in the voxel is encoded by calculating the feature mean of the points contained in the voxel; specifically, the method includes: inputting a point set P in a non-empty voxel = {p1, p2, ..., p n}, each non-empty voxel is encoded as follows:

[0013]

[0014] Where p j is the voxel v i The feature of the j-th point in the voxel encoding layer is used to obtain a non-empty voxel feature set n represents the number of points in the set.

[0015] Furthermore, in step S2, the sparse convolution encoder includes multiple processing modules, each processing module includes a three-dimensional sparse convolution layer and two residual convolution blocks, each residual convolution block includes two layers of semi-manifold sparse convolution layers, which are used to use residual connections as shortcuts to connect input and output; each convolution layer is connected to a batch normalization layer and a nonlinear activation function; each processing module downsamples and extracts features on three-dimensional voxel features, and splices the sparse feature outputs of the last three processing modules to obtain multi-scale sparse features.

[0016] Furthermore, the splicing method of the last three processing modules includes: amplifying the voxel coordinates output by the last two processing modules by 2 times and 4 times respectively, connecting them with the voxel coordinates output by the third-to-last processing module, and then splicing the feature outputs of the three processing modules using the feature sequence splicing method to obtain multi-scale sparse features. Where d is the number of feature channels, n4, n5 and n6 are the number of voxel features output by the third to last processing module, the second to last processing module and the last processing module respectively; finally, the multi-scale sparse features are compressed along the high dimension to obtain the multi-scale BEV sparse features

[0017] Furthermore, the sparse attention encoder includes multiple layers of strip window-based deep learning models; wherein the strip windows cover the entire point cloud scene in a certain direction, and the strip window directions of adjacent deep learning model layers alternate with each other, thereby providing a receptive field covering the entire scene.

[0018] Furthermore, the sparse attention coding layer includes the following steps when processing data:

[0019] Input N non-empty voxels in and Represent voxel ν i coordinates and characteristics of;

[0020] Divide the above N non-empty voxels into corresponding strip windows, and the size of each strip window is s×w, where s and w are the size of the point cloud scene and the width of the strip window respectively:

[0021]

[0022] Where, and Represent voxel ν i Bar window index in the X and Y directions;

[0023] Calculate the Query, Key, and Value matrices of the linear attention layer respectively:

[0024] q i =φ(f i W q ),k i =φ(f i W k ),v i =f i W v ,

[0025] Where W q 、W k and W v Represent the learnable weight matrix, φ(·) represents the ReLU nonlinear activation function, and the Query matrix is ​​obtained. Key Matrix and Value matrix c represents the feature dimension of the vector, and N represents the number of non-empty voxels in the input;

[0026] Next, calculate the X-axis direction strip window matrix:

[0027]

[0028] Where, Indicates whether the i-th voxel belongs to the j-th strip window; m represents the number of divided windows;

[0029] Use the KV matrix to calculate the new Value matrix:

[0030]

[0031] Where, Indicates the The kv matrix of a strip window, the denominator is a normalized term;

[0032] The calculation process for the Y-axis direction is the same as that for the X-axis direction;

[0033] Finally, the new Value matrix passes through a feedforward network to obtain the updated features of the sparse voxels: f i ′=FFN(v i ′), FFN(·) represents the feedforward network module

[0034] Furthermore, the dynamic sparse detection head is used to generate high-quality positive samples for model training and output the final prediction box; in the training stage, first find the n voxels closest to the center of the target box for the real box Then calculate the selection cost of the voxel:

[0035]

[0036] Where Pred(·) represents a prediction module composed of multiple convolutional layers, which generates prediction boxes and classification values ​​based on the input voxel features; cls (·) and l reg (·) represents classification loss and regression loss respectively; b t Represents the true target box in the current training sample;

[0037] Next, calculate the intersection-over-union ratio between the n predicted boxes and the true box and add them together:

[0038]

[0039] Finally, the top k voxel optimal positive samples with the smallest selection cost are selected for model training optimization.

[0040] Furthermore, step S3 includes:

[0041] S31. Input the point cloud data of the training dataset in step S1 into the fully sparse hybrid detection network, and use the true box and the predicted box as inputs to the loss function; use the focal loss as the classification loss function and the smooth L1 loss as the box regression loss function; use the Adam optimization algorithm to update the parameters of the fully sparse hybrid detection network;

[0042] S32. Iterate the network training epochs on the training dataset until the overall loss function converges. The losses for different tasks and the overall loss are calculated as follows:

[0043]

[0044] Where, l cls and l b Represents classification loss and box regression loss respectively, N c and N p Represents the number of classification predictions and the number of positive anchor boxes, c i and Represents the target category true value and target category predicted value at the i-th position, λ b and λ d They represent the weight of the box regression loss and the weight of the target orientation loss, α represents the category balance weight, β represents the difficult sample adjustment index, and b ij 、 Represents the true value and predicted value of the j-th regression item of the i-th box.

[0045] Furthermore, step S1 includes:

[0046] S11. Use a lidar to record a point cloud dataset and sample the recorded point cloud data at a frequency of Y to obtain X frames of point cloud data. Then, annotate the three types of objects in the point cloud with 3D ground truth boxes: motor vehicles, pedestrians, and non-motor vehicles. Then, divide the X frames of point cloud data into a training dataset and a test dataset at a 1:1 ratio.

[0047] S12. Perform 3D real-world frame pasting data augmentation; randomly select n 3D real-world frames and the target point clouds contained in the real-world frames from the training data set, and then perform collision detection. If the selected 3D real-world frame overlaps with the existing 3D real-world frame in the current point cloud sample, it is considered that a collision has occurred. Finally, the target point clouds contained in the 3D real-world frames that have not collided are pasted. Paste copied to current point cloud Get point cloud Among them, N0 and N s Represents the number of points in the original point cloud P0 and the single target point cloud P s The number of points;

[0048] S13. Perform random flip data enhancement to randomly flip the point cloud P1 along the x-axis or y-axis;

[0049] S14. Perform random rotation data augmentation, randomly generate a rotation angle θ1 in the interval [-π / 4, π / 4], and rotate the point cloud P1 around the z-axis by an angle θ1;

[0050] S15. Perform random scaling data enhancement on the point cloud; randomly generate a scale factor s1 in the interval [0.95, 1.05], and multiply the overall coordinate value of the point cloud P1 by the scale factor s1;

[0051] S16. shuffle the order of the points in point cloud P1;

[0052] S17. Filter out points and 3D ground truth boxes outside the detection range in point cloud P1;

[0053] S18. Voxelize the point cloud P1 and define the voxel size as s x ×s y ×s z , divide each point in the point cloud into corresponding voxels; s x 、s y 、s z Represents the size of the voxel in the X, Y, and Z dimensions respectively.

[0054] Furthermore, in step S4, the trained fully sparse hybrid detection network is used to perform inference prediction on the point cloud in the test dataset, and the non-maximum suppression method is used to filter redundant prediction boxes to output the final three-dimensional detection prediction result.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. The present invention proposes a three-dimensional target detection method based on a fully sparse hybrid network, which combines a hybrid architecture of sparse convolution and attention mechanism. While maintaining the efficiency of the overall detection process, it can capture richer and more discriminative feature information. While improving computational efficiency through sparse convolution, the attention module further enhances the modeling capability of key area features, effectively improving the quality of feature expression, thereby significantly improving the accuracy and stability of detection, especially in complex scenes and long-distance target detection.

[0057] 2. The present invention proposes a three-dimensional object detection method based on a fully sparse hybrid network, which utilizes a dynamic sparse detection head that adaptively selects the optimal set of positive samples for training based on the sample distribution. Compared with traditional fixed strategies, this dynamic mechanism can more rationally distribute positive and negative samples, alleviate sample imbalance, and improve the convergence speed and final performance of the network model training phase. Thanks to the higher-quality screening and utilization of positive samples, the present invention can more fully exploit the effective information in sparse features, making the detection results more accurate and robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a schematic flow chart of the method of Example 1;

[0059] Figure 2 This is a schematic diagram of the method flow of Example 2. DETAILED DESCRIPTION

[0060] The present invention will be further described below with reference to specific embodiments. The accompanying drawings are for illustrative purposes only and are schematic, not actual, representations. They should not be construed as limiting the present invention. To better illustrate the embodiments of the present invention, some components in the drawings may be omitted, enlarged, or reduced in size, and do not represent the actual dimensions of the products. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted from the drawings.

[0061] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "up", "down", "left", "right", etc. indicate directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. This is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0062] Example 1

[0063] This embodiment is the first embodiment of a three-dimensional target detection method based on a fully sparse hybrid network. Figure 1 As shown, the following steps are included:

[0064] Step S1. Data processing: Acquire point cloud data, and perform data set division and preprocessing on the point cloud data.

[0065] S11. Use a LiDAR to record a point cloud dataset and sample the recorded point cloud data at a 2Hz frequency to obtain 10,000 frames of point cloud data. Then, annotate the three types of objects in the point cloud with 3D ground truth boxes: motor vehicles, pedestrians, and non-motor vehicles. Then, divide the 10,000 frames of point cloud data into a training dataset and a test dataset at a 1:1 ratio.

[0066] S12. Perform 3D real-world frame pasting data augmentation; randomly select n 3D real-world frames and the target point clouds contained in the real-world frames from the training data set, and then perform collision detection. If the selected 3D real-world frame overlaps with the existing 3D real-world frame in the current point cloud sample, it is considered that a collision has occurred. Finally, the target point clouds contained in the 3D real-world frames that have not collided are pasted. Paste copied to current point cloud Get point cloud Among them, N0 and N s Represents the number of points in the original point cloud P0 and the single target point cloud P s The number of points;

[0067] S13. Perform random flip data enhancement to randomly flip the point cloud P1 along the x-axis or y-axis;

[0068] S14. Perform random rotation data augmentation, randomly generate a rotation angle θ1 in the interval [-π / 4, π / 4], and rotate the point cloud P1 around the z-axis by an angle θ1;

[0069] S15. Perform random scaling data enhancement on the point cloud; randomly generate a scale factor s1 in the interval [0.95, 1.05], and multiply the overall coordinate value of the point cloud P1 by the scale factor s1;

[0070] S16. shuffle the order of the points in point cloud P1;

[0071] S17. Filter out points and 3D ground truth boxes outside the detection range in point cloud P1;

[0072] S18. Voxelize the point cloud P1 and define the voxel size as s x ×s y ×s z , divide each point in the point cloud into corresponding voxels; s x 、s y 、s z Represents the size of the voxel in the X, Y, and Z dimensions respectively.

[0073] Step S2: Build a fully sparse hybrid detection network.

[0074] S21. Build the voxel encoding layer. This layer is responsible for encoding the point cloud in the voxel. The encoding method is to find the feature mean of the points contained in the voxel. Input a point set P in a non-empty voxel = {p1,p2,…,p n}, each non-empty voxel is encoded as follows:

[0075]

[0076] Where p j is the voxel v i The feature of the j-th point in the voxel encoding layer is used to obtain a non-empty voxel feature set

[0077] S22. Build a sparse convolutional encoder. The sparse convolutional encoder consists of 6 processing modules, each of which contains a three-dimensional sparse convolution layer and two residual convolution blocks. Each residual convolution block consists of two layers of semi-manifold sparse convolution layers, using residual connections as shortcuts to connect the input and output. Each convolution layer is followed by a batch normalization layer and a nonlinear activation function ReLU (Rectified Linear Unit). Each processing module downsamples and extracts features from the three-dimensional voxel features, and splices the sparse feature outputs of the last three processing modules to obtain multi-scale sparse features. The specific splicing method is: enlarge the voxel coordinates output by the fifth processing module by 2 times, enlarge the voxel coordinates output by the sixth processing module by 4 times, and then align them with the voxel coordinates output by the fourth processing module. Then, the feature outputs of these three processing modules are spliced ​​using feature sequence splicing to obtain multi-scale sparse features. Where d is the number of feature channels, n4, n5 and n6 are the number of voxel features output by the 4th, 5th and 6th processing modules respectively. Finally, the multi-scale sparse features are compressed along the height dimension to obtain the multi-scale BEV (Bird's Eyes View) sparse features. The specific compression method is to directly add and sum the voxel features with the same BEV plane coordinates but different height coordinates.

[0078] S23. Build a sparse attention encoder. The sparse attention encoder consists of multiple layers of transformers based on strip windows. The strip windows cover the entire point cloud scene in a certain direction, and the strip windows of adjacent transformer layers alternate in direction, providing a receptive field that covers the entire scene.

[0079] Specifically, input N non-empty voxels in and Represent voxel ν iCoordinates and features; First, divide the above N non-empty voxels into corresponding strip windows, and the size of each strip window is s×w, where s and w are the size of the point cloud scene and the width of the strip window respectively:

[0080]

[0081] Where, and Represent voxel ν i Bar window index in the X and Y directions;

[0082] Next, calculate the Query, Key, and Value matrices of the linear attention layer respectively:

[0083] q i =φ(f i W q ),k i =φ(f i W k ),v i =f i W v ,

[0084] Where W q 、W k and W v Represent the learnable weight matrix, φ(·) represents the ReLU nonlinear activation function, and the Query matrix is ​​obtained. Key Matrix and Value matrix

[0085] Next, take the strip window in the X-axis direction as an example to calculate matrix:

[0086]

[0087] Where, Indicates whether the i-th voxel belongs to the j-th strip window; when calculating the strip window in the Y-axis direction, it is only necessary to add Replace with

[0088] Then, use the KV matrix to calculate the new Value matrix:

[0089]

[0090] Where, Indicates the The kv matrix of a strip window, the denominator is a normalized term;

[0091] The calculation process for the Y-axis direction is the same as that for the X-axis direction;

[0092] Finally, the new Value matrix passes through a feed-forward network to obtain the updated features of the sparse voxels:

[0093] f i ′=FFN(v i ′).

[0094] S24. Build a dynamic sparse detection head. The dynamic sparse detection head is used to generate high-quality positive samples for model training and output the final prediction box. During the training phase, the first step is to find the n voxels closest to the center of the target box for the true box. Then calculate the selection cost of the voxel:

[0095]

[0096] Where Pred(·) represents a prediction module composed of multiple convolutional layers, which generates prediction boxes and classification values ​​based on the input voxel features; cls (·) and l reg (·) denotes classification loss and regression loss respectively;

[0097] Next, calculate the intersection-over-union ratio between the n predicted boxes and the true box and add them together:

[0098]

[0099] Finally, the top k voxel optimal positive samples with the smallest selection cost are selected for model training optimization.

[0100] Step S3: Input the training data set into the fully sparse hybrid detection network and iteratively train the fully sparse hybrid detection network.

[0101] S31. Input the point cloud data of the training dataset in step S1 into the fully sparse hybrid detection network, and use the true box and the predicted box as the input of the loss function; use the focal loss as the classification loss function and the smooth L1 loss as the box regression loss function; use the Adam optimization algorithm to update the parameters of the fully sparse hybrid detection network;

[0102] S32. Iterate the network training epochs on the training dataset until the overall loss function converges. The losses for different tasks and the overall loss are calculated as follows:

[0103]

[0104] l=l cls +λ b lb

[0105] Where, l cls and l b Represents classification loss and box regression loss respectively, N c and N p Represents the number of classification predictions and the number of positive anchor boxes, c i and Represents the target category true value and target category predicted value at the i-th position, λ b and λ d denote the weight of the box regression loss and the weight of the target orientation loss respectively.

[0106] Step S4: Use the trained model to perform inference and prediction on the test set point cloud data.

[0107] The trained fully sparse hybrid detection network is used to perform inference prediction on the point cloud in the test dataset, and the non-maximum suppression method is used to filter redundant prediction boxes to output the final 3D detection prediction results.

[0108] This embodiment provides a three-dimensional target detection method based on a fully sparse hybrid network, which utilizes the complementary advantages of the hybrid architecture of sparse convolutional encoders and sparse attention encoders to greatly increase the receptive field of the feature extraction operator while retaining the high efficiency of the convolutional network, thereby obtaining an efficient and accurate three-dimensional target detector; the use of a dynamic sparse detection head can provide high-quality and balanced training samples during the training phase, so that the detection model can be better optimized; the present invention effectively improves the quality of feature expression, thereby significantly improving the accuracy and stability of detection, especially in complex scenes and long-distance target detection; the present invention can more fully mine the effective information in sparse features, making the detection results more accurate and robust.

[0109] Example 2

[0110] This embodiment is a second embodiment of a three-dimensional target detection method based on a fully sparse hybrid network. This embodiment is similar to the first embodiment. Figure 2 As shown, the following steps are included:

[0111] Step 1. Obtain raw point cloud data. Use a LiDAR device in an autonomous driving scenario to collect point cloud data, sampling at a 2 Hz frequency to obtain 10,000 frames of point cloud data.

[0112] Step 2: Dataset Division: The collected point cloud data is divided into a 70% training set, a 20% validation set, and a 10% test set. Three-dimensional ground truth box annotations are performed for three types of targets: motor vehicles, pedestrians, and non-motor vehicles.

[0113] Step 3. Training set data enhancement: Perform random spatial transformation (rotation, flipping, scaling) and 3D ground truth object pasting enhancement on the training set data to increase sample diversity.

[0114] Step 4. Voxelization and Feature Encoding: The enhanced training and test dataset point clouds are pre-processed using voxelization. Sparse voxel features are extracted using the mean value method for the point set within the voxel.

[0115] Step 5. Build a fully sparse hybrid detection network. This involves constructing a hybrid network architecture consisting of a sparse convolutional encoder, a sparse attention encoder, and a dynamic sparse detection head. The sparse convolution module extracts local features, the sparse attention module models long-range feature interactions, and the dynamic detection head selects high-quality positive samples for training.

[0116] Step 6. Multi-task loss training and parameter optimization. Use FocalLoss as the classification loss function and Smooth L1 Loss as the regression loss function. Use the Adam optimizer to adaptively adjust the learning rate and iterate the training on the training set until convergence.

[0117] Step 7. Inference and post-processing: During the testing phase, the trained model is used to perform inference and prediction on the test set point cloud data, and the non-maximum suppression (NMS) method is used to filter redundant prediction boxes and output the final 3D detection box results.

[0118] In this embodiment, data augmentation expands the distribution of training samples and improves the generalization performance of the model; voxelization processing and sparse feature encoding effectively reduce the consumption of computing resources; sparse convolution combined with sparse attention mechanism enhances the long-distance feature modeling capability; dynamic sparse detection head adaptively screens positive samples, improving training efficiency and model stability; multi-task loss balances classification and regression tasks, effectively improving detection accuracy.

[0119] Example 3

[0120] This embodiment is an embodiment of a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described in the first embodiment are implemented.

[0121] Example 4

[0122] This embodiment is a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first embodiment are implemented.

[0123] In the specific contents of the above-mentioned specific implementation methods, the various technical features can be combined in any non-contradictory manner. In order to make the description concise, not all possible combinations of the above-mentioned technical features are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A three-dimensional target detection method based on a fully sparse hybrid network, characterized in that: The following steps are involved: S1. Data Processing: Acquire point cloud data, divide and preprocess the point cloud data into datasets, and divide the point cloud dataset into training dataset and test dataset. S2. Build a fully sparse hybrid detection network: The fully sparse hybrid detection network includes a voxel encoding layer for encoding point clouds in voxels, a sparse convolutional encoder for extracting local features, a sparse attention encoder for modeling long-range feature interactions, and a dynamic sparse detection head for selecting high-quality positive samples for training. S3. Input the training dataset into the fully sparse hybrid detection network and iteratively train the fully sparse hybrid detection network; S4. Use the trained model to perform inference and prediction on the test set point cloud data.

2. The three-dimensional target detection method based on a fully sparse hybrid network according to claim 1, characterized in that: In step S2, the point cloud in the voxel is encoded by finding the feature mean of the points contained in the voxel; specifically, the method includes: inputting a point set P in a non-empty voxel = {p1, p2, ..., p n }, each non-empty voxel is encoded as follows: Where p j is the voxel v i The feature of the j-th point in the voxel encoding layer is used to obtain a non-empty voxel feature set n represents the number of points in the set.

3. The three-dimensional target detection method based on a fully sparse hybrid network according to claim 2, characterized in that: In step S2, the sparse convolution encoder includes multiple processing modules, each processing module includes a three-dimensional sparse convolution layer and two residual convolution blocks, each residual convolution block includes two layers of semi-manifold sparse convolution layers, which are used to use residual connections as shortcuts to connect input and output; each convolution layer is connected to a batch normalization layer and a nonlinear activation function; each processing module downsamples and extracts features on three-dimensional voxel features, and splices the sparse feature outputs of the last three processing modules to obtain multi-scale sparse features.

4. The three-dimensional target detection method based on a fully sparse hybrid network according to claim 3, characterized in that: The splicing method of the last three processing modules includes: magnifying the voxel coordinates output by the last two processing modules by 2 times and 4 times respectively, connecting them with the voxel coordinates output by the third-to-last processing module, and then splicing the feature outputs of the three processing modules using the feature sequence splicing method to obtain multi-scale sparse features. Where d is the number of feature channels, n4, n5 and n6 are the number of voxel features output by the third to last processing module, the second to last processing module and the last processing module respectively; finally, the multi-scale sparse features are compressed along the high dimension to obtain the multi-scale BEV sparse features 5. The three-dimensional target detection method based on a fully sparse hybrid network according to claim 1, characterized in that: The sparse attention encoder includes a multi-layer strip window-based deep learning model; The strip window covers the entire point cloud scene in a certain direction, and the directions of the strip windows of adjacent deep learning model layers alternate with each other, thereby providing a receptive field covering the entire scene.

6. The three-dimensional target detection method based on a fully sparse hybrid network according to claim 5, characterized in that: The sparse attention coding layer includes the following steps when processing data: Input N non-empty voxels in and Represent voxel ν i coordinates and characteristics of; Divide the above N non-empty voxels into corresponding strip windows, and the size of each strip window is s×w, where s and w are the size of the point cloud scene and the width of the strip window respectively: Where, and Represent voxel ν i Bar window index in the X and Y directions; Calculate the Query, Key, and Value matrices of the linear attention layer respectively: q i =φ(f i W q ),k i =φ(f i W k ),v i =f i W v , Where W q 、W k and W v Represent the learnable weight matrix, φ(·) represents the ReLU nonlinear activation function, and the Query matrix is ​​obtained. Key Matrix and Value matrix c represents the feature dimension of the vector, and N represents the number of non-empty voxels in the input; Next, calculate the X-axis direction strip window matrix: Where, Indicates whether the i-th voxel belongs to the j-th strip window; m represents the number of divided windows; Use the KV matrix to calculate the new Value matrix: Where, Indicates the The kv matrix of a strip window, the denominator is a normalized term; The calculation process for the Y-axis direction is the same as that for the X-axis direction; Finally, the new Value matrix passes through a feedforward network to obtain the updated features of the sparse voxels: f i ′=FFN(v i ′), FFN(·) represents the feedforward network module.

7. The three-dimensional target detection method based on a fully sparse hybrid network according to claim 6, characterized in that: The dynamic sparse detection head is used to generate high-quality positive samples for model training and output the final prediction box; in the training phase, first find the n voxels closest to the center of the target box for the real box Then calculate the selection cost of the voxel: Where Pred(·) represents a prediction module composed of multiple convolutional layers, which generates prediction boxes and classification values ​​based on the input voxel features; cls (·) and l reg (·) represents classification loss and regression loss respectively; b t Represents the true target box in the current training sample; Next, calculate the intersection-over-union ratio between the n predicted boxes and the true box and add them together: Where, IoU(·) represents the calculation of intersection over union; Finally, the top k voxel optimal positive samples with the smallest selection cost are selected for model training optimization.

8. The three-dimensional target detection method based on a fully sparse hybrid network according to claim 7, characterized in that: Step S3 includes: S31. Input the point cloud data of the training dataset in step S1 into the fully sparse hybrid detection network, and use the true box and the predicted box as inputs to the loss function; use the focal loss as the classification loss function and the smooth L1 loss as the box regression loss function; use the Adam optimization algorithm to update the parameters of the fully sparse hybrid detection network; S32. Iterate the network training epochs on the training dataset until the overall loss function converges. The losses for different tasks and the overall loss are calculated as follows: l=l cls +λ b L b Where, l cls and l b Represents classification loss and box regression loss respectively, N c and N p Represents the number of classification predictions and the number of positive anchor boxes, c i and Represents the target category true value and target category predicted value at the i-th position, λ b and λ d They represent the weight of the box regression loss and the weight of the target orientation loss, α represents the category balance weight, β represents the difficult sample adjustment index, and b ij 、 Represents the true value and predicted value of the j-th regression item of the i-th box.

9. The three-dimensional target detection method based on a fully sparse hybrid network according to any one of claims 1 to 8, characterized in that: Step S1 includes: S11. Use a lidar to record a point cloud dataset and sample the recorded point cloud data at a frequency of Y to obtain X frames of point cloud data. Then, annotate the three types of objects in the point cloud with 3D ground truth boxes: motor vehicles, pedestrians, and non-motor vehicles. Then, divide the X frames of point cloud data into a training dataset and a test dataset at a 1:1 ratio. S12. Perform 3D real-world frame pasting data augmentation; randomly select n 3D real-world frames and the target point clouds contained in the real-world frames from the training data set, and then perform collision detection. If the selected 3D real-world frame overlaps with the existing 3D real-world frame in the current point cloud sample, it is considered that a collision has occurred. Finally, the target point clouds contained in the 3D real-world frames that have not collided are pasted. Paste copied to current point cloud Get point cloud Among them, N0 and N s Represents the number of points in the original point cloud P0 and the single target point cloud P s The number of points; S13. Perform random flip data enhancement to randomly flip the point cloud P1 along the x-axis or y-axis; S14. Perform random rotation data augmentation, randomly generate a rotation angle θ1 in the interval [-π / 4, π / 4], and rotate the point cloud P1 around the z-axis by an angle θ1; S15. Perform random scaling data enhancement on the point cloud; randomly generate a scale factor s1 in the interval [0.95, 1.05], and multiply the overall coordinate value of the point cloud P1 by the scale factor s1; S16. shuffle the order of the points in point cloud P1; S17. Filter out points and 3D ground truth boxes outside the detection range in point cloud P1; S18. Voxelize the point cloud P1 and define the voxel size as s x ×s y ×s z , divide each point in the point cloud into corresponding voxels; s x 、s y 、s z Represents the size of the voxel in the X, Y, and Z dimensions respectively.

10. The three-dimensional target detection method based on a fully sparse hybrid network according to claim 9, characterized in that: In step S4, the trained fully sparse hybrid detection network is used to perform inference prediction on the point cloud in the test dataset, and the non-maximum suppression method is used to filter redundant prediction boxes to output the final three-dimensional detection prediction result.

Images