A three-dimensional dynamic target detection method and device based on voxel point cloud fusion

Abstract

The present invention proposes a three-dimensional dynamic target detection method and device based on voxel point cloud fusion, which adopts a two-stage multiple and multi-scale feature fusion detection architecture. In the first stage, the point cloud is directly processed to extract key point features and the voxel space is divided to extract multi-scale voxel features, and the two features are initially fused to generate a pre-selection box. In the second stage, a reference point is set in each voxel and the surrounding key points are absorbed for a second feature fusion. The final features are input into the detection module to optimize the pre-selection box. In addition, in order to solve the problem of inconsistent classification and positioning reliability, a forced consistency loss function is proposed to further improve the accuracy of detection. The algorithm of the present invention is compared with other algorithms in Kitti, Waymo, and Nuscene data sets, and portability and ablation studies are carried out through physical vehicle platforms. The results show that the algorithm of the present invention has high accuracy for three-dimensional dynamic target detection and strong robustness, portability and generalization capabilities.

Description

Technical Field

[0001] The present invention belongs to computer technology, and specifically relates to a three-dimensional dynamic target detection method, and in particular to a three-dimensional dynamic target detection method and device based on voxel point cloud fusion. Background Art

[0002] In recent years, the target detection algorithm of autonomous driving in three-dimensional space has attracted widespread attention from academia and industry. Three-dimensional target detection mainly uses cameras and lidar as sensors for collecting data. Van et al. comprehensively analyzed and studied the advantages and disadvantages of various sensors in the field of three-dimensional target detection and provided ideas for multi-sensor fusion. Li et al. proposed a candidate region generation module method based on structural perception for the application of target detection in the field of autonomous driving, which provides a theoretical basis for the evaluation of the algorithm using a physical platform in this invention.

[0003] Three-dimensional dynamic multi-target detection algorithms can be mainly divided into two types: image-based methods and point cloud-based methods. Among them, image-based methods have always been the focus of exploration in traditional three-dimensional target detection, and can be mainly divided into three types: monocular, binocular, and depth camera. Image-based detection methods are essentially to segment images and obtain dense features for matching three-dimensional targets. Ku et al. used a monocular camera to generate 3D suggestions for targets in the scene through a 2D target detection method, and designed a projection alignment method to optimize the 3D positioning problem. For binocular cameras, Wang proposed a three-dimensional target matching algorithm that integrates multi-scale local features and depth features and converted the depth map based on binocular images into a radar point cloud form, using a 3D point cloud algorithm for training and detection. However, both monocular and binocular cameras lack direct depth information, and the depth information obtained by geometric calculation based on visual images has errors, making it difficult to generate an accurate 3D bounding box. Depth cameras use the time of flight (ToF) of the emitted and received modulated infrared pulses to infer depth. At present, Elkhalili et al. have proved that depth cameras have been successfully used in safe driving. However, camera sensors are easily affected by light and weather conditions. Chen et al. used depth cameras and proposed a method based on multi-scale pyramid structure and conditional generative adversarial network (GAN) to solve the above problems. With the reduction in the cost of lidar sensors, more and more algorithms have been developed in recent years to process point cloud data, and three main categories have been derived: direct point cloud processing method, voxel space division method, and mapping method. The pioneer of the method of directly processing point clouds is a framework called Pointnet proposed by Qi et al., which can directly input disordered point clouds into the system to perform target pose correction, feature extraction, confidence prediction, classification, etc. Although the method of directly processing point clouds can obtain a larger receptive field through feature extraction, the computational complexity is high. The method of dividing voxels is also a common idea in point cloud target detection. Zhou et al. pioneered the VoxelNet framework, which divides point clouds into voxels, stacks voxel features of different scales, extracts more complex feature information, and generates detection boxes in three-dimensional space. Although the voxel-based method is more efficient in calculation, it is limited by the size of the convolution kernel and inevitably reduces the positioning accuracy. Finally, there is the mapping method. Lang et al. encode point clouds and map them into pseudo-two-dimensional data, and use mature 2D CNN methods for detection. For example, Li et al. proposed a real-time target detection method based on an improved Faster R-CNN network, which avoids time-consuming 3D convolution operations. However, the mapping process will inevitably lead to the loss of feature information. The above point cloud-based processing methods all have their own advantages and disadvantages, so it is inevitable to consider the fusion of multiple features.The fusion of point cloud data and binocular view can improve the accuracy of target detection. Through iterative self-learning 3D target detection method, the accuracy in the target area can be improved. Tang et al. proposed a point cloud matching algorithm. Qi et al. proposed a cone method called F-PointNet, which fuses camera images with point cloud data to determine the target area in the point cloud. However, this method requires additional annotation of camera images, and its detection effect will also be affected by 2D detection. Summary of the invention

[0004] In view of the problem that the methods currently used in the field of three-dimensional target detection generally have insufficient target context features in feature extraction and cannot achieve accurate dynamic multi-target detection, the present invention proposes a three-dimensional dynamic target detection algorithm based on voxel point cloud fusion from the point cloud. The proposed algorithm was tested and evaluated on Kitti, Waymo, Nuscene datasets and physical vehicle platforms. Experiments show that the algorithm performance has been significantly improved by combining multi-scale features. In addition, considering that the inconsistency between classification confidence and positioning confidence will lead to inaccurate target detection. For example, there are two pre-selected boxes A and B, pre-selected box A has a higher classification confidence and a lower positioning confidence, and pre-selected box B has a higher positioning confidence and a slightly lower classification confidence than box A. The non-maximum suppression (NMS) method will automatically filter out boxes with lower confidence. Therefore, the present invention proposes a mandatory consistent loss function that can retain pre-selected boxes that are closer to the true value, and the loss function does not require changes to the framework of target detection, is easy to implement, and does not require too much additional computational cost.

[0005] The present invention adopts the following technical solution:

[0006] A three-dimensional dynamic target detection method based on voxel point cloud fusion, after collecting the three-dimensional dynamic target, it goes through the first stage and the second stage, and then the point cloud outputs the three-dimensional dynamic target to complete the three-dimensional dynamic target detection; specifically, after collecting the three-dimensional dynamic target, the point cloud is input; the first stage is to extract key point features, voxel features, feature fusion, and pre-selection box generation from the point cloud; the second stage is to optimize the pre-selection box and calculate the loss function. More specifically, in the first stage, by dividing the voxels for 3D convolution operation, the submanifold sparse convolution method is used as the backbone network for feature encoding and pre-selection box generation, and the FPS iterative farthest point sampling method is used to extract key points from the point cloud and cluster them, extract the features of the surrounding area, and finally fuse the features of the key points in the corresponding voxel space with the features of the surrounding area of ​​the key points to obtain the high-level features of the key points in the point cloud space; in the second stage, reference points are selected in the ROI area generated based on voxels, clustered, and the key point features generated in the first stage are fused to optimize the pre-selection box, and the loss function is calculated through the confidence-enforced constraint.

[0007] In the present invention, in the first stage, the key point features are extracted from the point cloud: key point sampling layer, clustering layer, feature extraction layer, foreground point segmentation layer;

[0008] (1) Sampling layer: In the past, random sampling methods were difficult to obtain global information of point clouds. This invention introduces the FPS (Iterative Farthest Point Sampling) method to sample key points from point clouds. The specific principle is as follows: Input point cloud set Total n Points, randomly selected As the starting point, and write a collection of key points ,calculate and the remaining n-1 The Euclidean distance of points, choose the farthest point Write Collection The next iteration considers both points at the same time and calculates the remaining n-2 Point to Set K The distance between two points in the set is taken as the shortest distance from the point to the point set, and we get n-2 Point to Set K distance, select the farthest point Write Collection , continue to iterate the above process until the required number of points are sampled;

[0009] (2) Clustering layer: Then, multi-resolution grouping (MRG) is used to cluster the key points. R Divide the spherical area by radius, extract the information of the points contained in the area, and obtain the features associated with the target; the extracted features are composed of two feature vectors in series, the first half is composed of feature extraction from all points in the original point cloud data, and the second half is obtained by inputting the features of the first half into the PointNet network;

[0010] (3) Feature extraction layer: Use the PointNet network to perform convolution and pooling operations on each local feature obtained in the clustering layer, and use the obtained multi-resolution features as the local features of the center point to obtain features with consistent dimensions;

[0011] (4) Foreground point segmentation layer: The key point prediction weighting layer is used to detect the key points obtained by the feature extraction layer and screen out the foreground points. The specific principle is to input the key point features into a two-layer perceptron network (MLP) to obtain the weight information of the foreground points. The first layer of MLP (128, 256) network has two convolution processes. The first convolution process is to use 128 1×128×1 convolution kernels to convolve with the input features to obtain n 1×128 feature vectors. The second convolution process is to use 256 1×1×128 convolution kernels to convolve and obtain n 1×256 feature vectors. The convolution process is then transformed into The feature vector becomes n×256; the second-layer MLP (128, 1) network has two convolution processes. The first convolution uses 128 1×256×1 convolution kernels to convolve with the input features to obtain n 1×128 feature vectors. The second convolution uses 1 1×1×128 convolution kernel to convolve to obtain n 1×1 feature vectors. Finally, the foreground point prediction value is converted to n×1 through dimensionality transformation. When processing point cloud data containing multiple points, each point is multiplied by the same weight, and the weight range is between (0, 1). Finally, it is multiplied with the feature vector to obtain the final feature of the key point. Each key point feature The prediction feature weights are expressed as follows

[0012] (1)

[0013] in It is a two-layer MLP network that predicts the confidence of foreground points between (0, 1). Its calculation formula is divided into two parts: forward propagation and weight update. The forward propagation formula is as follows:

[0014] (2)

[0015] In the forward propagation formula, is the hidden layer network weight, the activation function is the softmax function, is the output layer network weight, output label One-Hot encoding is used, and the loss function uses the focal loss function. The gradient calculation and weight update formulas for the output layer and hidden layer are as follows:

[0016]

[0017] in is the result of loss function calculation, is the gradient descent weight.

[0018] In the present invention, in the first stage, voxel feature extraction consists of three parts: voxel partitioning layer, VFE feature extraction layer, and submanifold sparse convolution layer;

[0019] (1) Voxel partitioning layer divides the voxel space. The 3D space is subdivided into voxels of equal size and spacing. The point cloud is defined along the Z, Y, X The range of the axis is D, H, W 3D space, defining the size of each voxel as , then the number of divisions along each coordinate axis is , the entire point cloud space is divided into N voxels of equal size.

[0020]

[0021] (2) VFE feature extraction layer uses the VFE layer to encode the voxels. non-empty voxels of points, use Indicates that Contains voxels i Points X, Y, Z Coordinates and reflectivity , first calculate the voxel V The local mean in is taken as the centroid, expressed as ,calculate The offset relative to the center of mass is expressed as , we can get the input feature set ; Next, use the fully connected layer (FCN) to transform the input 7-dimensional vector The map is 32-dimensional, and each voxel has a total of t A 32-dimensional feature vector is obtained by using the maximum pooling operation to obtain the global variable of the voxel, which is then concatenated with the feature of each point. At this time, the feature of each point is 64-dimensional. Repeat the above process again to obtain the voxel t A 128-dimensional feature vector is generated, and finally the 128-dimensional global feature of the voxel is obtained through maximum pooling;

[0022] (3) If there is no relevant input point in the point cloud space, the submanifold sparse convolution layer will not calculate the output point; a hash table is constructed to store the positions with values ​​in the feature map. Suppose the input feature map is , where the non-zero elements are a , will Encoded as a The characteristic matrix and a hash table , Storage All non-zero eigenvectors in , m represents the dimension of the feature vector, It is a hash table consisting of key-value pairs. Key refers to the row index of the feature matrix, and its value is an integer between [0, a-1]. Value indicates the corresponding element in The index in is a d dimensional vector, sparse feature map output The same code is and , . yes The matrix of n is the dimension of the output feature vector; the size of the convolution kernel and the convolution step size are defined as follows:

[0023]

[0024] in refers to the dimension of the input, refers to the dimension of the output, and Satisfy the constraints:

[0025]

[0026] in f represents the size of the convolution kernel, s represents the convolution step size, p Represents zero value filling;

[0027] The submanifold sparse convolution layer is divided into two parts: zero padding of null values ​​and forced clearing; yes d dimensional tensor, defining the size of the receptive field as (Refers to a side length of f of d dimensional cube), with An element in Centered around All the empty value positions in the space are filled with zeros, and then convolution is performed with the convolution kernel; let The zero value area is ,Will The corresponding zero value area in is set to zero;

[0028] After the submanifold sparse convolution layer feature extraction, the meaning of the three-dimensional convolution parameters is , k is the convolution kernel size, s is the step length, p Fill with zero values, convert the 4-fold downsampled 3D feature map into a 2D bird's-eye view feature map, and use an anchor-based method to generate a high-quality 3D pre-selection box. Specifically, along ZThe feature quantities are stacked in the axis to obtain a four-fold downsampled bird's-eye view, that is, 400×352 voxels on the XY plane, each voxel is represented by a 32-dimensional feature vector. The input box generation network, that is, the pre-selected box is generated according to the generated regression map and probability score map, and each voxel in the bird's-eye view is evaluated to generate two pre-selected anchor boxes, which are 0° and 90° directions, respectively, to prepare for the second stage of box optimization.

[0029] In the present invention, in the first stage, feature fusion fuses the features obtained above; define the submanifold sparse convolution layer k The set of voxel feature vectors at level , the corresponding 3D coordinate set is , which is calculated from the voxel coordinate index and the actual voxel size in 3D space, where Representative k The number of non-empty voxels in the level voxel feature. As the center, Detect adjacent non-empty voxels for the radius and use the collection express:

[0030]

[0031] in is the local relative coordinate, used to correspond to the voxel feature The relative position in the image is then normalized to obtain the non-empty voxel sets at different scales. Features:

[0032]

[0033] in Represents from the set Random sampling is the most T The calculation is performed on each voxel, and Represents a multi-layer perceptron network, which is used to encode voxel features and relative positions. Through the maximum pooling operation The feature vectors with different numbers of neighboring voxels are mapped to , the features obtained from convolution operations of different scales can be cascaded to obtain Multi-scale features:

[0034]

[0035] The original point cloud features can be obtained by formula (9): and Bird's Eye View Features , further get the key points Features:

[0036]

[0037] The structural information of the entire 3D scene can be retained very effectively, which is beneficial for subsequent detection.

[0038] In the present invention, in the first stage, the pre-selection box generates a probability score map and a regression map by analyzing the bird's-eye view. The probability score map represents the score of the feature. Each point in the bird's-eye view will generate a pre-selection box at two angles in the regression map. Each box has seven parameters. , representing the X , Y , Z The axis coordinates and the box height, width, length, and yaw information.

[0039] In the present invention, in the second stage, the pre-selected box optimization extracts and fuses the features of each ROI from the key point features to generate an accurate and reliable 3D detection box (ROI). Specifically, for each ROI in the 3D space, 6 points are equally spaced along the X, Y, and Z axes, that is, 6×6×6 reference points are uniformly generated in the space, which is expressed as , centered at the reference point, r The features of the key points around the radius aggregation, the feature set of its adjacent key points is as follows:

[0040]

[0041] in For key points Relative to reference point The local relative position of , and then aggregate the adjacent key point feature sets:

[0042]

[0043] in and Same as the definition in formula (9);

[0044] Get key points in the ROI area to aggregate features Then select 0°, 30°, 60°, 90°, 120°, 150°, and generate pre-selected boxes. Then input the box optimization network to calculate the residual of the box center, size, and direction.

[0045] The box optimization network consists of two MLP layers. The network has two branches, one for confidence prediction and the other for pre-selected box optimization. k ROIs are normalized to obtain classification confidence as follows:

[0046]

[0047] in The value range of is [0, 1], It is k The interaction ratio of each ROI and the real data label is then used to train the confidence branch in combination with the mandatory consistent loss; the mandatory consistent loss is as follows:

[0048]

[0049] in D and G Represents the pre-selected box and the real data label, Represents the positioning confidence of the box.

[0050] In the present invention, in the second stage, the loss function consists of three parts: pre-selection box loss , key point extraction loss , box optimization loss ;

[0051] (1) The loss of the pre-selected box is as follows:

[0052]

[0053] in represents the prediction residual, represents the regression target, and the smooth_L1 loss function is used to perform regression calculations on the two. For classification loss, the focal loss function is used to balance positive and negative samples, and its calculation formula is as follows:

[0054]

[0055] The balance factor , used to balance the uneven ratio of positive and negative samples, factor , used to reduce the loss of easy-to-classify samples, is the output probability of belonging to the true value:

[0056] (2) Key point extraction loss It is the focal loss function for foreground point segmentation. The foreground points provide rich information for predicting the target position and direction. The key points obtained by the feature extraction layer are detected through a key point prediction weighted layer (PKW) to screen out the foreground points. In the training stage, a function based on focal loss is used in combination with the ground truth value, that is, by checking whether each key point is inside or outside the ground truth 3D box, to deal with the imbalance problem between foreground points and background points in the training set.

[0057] (3) The box optimization loss is calculated as follows:

[0058]

[0059] in is a mandatory consistent loss function, is the prediction residual, is the regression target, and The encoding method is the same; the total loss function of the present invention is the sum of the three loss functions with the same weights.

[0060] The present invention discloses a device for three-dimensional dynamic target detection based on voxel point cloud fusion, which is used for the above-mentioned three-dimensional dynamic target detection method based on voxel point cloud fusion, and includes an image acquisition module and a data processing module. The data processing module includes a point cloud input module, a key point feature extraction module for point cloud, a voxel feature extraction module, a feature fusion module, a pre-selection box generation module, a pre-selection box optimization module, a loss function calculation module and a point cloud output module; the image acquisition module is a radar sensor and a camera.

[0061] The present invention discloses an electronic device, comprising a processor and a memory, wherein the memory stores instructions executed by the processor, so that the processor can execute the three-dimensional dynamic target detection method based on voxel point cloud fusion.

[0062] The present invention discloses a computer-readable storage medium storing computer instructions, wherein the computer instructions enable a computer to execute the three-dimensional dynamic target detection method based on voxel point cloud fusion.

[0063] The innovative features of the present invention are as follows:

[0064] (1) Combining the advantages of the flexible receptive field of the direct point cloud processing method and the high computational efficiency of the spatial voxel division method, a two-stage 3D object detection algorithm is proposed. The pre-selected box is generated in the first stage and the box is optimized by two fusions in the second stage, which can obtain rich point cloud features.

[0065] (2) In the process of extracting voxel space features, the submanifold sparse convolution method is introduced to encode the non-empty voxel space and extract high-dimensional features. The FPS iterative farthest point sampling method is also introduced to make up for the defect that the divided voxels cannot describe the entire space. The two are combined to obtain high-level features of the global space.

[0066] (3) A mandatory consistent loss function is proposed to constrain the classification confidence and localization confidence. This loss function is easy to implement and does not require modifying the target detection framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is the system block diagram;

[0068] Figure 2 It is a point-based feature extraction process;

[0069] Figure 3 It is the foreground point segmentation (PKW) module;

[0070] Figure 4 It is the submanifold sparse convolution feature extraction layer;

[0071] Figure 5 Optimize the network for the box;

[0072] Figure 6 mAP comparison of various methods for BEV bird's eye view;

[0073] Figure 7 is the Recall-Precision curve;

[0074] Figure 8 Visualize the detection results of point cloud data;

[0075] Fig. 9 For ablation study based on Kitti dataset;

[0076] Fig.10 Loss function curves: (a) loss curve of each module, (b) total loss curve;

[0077] Fig.11 The accuracy detection curves are: (a) the detection accuracy of each category of targets of the three methods; (b) the detection accuracy of targets of the three methods at different distances;

[0078] Fig.12 To visualize the detection results. DETAILED DESCRIPTION

[0079] The present invention proposes a three-dimensional dynamic target detection algorithm framework based on voxel point cloud fusion, which is a voxel point cloud fusion algorithm framework. Figure 1 As shown, the first stage of the framework performs 3D convolution operations by dividing voxels, and in the traditional VoxelNet

[10] Based on this, we innovated and introduced submanifold sparse convolution

[19] The method is used as the backbone network for feature encoding and pre-selection box generation. At the same time, the FPS iterative farthest point sampling method is used to extract key points from the point cloud and cluster them, and the features of the surrounding area are extracted. Finally, the features of the key points in the corresponding voxel space are fused with the features of the area around the key points to obtain high-level features of the key points in the point cloud space. In the second stage, reference points are selected in the ROI area generated based on voxels, clustered within a certain range, and the key point features generated in the first stage are fused to optimize the pre-selection box. A more accurate detection box generation and position optimization effect are achieved through a loss function with confidence enforced constraints. Based on the method disclosed in the present invention, the specific steps and operations are prior art. Example

[0080] The sensor is used to collect data on the physical object, and the point cloud file is obtained by conventional methods, and then the following steps are performed.

[0081] Key point feature extraction: Directly extracting key points from the point cloud of the detected object is mainly divided into four steps: key point sampling layer, clustering layer, feature extraction layer, and foreground point segmentation layer.

[0082] (1) Sampling layer: In the past, random sampling methods were difficult to obtain global information of point clouds. Therefore, the FPS (Iterative Farthest Point Sampling) method was introduced to sample a small number of key points from the point cloud. The specific principle is as follows: Input point cloud set Total n Points, randomly selected As the starting point, and write a collection of key points , calculate and the remaining n-1 The Euclidean distance of points, choose the farthest point Write Collection The next iteration needs to consider both points at the same time, so the remaining n-2 Point to Set K The distance between two points in the set is taken as the shortest distance from the point to the point set, and we get n-2 Point to Set K distance, select the farthest point Write Collection , continue to iterate the above process until the required number of points is sampled. The number of key points sampled in the point cloud space of the Kitti dataset and the Nuscene dataset is 2048, and the number of key points sampled in the Waymo dataset is 4096.

[0083] (2) Clustering layer: Then, with the key point as the center, RThe spherical region is divided into radii, and the information of the points contained in the region is extracted. This method can obtain the features associated with the target. Since the point cloud is unevenly distributed in space, if each region uses the same spherical radius, there will be too few sampling points in some sparse areas, resulting in insufficient feature information. Therefore, the present invention introduces a multi-resolution combination (MRG) method, such as Figure 2 As shown in the figure, the extracted features are composed of two feature vectors in series. The first half is composed of feature extraction from all points in the original point cloud data, and the second half is obtained by inputting the features of the first half into the PointNet network.

[0084] (3) Feature extraction layer: Use the PointNet network to perform convolution and pooling operations on the local features obtained by the clustering layer. The obtained multi-resolution features are used as the local features of the center point. Although the local features given by the clustering layer may be composed of different numbers of points, the PointNet network can obtain features with consistent dimensions.

[0085] (4) Foreground point segmentation layer: The foreground points provide rich information for predicting the position and direction of the target. The present invention proposes a key point prediction weighting layer (PKW), such as Figure 3 As shown in the figure, the key points obtained by the feature extraction layer are detected to screen out the foreground points. The specific principle is to input the key point features into a two-layer perceptron network (MLP) to obtain the weight information of the foreground points. The first layer of MLP (128, 256) network has two convolution processes. The first convolution process is to use 128 1×128×1 convolution kernels with the input features to obtain n 1×128 feature vectors. The second convolution process uses 256 1×1×128 convolution kernels to obtain n 1×256 feature vectors. The feature vectors are converted to n×256 through dimensional transformation. The second-layer MLP (128, 1) network also has two convolution processes. The first convolution uses 128 1×256×1 convolution kernels to convolve with the input features to obtain n 1×128 feature vectors. The second convolution uses a 1×1×128 convolution kernel to convolve and obtain n 1×1 feature vectors. The purpose of this process is to obtain the possibility of whether each key point belongs to the foreground point, and finally the foreground point prediction value is converted to n×1 through dimensional transformation. Figure 3In the above, Shared means that when processing point cloud data containing multiple points, each point is multiplied by the same weight, because in the point cloud, each point is not independent and should not be regarded as an independent sample. In order to simplify the calculation, the weight range is between (0, 1), and finally multiplied with the feature vector to obtain the final feature of the key point. In the training stage, a function based on focal loss is used in combination with the real data label, that is, by checking whether each key point is inside or outside the 3D box of the real data label, to deal with the imbalance between the foreground and background points in the training set. Each key point feature The prediction feature weights can be expressed as:

[0086] (1)

[0087] in It is a two-layer MLP network to predict the confidence of the foreground point between (0, 1). Its calculation formula is divided into two parts: forward propagation and weight update:

[0088] (2)

[0089] In the forward propagation formula, is the hidden layer network weight, the activation function is the softmax function, is the output layer network weight, output label One-Hot encoding is used. The loss function uses the focal loss function. The gradient calculation and weight update formulas for the output layer and hidden layer are:

[0090]

[0091] in is the result of loss function calculation, is the gradient descent weight, which is performed during the training process.

[0092] Voxel feature extraction: The advantage of the voxel-based feature extraction method is that it has high computational efficiency and can obtain more accurate upper and lower information of the target. This method consists of three parts: voxel division layer, VFE feature extraction layer, and submanifold sparse convolution layer.

[0093] (1) Dividing the voxel space

[0094] Subdivide the 3D space into equally spaced voxels, assuming that the point cloud is along Z, Y, X The range of the axis is D, H, W 3D space, defining the size of each voxel as , then the number of divisions along each coordinate axis is , the entire point cloud space is divided into N voxels of equal size.

[0095]

[0096] Taking Kitti's 3D point cloud dataset as an example, for vehicle detection, , the size of each voxel is , pedestrian or cycling target detection, then , the size of each voxel is .

[0097] When the LiDAR is working, the point cloud is unevenly distributed in the entire 3D space. After dividing the voxel space, the number of points contained in each voxel is different. Usually, the point cloud of a high-resolution LiDAR consists of more than 100K points. Directly processing all the points will not only increase the computational cost, but also may cause errors in detection due to uneven point density. Therefore, a threshold is introduced T , for more than T The voxels are randomly sampled to select a fixed T The purpose of using this method is, on the one hand, to reduce the amount of calculation, and on the other hand, to reduce the imbalance of the number of points between voxels and reduce the sampling deviation. For example, when detecting a vehicle T=35 , when detecting pedestrians T=45 .

[0098] (2) VFE feature extraction layer

[0099] like Figure 1 The voxel-based method in describes the process of encoding voxels using VFE layers. non-empty voxels of points, using Indicates that Contains voxels i Points X, Y, Z Coordinates and reflectivity , first calculate the voxel V The local mean in is taken as the centroid, expressed as .calculate The offset relative to the center of mass is expressed as , we can get the input feature set In order to avoid information loss, the fully connected layer (FCN) is used to transform the input 7-dimensional vector The map is 32-dimensional, so each voxel now has a total of t A 32-dimensional feature vector is obtained by using the maximum pooling operation to obtain the global variable of the voxel, which is then concatenated with the feature of each point. At this time, the feature of each point is 64-dimensional. Repeat the above process again to obtain the voxel tThe 128-dimensional feature vector is then used to obtain the 128-dimensional global feature of the voxel. Therefore, the feature dimension of the entire point cloud space is (128, 10, 400, 352). The size of the Kitti dataset point cloud space and the size of each voxel can be used to calculate that the space is divided into . That is 10×400×352 voxels.

[0100] (3) Submanifold 3D sparse convolution

[0101] The main principle of spatial sparse convolution is that if there is no relevant input point in the point cloud space, the output point will not be calculated. For example, the Kitti dataset divides the point cloud into 5k-8k voxels, and its sparsity is close to 0.005. The calculation cost is very high when using ordinary convolution. Therefore, the submanifold sparse convolution method is introduced. Only when the relevant position of the input point cloud is in an activated state, the corresponding position of the output point cloud will be activated. This method avoids the generation of too many activated positions, which leads to a decrease in speed in the subsequent convolution process.

[0102] The input and output of 3D sparse convolution are both sparse feature maps, so there is no need to store all the values ​​in the feature map indiscriminately. You only need to build a hash table to store the locations with values ​​in the feature map. Suppose the input feature map is , where the non-zero elements are a , will Encoded as a The characteristic matrix and a hash table . Storage All non-zero eigenvectors in , m represents the dimension of the feature vector, It is a hash table consisting of key-value pairs. Key refers to the row index of the feature matrix, and its value is an integer between [0, a-1]. Value indicates the corresponding element in The index in is a d dimensional vector, sparse feature map output The same code is and , considering that the sparse convolution process will maintain the sparsity of the input feature map, . yes The matrix of n is the dimension of the output feature vector.

[0103] Sparse convolution is the same as ordinary convolution. It is necessary to define parameters such as the size of the convolution kernel and the convolution step size.

[0104]

[0105] in refers to the dimension of the input, refers to the dimension of the output, and Satisfy the constraints:

[0106]

[0107] in f represents the size of the convolution kernel, s represents the convolution step size, p Represents zero-value filling.

[0108] Submanifold 3D sparse convolution is mainly divided into two parts: zero padding and forced zero clearing. Zero padding is to maintain normal convolution calculation, and forced zero clearing is to maintain the sparsity of features. yes d dimensional tensor, the size of the receptive field is (Refers to a side length of f of d dimensional cube). Sparse convolution will be An element in Centered around All empty values ​​in the space are filled with zeros, and then convolution is performed with the convolution kernel. In order to keep the input size and output size consistent, it is necessary to Zero value is added on top. Due to the influence of the receptive field area, the position with zero value will get non-zero value result after convolution, which will cause feature distortion. In order to maintain the sparsity of the original feature, it is assumed The zero value area is ,Will The corresponding zero value areas in are set to zero.

[0109] By introducing submanifold sparse convolution, the convolution calculation of the zero-value area can be ignored, then d The computational complexity of sparse convolution is in a is a non-zero value within the receptive field, m and n are the feature dimensions of input and output respectively, while traditional convolution requires Compared with the two, the introduction of submanifold sparse convolution greatly reduces the amount of operations.

[0110] like Figure 4 The following is a submanifold sparse convolution feature extraction layer, where the white box represents conventional convolution, the yellow box represents submanifold convolution, and the red box represents dimensional reconstruction. The meaning of the three-dimensional convolution parameters is , k is the convolution kernel size,s is the step length, p Fill with zero values, that is, by converting the 4-fold downsampled 3D feature map into a 2D bird's-eye view feature map, a high-quality 3D pre-selection box can be generated using an anchor-based method. Specifically, along Z The feature quantities are stacked on the axis to obtain a four-fold downsampled bird's-eye view, that is, there are 400×352 voxels on the XY plane, and each voxel is represented by a 32-dimensional feature vector. The input box generation network, that is, the pre-selected box is generated according to the generated regression map and probability scoremap, and each voxel in the bird's-eye view is evaluated to generate two pre-selected anchor boxes, which are 0° and 90° directions respectively. Since the features at this time are not rich enough, the pre-selected box only uses two angles to determine the approximate position of the target, in preparation for the second stage of box optimization.

[0111] Feature fusion algorithm. The features obtained above need to be fused. The feature extraction layer performs multiple convolutions on the feature map to obtain 1x, 2x, and 4x downsampled features. The set of voxel feature vectors at the kth level of 3D convolution is defined as , the corresponding 3D coordinate set is , is calculated from the voxel coordinate index and the actual voxel size in 3D space, where Represents the number of non-empty voxels in the k-th level voxel feature. As the center, Detect adjacent non-empty voxels for the radius and use the collection express.

[0112]

[0113] in is the local relative coordinate, used to correspond to the voxel feature The relative position in the image is then normalized to obtain the non-empty voxel sets at different scales. characteristics.

[0114]

[0115] in Represents from the set Random sampling is the most T The calculation is performed on each voxel, and Represents a multi-layer perceptron network, which is used to encode voxel features and relative positions. Through the maximum pooling operation The feature vectors with different numbers of neighboring voxels are mapped to , the features obtained from convolution operations of different scales can be cascaded to obtain multi-scale features.

[0116]

[0117] The purpose of this process is to focus on key points. The non-empty voxel features of different scales within the range are aggregated. Due to the different feature dimensions, a multi-layer perceptron is used for convolution operation to unify the feature dimension, that is, 128 dimensions. Since the number of non-empty voxels around different key points is different, a maximum pooling operation is used to obtain dimensional features. The above operations are followed for 1x, 2x, and 4x cases, and we get Dimensional features.

[0118] In order to obtain richer features, considering the 2D bird's-eye view Z The axis has a large receptive field and can obtain key points through bilinear interpolation Features mapped to the bird's-eye view, and The corresponding features in the original point cloud can make up for the loss in the voxel quantization process. The original point cloud features can be obtained by formula (9): and Bird's Eye View Features .

[0119] To sum up, the key points are characteristics.

[0120]

[0121] The structural information of the entire 3D scene can be retained very effectively, which is beneficial for subsequent detection.

[0122] Pre-selected box generation and box optimization network

[0123] The generation of the pre-selected box is to score the feature information of the bird's-eye view. Each point in the bird's-eye view represents the original 3D space. Z Therefore, the bird's-eye view is analyzed and processed to obtain a probability score map and a regression map. The probability score map represents the score of the feature. Each point in the bird's-eye view will generate two angle pre-selected boxes in the regression map. Each box has seven parameters. , representing the X, Y, Z Axis coordinates and box height, width, length, and yaw information.

[0124] In order to generate an accurate and reliable 3D detection frame (ROI), it is necessary to extract and fuse the features of each ROI from the key point features. Therefore, the present invention adopts a ROI feature extraction method based on key points, such as Figure 5As shown in the figure, it is used for multi-scale ROI feature encoding. Similar to the method adopted in formula (9), for each ROI in 3D space, 6 points are equally spaced along the X, Y, and Z axes, that is, 6×6×6 reference points are uniformly generated in the space. The reference points are only used as set coordinates to aggregate the features of the surrounding key points. They do not have their own feature information, and the number of reference points does not need to be updated in the subsequent training process. It is expressed as , centered at the reference point, r is the feature of the key points around the radius aggregation, and the feature set of its adjacent key points is:

[0125]

[0126] in For key points Relative to reference point The local relative position of the key points is then aggregated into adjacent key point feature sets.

[0127]

[0128] in and Same as the definition in formula (9), the present invention sets multiple radii r , and the key point features of different receptive fields are connected in series to obtain richer multi-scale feature information. In obtaining the key point aggregation features in the ROI area Then select six angles of 0°, 30°, 60°, 90°, 120°, and 150° to generate pre-selected boxes. Then input the box optimization network to calculate the residual of the box center, size, and direction. The box optimization network consists of two MLP layers. The network has two branches, one for confidence prediction and the other for pre-selected box optimization. k ROIs are normalized to obtain classification confidence for

[0129]

[0130] in The value range of is [0,1], It is k The interaction ratio between the ROI and the real data label is calculated, and then the confidence branch is trained. Considering that the classification confidence and the positioning confidence are usually not completely consistent, a mandatory consistency loss is proposed to ensure the consistency between the positioning and classification confidence, so that the pre-selected box with higher positioning confidence also has higher classification confidence. The loss can be expressed as:

[0131]

[0132] in D andG Represents the pre-selected box and the real data label, The positioning confidence of the representative box. In order to optimize the loss function, the classification confidence and the positioning confidence are required to be improved as much as possible. Although the mandatory consistency loss function proposed in the present invention is similar to the interaction ratio (IOU), the purpose and function are different. The IOU attempts to produce a more accurate regression by optimizing the confidence, while the purpose of the mandatory consistency loss is to ensure the consistency of the positioning and classification confidence and obtain a more accurate bounding box.

[0133] Loss function The loss function of this invention consists of three parts: pre-selection box loss , key point extraction loss , box optimization loss .

[0134] (1) Pre-selection box loss

[0135]

[0136] in represents the prediction residual, represents the regression target, and the smooth_L1 loss function is used to perform regression calculations on the two. For classification loss, the focal loss function is used to balance positive and negative samples, and its calculation formula is as follows:

[0137]

[0138] The balance factor , used to balance the uneven ratio of positive and negative samples, factor , used to reduce the loss of easy-to-classify samples, is the output probability of belonging to the true value;

[0139] (2) Key point extraction loss Given in the foreground point segmentation section above, it is also calculated using the focal loss function;

[0140] (3) Box optimization loss

[0141]

[0142] in is a mandatory consistent loss function, is the prediction residual, is the regression target, and The encoding method is the same; the total loss function is the sum of the above three loss functions with the same weights.

[0143] Experiment and analysis

[0144] The present invention uses Kitti, Waymo and Nuscene's 3D target detection datasets to verify the proposed algorithm framework. Mainly, multiple methods are compared and verified on the Kitti dataset, PR curves are drawn, ablation studies are performed and visualization results are displayed. In order to verify the generalization performance of this method, multiple methods are also compared and verified on the Waymo and Nuscene large datasets, and the loss functions of multiple methods are evaluated and curves are drawn. Finally, based on the physical vehicle, the present invention relies on the test platform of Tsinghua University's Autonomous Driving Research Institute, uses LiDAR to collect point cloud data, and performs annotation work. The code is ported to the ROS platform to test the portability and generalization ability of the algorithm. At the same time, ablation studies of submodules are conducted, and mAP test accuracy tables are drawn.

[0145] Kitti Dataset

[0146] The Kitti dataset has 7481 training samples and 7518 test samples, using the mean average precision (mAP) as the evaluation criterion, where the IoU threshold for cars is 0.7, the IoU threshold for pedestrians and cyclists is 0.5, and the recall rate is 40. Table 1 shows the mAP comparison of the proposed method with other excellent methods on the Kitti test dataset, where Second, PointPillar, and Part-A2 are based on the method of dividing voxels, and their backbone network is based on VoxelNet and has made a lot of innovative improvements. PointRCNN is based on the method of directly processing point clouds, and its backbone network is based on PointNet and has made a lot of improvements. AVOD and F-PointNet are based on the method of combining images and point clouds.

[0147] As shown in Table 1, the proposed method is significantly better than other methods in vehicle detection, that is, the mAP is improved by 0.29%, 1.45%, and 0.49% in the simple, medium, and difficult levels. [11，20，22] Instead of providing separate classes of trained models for cyclists and pedestrians, use a unique trained model.

[0148] The detection results under the bird's eye view are as follows Figure 6 As shown, the IoU threshold is 0.7 and the recall rate is still 40. It can be seen from the broken line that the method proposed in the present invention has advantages in simple and medium difficulty, and the average accuracy (mAP) is increased by 0.34% and 0.33% respectively. In addition, the accuracy is also high in difficult cases, reaching 69.55%, which is competitive with other methods.

[0149]

[0150] like Figure 7 As shown in the figure, for various methods tested on the Kitti dataset, the recall and precision curves are plotted. The precision is calculated by dividing the predicted number of positive samples by the number of samples predicted as positive samples (including negative samples that are incorrectly predicted as positive samples) when a certain threshold is set. The recall is calculated by dividing the number of correctly classified samples by the number of all positive samples when a certain threshold is set. It can be seen from the curve that under different thresholds, the method proposed in the present invention can better balance precision and recall, and converges more slowly than other methods.

[0151] The Kitti dataset was tested on scenes such as roads, cities, and campuses. The detection results are as follows: Figure 8 As shown, the upper half of each sub-image is point cloud data, and the lower half is an RGB image. In the detection result, the green frame is a car, the blue frame is a pedestrian, and the yellow frame is a cyclist. It can be seen from the results that the detection result of the point cloud data of the present invention is accurate in position and has high classification accuracy, and can detect occluded targets in the RGB image.

[0152] In order to verify the effectiveness of the fusion algorithm proposed in this invention, the model is divided into three submodules: 3D convolution extraction based entirely on voxels, voxel features combined with frame optimization network extraction, and voxel features fused with key points in the frame optimization stage for feature extraction (method of this invention). The model is verified on the easy, medium, and difficult standards of the Kitti dataset, and the detection result curves are shown in the figure below. Fig. 9 shown.

[0153] From the ablation study chart, it can be seen that the multi-feature fusion method proposed in the present invention can significantly improve the detection accuracy, and the mAP reaches 92.18%, 83.96%, and 81.86% respectively at three difficulty levels. However, the effect of using only voxel-based feature extraction is poor, especially at the difficulty level, which is 7.61% lower than the detection accuracy of the method of the present invention.

[0154] Finally, in order to verify the time efficiency of the algorithm proposed in the present invention, we selected PointRCNN, an excellent method based on direct processing of point clouds, SECOND, an excellent method based on voxels, and F-PointNet, a multi-sensor fusion method, and compared them with the method proposed in the present invention in the Kitti dataset. The detection time of each frame of point cloud data was detected. The results are shown in Table 2.

[0155]

[0156] As can be seen from Table 2, the algorithm proposed in the present invention has a significant improvement in time efficiency compared to the other three algorithms, reaching 0.06s. The multi-sensor fusion method needs to process data in different forms, and the computational cost is relatively high. The method based on direct processing of point clouds requires sampling, clustering, and feature extraction of disordered point clouds, and does not use the foreground point segmentation layer, namely the PKW module. Each point needs to be processed, which undoubtedly requires a lot of time and computational cost. Although the voxel-based method has improved the traditional three-dimensional convolution method, it still has the problem of insufficient time efficiency. The algorithm proposed in the present invention combines direct processing of point clouds and proposes a foreground point segmentation module (PKW) and a method for dividing voxel space and introduces a submanifold sparse convolution method, which ensures time efficiency on the basis of realizing multi-scale feature fusion.

[0157] Waymo Dataset

[0158] The model proposed in this invention is evaluated on the large-scale Waymo dataset to further verify the effectiveness of the method. X and Y The detection range of the axis is , the detection range of the Z axis is , and set the voxel size to The method of the present invention is evaluated using the official evaluation tool, where the IoU threshold is set to 0.7, and the point cloud space is divided into .

[0159] As shown in Table 3, the proposed method is significantly better than other advanced methods, with the mAP of 3D object detection increased by 6.21% and the mAP of BEV object detection increased by 2.83%. It is proved that the multi-scale voxel point cloud fusion feature proposed in the present invention can effectively obtain more accurate context information to improve 3D detection performance. The model is trained using 4 GTX1080ti GPUs, with the batch size set to 16, the learning rate set to 0.01, and the training 80 times. The following is obtained: Fig.10 The loss function curve shown in Figure 1 is the loss function curve of each module of the algorithm, that is, , (b) is the total loss function curve. It can be seen from the curve that the gradient descent is very rapid at the beginning, and may gradually become flat in the later stage. This is a normal performance, and there will not be a large drop in the later stage.

[0160]

[0161] Nuscene Dataset

[0162] In order to prove that the algorithm proposed in this invention has good generalization ability, the Nuscene dataset is also tested, which contains a variety of scenes. The test results are shown in Table 4. It can be seen that the method proposed in this invention still performs well on the Nuscene dataset, with a detection accuracy of 90.57% and 79.24% for cars and trucks. The detection accuracy of pedestrians and cyclists is 57.94% and 91.35%, which is better than the other two methods.

[0163]

[0164] References:

[0165] 11. Li Xiaoguang, Fu Chenping, Li Xiaoli, Wang Zhanghui. Improved Faster R-CNN algorithm for multi-scale object detection[J]. Journal of Computer-Aided Design & Graphics, 2019, 31(07): 1095-1101.

[0166] 15. Geiger A, Lenz P, Stiller C, et al . Vision meets robotics: Thekitti dataset[J]. The International Journal of Robotics Research, 2013, 32(11): 1231-1237.

[0167] 20. Shi S, Wang Z, Wang X, et al . Part-aˆ 2 net: 3d part-aware and aggregation neural network for object detection from point cloud[J]. arXivpreprint arXiv:1907.03670, 2019, 2(3).

[0168] 21. Shi S, Wang

[0169] 22. Ku J, Mozifian M, Lee J, et al . Joint 3d proposal generation and object detection from view aggregation[C] / / 2018 IEEE / RSJ InternationalConference on Intelligent Robots and Systems (IROS). IEEE, 2018: 1-8.

[0170] 24. Qi CR, Yi L, Su H, et al. Pointnet++: Deep hierarchical featurelearning on point sets in a metric space[J]. arXiv preprint arXiv:1706.02413, 2017.

[0171] Physical platform testing

[0172] Finally, the present invention relies on the physical vehicle platform and test site of the Autonomous Driving Research Institute of Tsinghua University to verify the proposed algorithm. Since the algorithm proposed in the present invention processes point clouds, a laser radar sensor is used for experiments, and data is collected using a physical platform, and finally a point cloud file in Rosbag format is obtained. The test process is divided into three parts: real-time dynamic multi-target detection, ablation comparison study, and accuracy detection. In order to conduct ablation research and accuracy detection, and to produce a small data set in point cloud format, the Rosbag file is played, the point cloud frame is extracted, and the target real data label and location information are framed in the point cloud through the annotation software, and four types of targets are calibrated, namely small cars, vans, pedestrians, and cyclists. In addition, the spatial size information of the point cloud in the data set refers to the Kitti data set.

[0173] The present invention conducted an ablation study, extracted the feature extraction module proposed in the present invention separately for testing, transplanted each sub-module to the ROS-based operating platform, input the collected data into the sub-module, compared it with the real data label, obtained the detection accuracy data, and drew an ablation study table, as shown in Table 5.

[0174]

[0175] in As introduced above, it can be seen from Table 5 that the method of multi-feature fusion proposed in the present invention has the best detection accuracy, reaching 85.02%, while the feature detection accuracy obtained by directly processing the point cloud is only 80.26%, which is 4.76% lower than the same period last year. In addition, the feature detection effects of different scales based on voxel extraction show that the submanifold sparse convolution proposed in the present invention has a good effect, and the feature fusion obtained by multiple convolutions can significantly improve the detection accuracy, reaching 84.44%.

[0176] The accuracy detection research is divided into two parts: target category and positioning detection, and accuracy detection at different distances. Fig.11 The curve diagram of Fig.11 As shown in (a), from the data collected from the physical platform and the data set input model to obtain the detection results of each category, the detection accuracy of vehicles is relatively high, reaching 86.25% and 87.38%, and the average accuracy is also 84.15%, indicating that the method proposed in the present invention has strong generalization ability and performs well in various data sets. Fig.11 As shown in (b), the target detection capabilities of the three models are studied at different distances. For example, in the detection of car categories, the detection accuracy only decays by 13.67% from a distance of 0-70.4 m. In the detection of cyclists, the method proposed in the present invention has comparable capabilities to the excellent methods in this field, with a margin of 2.17% at a distance of 30-48 m. This shows that the algorithm proposed in the present invention has good robustness and can achieve more accurate target detection through multi-feature fusion.

[0177] Finally, we use the ROS-based operating platform and input the point cloud data into the trained model for real-time dynamic multi-target detection. The detection speed reaches 0.06s. The visualization results are shown in the figure. Fig.12 As shown in the figure, the detection result is accurate, and the targets that can be identified include box trucks, small vehicles, pedestrians, and cyclists. This further proves that the algorithm proposed in the present invention has strong robustness and generalization ability and fast computing speed.

[0178] The present invention proposes a 3D dynamic target detection method based on voxel point cloud fusion, which is a novel method for fusing voxel and directly processed point cloud features, solving the problem of inaccurate detection results caused by insufficient features in traditional methods. On this basis, a mandatory consistent loss function is proposed to optimize the problem of inconsistent confidence. Experiments were conducted on Kitti, Waymo, Nuscene datasets and physical platforms, and the results showed that compared with other excellent methods, the method proposed in the present invention has significant effects, strong portability and generalization capabilities.

Claims

1. A 3D dynamic target detection method based on voxel point cloud fusion, after collecting the 3D dynamic target, it goes through the first stage and the second stage, and then outputs the 3D dynamic target from the point cloud to complete the 3D dynamic target detection; it is characterized by: After collecting the three-dimensional dynamic target, the point cloud is input; the first stage is to extract key point features, voxel features, feature fusion, and pre-selection box generation from the point cloud; the second stage is pre-selection box optimization and loss function calculation; in the second stage, the pre-selection box optimization extracts and fuses the features of each ROI from the key point features to generate an accurate and reliable 3D detection box; for each ROI in the 3D space, 6 points are equally spaced along the X, Y, and Z axes, that is, 6×6×6 reference points are uniformly generated in the space, expressed as N=(n1, n2, ..., n 216 ), with the reference point as the center and r as the radius to aggregate the features of the surrounding key points, the feature set of its adjacent key points is as follows: where p j -n i The key point p j Relative to the reference point n i The local relative position of , and then aggregate the adjacent key point feature sets: Where G(·) and T(·) are defined the same as in formula (9); Get key points in the ROI area to aggregate features Then select six angles of 0°, 30°, 60°, 90°, 120°, and 150° to generate pre-selected boxes, and input them into the box optimization network to calculate the residual of the box center, size, and direction; The box optimization network consists of two MLP layers. The network has two branches, one for confidence prediction and the other for pre-selected box optimization. The kth ROI is normalized to obtain the classification confidence y k as follows: y k =min(1),max(0,2IoU k -0.5)) (14) where y k The value range is [0, 1], IoU k is the interaction ratio between the kth ROI and the real data label, and then the confidence branch is trained in combination with the mandatory consistent loss; the mandatory consistent loss is as follows: Where D and G represent the pre-selected box and the real data label, and Area(·) represents the positioning confidence of the box; The loss function consists of three parts: pre-selected box loss L rpn , key point extraction loss L reg , box optimization loss L rcnn ; The pre-selected box loss is as follows: in Represents the prediction residual, Δr a represents the regression target, and the smooth_L1 loss function is used to perform regression calculations on the two; L cls For classification loss, the focal loss function is used to balance positive and negative samples, and its calculation formula is as follows: L cls =-α(1-c t ) γ logc t (17) The balance factor α = 0.25 is used to balance the uneven ratio of positive and negative samples, and the factor γ = 2.0 is used to reduce the loss of easy-to-classify samples. t is the output probability of the true value; the key point extraction loss is the focus loss function during foreground point segmentation; the box optimization loss is calculated as follows: Where L EC is a mandatory consistent loss function, is the prediction residual, Δr p is the regression target, and Δr a The encoding method is the same; the total loss function is the sum of the above three loss functions with the same weights.

2. The three-dimensional dynamic target detection method based on voxel point cloud fusion according to claim 1, characterized in that: In the first stage, the 3D convolution operation is performed by dividing the voxels, and the submanifold sparse convolution method is used as the backbone network for feature encoding and pre-selection box generation. At the same time, the FPS iterative farthest point sampling method is used to extract key points from the point cloud and cluster them, and the features of the surrounding areas are extracted. Finally, the features of the key points in the corresponding voxel space are fused with the features of the surrounding areas of the key points to obtain the high-level features of the key points in the point cloud space; In the second stage, reference points are selected in the ROI area generated based on voxels, clustered, and the key point features generated in the first stage are integrated to optimize the pre-selected box, which is then calculated using a loss function with confidence constraint enforcement.

3. The three-dimensional dynamic target detection method based on voxel point cloud fusion according to claim 1, characterized in that: In the first stage, the key point features are extracted from the point cloud, which are the key point sampling layer, clustering layer, feature extraction layer, and foreground point segmentation layer; the key point sampling layer uses the iterative farthest point sampling method to sample key points from the point cloud, including the following steps: input point cloud set P = {p1, p2, p3, ..., p n There are n points in total, randomly select p i As the starting point, write the key point set K = {p i }, calculate p i And the Euclidean distance of the remaining n-1 points, select the farthest point p j Write set K = {p i , p j }, the next iteration considers two points at the same time, calculates the distances from the remaining n-2 points to the two points in the set K, takes the shortest as the distance from the point to the point set, obtains the distances from n-2 points to the set K, and selects the farthest point p k Write set K = {p i , p j , p k }, continue to iterate the above process until the required number of points are sampled; the clustering layer uses multi-resolution combination to divide the spherical area with the key point as the center and R as the radius, extracts the information of the points contained in the area, and obtains the features associated with the target; the extracted features are composed of two feature vectors in series, the first half is composed of feature extraction of all points in the original point cloud data, and the second half is obtained by inputting the features of the first half into the PointNet network; the feature extraction layer uses the PointNet network to perform convolution and pooling operations on each local feature obtained by the clustering layer, and the obtained multi-resolution features are used as the local features of the center point to obtain features with consistent dimensions; the foreground point segmentation layer uses the key point prediction weighted layer to detect the key points obtained by the feature extraction layer and screen out the foreground points.

4. The three-dimensional dynamic target detection method based on voxel point cloud fusion according to claim 3 is characterized in that: The foreground point segmentation layer inputs the key point features into the two-layer perceptron network to obtain the weight information of the foreground point; the first layer MLP (128, 256) network has two convolution processes. The first convolution process is to use 128 1×128×1 convolution kernels to convolve with the input features to obtain n 1×128 feature vectors. The second convolution process uses 256 1×1×128 convolution kernels to obtain n 1×256 feature vectors, and the feature vectors are converted to n×256 through dimensional transformation; the second layer MLP (128 ,1) The network has two convolution processes. The first one uses 128 1×256×1 convolution kernels to convolve with the input features to obtain n 1×128 feature vectors. The second one uses 1 1×1×128 convolution kernel to convolve and obtain n 1×1 feature vectors. Finally, the foreground point prediction value is converted to n×1 through dimensionality transformation. When processing point cloud data containing multiple points, each point is multiplied by the same weight, and the weight range is between (0, 1). Finally, it is multiplied with the feature vector to obtain the final feature of the key point. Each Key point features The prediction feature weights of are expressed as follows: Where ψ(·) is a two-layer MLP network to predict the confidence of the foreground point between (0, 1); its calculation formula is divided into two parts: forward propagation and weight update. The forward propagation formula is as follows: In the forward propagation formula, is the hidden layer network weight, the activation function is the softmax function, is the output layer network weight, output label One-Hot encoding is used; the loss function uses the focal loss function; the gradient calculation and weight update formulas of the output layer and hidden layer are as follows: Where E n is the result of loss function calculation, and η is the gradient descent weight.

5. The three-dimensional dynamic target detection method based on voxel point cloud fusion according to claim 1, characterized in that: In the first stage, voxel feature extraction consists of three parts: voxel partitioning layer, VFE feature extraction layer, and submanifold sparse convolution layer; the voxel partitioning layer subdivides the 3D space into voxels of equal size and spacing, defines the point cloud as a 3D space with a range of D, H, and W along the Z, Y, and X axes, and defines the size of each voxel as v D , v H , v W , then the number of divisions along each coordinate axis is D′, H′, W′, and the entire point cloud space is divided into N voxels of equal size: VFE feature extraction layer uses the VFE layer to encode the features of voxels. For non-empty voxels containing t ≤ T points, use Indicates that p i Contains the X, Y, Z coordinates and reflectivity r of the i-th point in the voxel i , first calculate the local mean in voxel V as the centroid, expressed as (v x , v y , v z ), calculate p i The offset relative to the center of mass is expressed as (x i -v x ,y i -v y , z i -v z ), we can get the input feature set Next, we use the fully connected layer (FCN) to transform the input 7-dimensional vector V bt Mapped to 32 dimensions, each voxel has t 32-dimensional feature vectors, use the maximum pooling operation to get the global variable of the voxel, and concatenate it with each point feature. At this time, the feature of each point is 64 dimensions. Repeat the above process again to get t 128-dimensional feature vectors of the voxel, and finally get the 128-dimensional global feature of the voxel through maximum pooling; the submanifold sparse convolutional layer constructs a hash table to store the positions with values ​​in the feature map. Let the input feature map be T in , where there are a non-zero elements, T in Encoded as an a×m feature matrix M in and a hash table H in , M in Store T in All non-zero eigenvectors in , m represents the dimension of the eigenvector, T in It is a hash table consisting of key-value pairs. Key refers to the row index of the feature matrix, which is an integer between [0, a-1]. Value represents the corresponding element in T in The index in is a d-dimensional vector, and the output sparse feature map T out The same code is M out and H out , H out =H in , M out is a×n matrix, n is the dimension of the output feature vector; the size of the convolution kernel and the convolution step are defined as follows: Among them A i refers to the dimension of the input, B i Refers to the dimension of the output, A i and B i Satisfy constraints Where f represents the size of the convolution kernel, s represents the convolution step, and p represents zero-filling; the submanifold sparse convolution layer is divided into two parts: null value zero filling and forced zero clearing; T in It is a d-dimensional tensor, and the size of the receptive field is defined as f d (referring to a d-dimensional cube with side length f), with T in An element t in =T in (a1, a2, ..., a d , m0) as the center, and f d All the empty value positions in the space are filled with zeros, and then convolution is performed with the convolution kernel; let T in The zero value area is D zero , T ont The corresponding zero-value area in is set to zero; after the sub-manifold sparse convolution layer feature extraction, the meaning of the three-dimensional convolution parameter is Conv(C in , C out , k, s, p), k is the convolution kernel size, s is the step size, and p is the zero value padding. By converting the 4-fold downsampled 3D feature map into a 2D bird's-eye view feature map, a high-quality 3D pre-selection box is generated using an anchor-based method.

6. The three-dimensional dynamic target detection method based on voxel point cloud fusion according to claim 1, characterized in that: In the first stage, feature fusion fuses the obtained features; the set of voxel feature vectors of the k-th level of the submanifold sparse convolutional layer is defined as The corresponding 3D coordinate set is It is calculated by the voxel coordinate index in three-dimensional space and the actual voxel size, where N k Represents the number of non-empty voxels in the k-th level voxel feature. i Centered, R k Detect adjacent non-empty voxels for the radius and use the collection express: in is the local relative coordinate, used to correspond to the voxel feature The relative position in the image is then normalized to obtain the non-empty voxel sets at different scales. Features: Where T(·) represents the In the above example, a maximum of T voxels are randomly sampled for calculation, and G(·) represents a multi-layer perceptron network, which is used to encode voxel features and relative positions. The feature vectors of different numbers of adjacent voxels are mapped to The features obtained from convolution operations of different scales can be cascaded to obtain p i Multi-scale features: The original point cloud features can be obtained by formula (9): and Bird's Eye View Features Further get the key point p i Features: The structural information of the entire 3D scene can be preserved very effectively.

7. The three-dimensional dynamic target detection method based on voxel point cloud fusion according to claim 1, characterized in that: In the first stage, the pre-selected box generation analyzes the bird's-eye view to obtain a probability score map and a regression map. For each point in the bird's-eye view, a pre-selected box with two angles will be generated in the regression map. Each box has seven parameters (x, y, z, h, w, l, r), which represent the coordinates of the X, Y, and Z axes in space and the height, width, length, and deflection angle information of the box.

8. A device for three-dimensional dynamic target detection based on voxel point cloud fusion, used in the three-dimensional dynamic target detection method based on voxel point cloud fusion according to claim 1, characterized in that: It includes an image acquisition module and a data processing module. The data processing module includes a point cloud input module, a point cloud key point feature extraction module, a voxel feature extraction module, a feature fusion module, a pre-selection box generation module, a pre-selection box optimization module, a loss function calculation module and a point cloud output module. The image acquisition module is a radar sensor and a camera.

9. An electronic device, comprising a processor and a memory, wherein the memory stores instructions executed by the processor, so that the processor can execute the three-dimensional dynamic target detection method based on voxel point cloud fusion according to claim 1; A computer-readable storage medium storing computer instructions, wherein the computer instructions enable a computer to execute the three-dimensional dynamic target detection method based on voxel point cloud fusion as described in claim 1.

Images