A laser point cloud 3D target detection model and method for complex traffic scenes

Abstract

The present invention discloses a laser point cloud three-dimensional target detection model and method for complex traffic scenes. The three-dimensional encoder in the model is conducive to the detection of long-distance targets and small targets. Sparse convolution and submanifold convolution can greatly improve the coding efficiency of voxel features. The residual structure of the two-dimensional encoder can maintain more complete information, which is conducive to the detection of long-distance targets and small targets. At the same time, the network is easier to optimize. Through self-calibration convolution, feature extraction can be performed at the original scale and self-calibration scale respectively, the receptive field is expanded, and more complete and rich features are extracted. Through spatial attention and channel attention, useful feature expression is enhanced in the spatial direction and channel direction, and useless information is suppressed. The final detection accuracy of the present invention is 81.88% for vehicles, 47.82% for pedestrians, and 69.81% for cyclists. The average accuracy is 66.25%, which is 9.9% higher than the existing VOXEL RCNN algorithm. 13.8FPS is achieved on the RTX 2080Ti display. The detection accuracy and speed can meet the perception needs of smart cars in complex traffic environments.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent vehicle perception, and specifically refers to a laser point cloud three-dimensional target detection model and method for complex traffic scenes. Background Art

[0002] Smart cars are complex systems that include perception, decision-making, and control. Among them, environmental perception technology is the foundation of smart cars, which provides necessary environmental information for subsequent decision-making and control. The accuracy of traditional machine learning algorithms can no longer meet the current operation requirements of smart cars. Therefore, perception algorithms based on deep learning have developed rapidly and have made great progress in the field of two-dimensional target detection and segmentation. However, cameras will be affected by darkness, rain, fog, strong light, etc., which will affect the detection effect. With the reduction of lidar costs and the improvement of computer computing power, three-dimensional target detection is constantly used in smart cars. Two-dimensional target detection can only provide the position of the target in a two-dimensional image, while three-dimensional target detection can provide the position, shape and heading angle of the target in the real environment, which plays a very important role in subsequent decision-making and planning. Summary of the invention

[0003] Most of the current three-dimensional target detection algorithms cannot adapt to complex traffic scenes, and the detection effects of long-distance vehicle detection and close-range small targets such as pedestrians and cyclists are poor, which cannot meet the perception needs of smart cars in complex traffic conditions. Based on the VOXEL RCNN algorithm, the present invention improves and designs a three-dimensional target detection model algorithm for complex traffic environments, so that it can improve the detection accuracy of long-distance targets and small targets and can be applied to complex traffic environments. The present invention achieves the above technical objectives through the following technical solutions.

[0004] The present invention proposes a laser point cloud 3D target detection model for complex traffic scenes, including a voxel processing module, a 3D encoder module, a 2D encoder module, a candidate box generation module, a candidate box pooling module, and a fully connected layer module;

[0005] The voxel processing module is used to voxelize the inputted laser radar point cloud data;

[0006] The three-dimensional encoder module: extracts features from non-empty voxels, and projects the obtained voxel features onto a bird's-eye view to generate a pseudo image expression;

[0007] The two-dimensional encoder module is used to extract features from the pseudo-image expression to obtain a two-dimensional feature expression;

[0008] The candidate box generation module uses two-dimensional features to perform target classification and regression to generate high-quality candidate boxes;

[0009] The candidate box pooling module performs voxel pooling on the candidate box to obtain pooling features;

[0010] The fully connected layer module: refines the candidate boxes based on the pooled features.

[0011] Furthermore, the specific process of the voxelization processing module performing voxelization processing on the laser radar point cloud data is as follows:

[0012] The input point cloud is pre-processed along the X, Y, and Z directions to voxelize the entire point cloud scene into uniform voxels, with the center of the lidar as the origin, the X axis forward along the vehicle, the Y axis to the left, and the Z axis upward. The range of the entire point cloud scene (X, Y, Z) is [(-75.2, 75.2), (-75.2, 75.2), (-5, 3)] meters, and the size of each voxel is set to (0.1, 0.1, 0.2) meters. Therefore, the entire scene is divided into 1504×1504×40 voxels of equal size. If there are more than 5 original points in each voxel, it is downsampled to 5, and the average of the x, y, z, coordinates and reflection intensity of all points in each voxel after downsampling is used as the original feature of the voxel.

[0013] Furthermore, the 3D encoder module includes three parts: residual structure, sparse convolution, and submanifold convolution. The input of the 3D encoder module is a 40×1504×1504 voxel expression, and each voxel has a 4-dimensional (x, y, z, reflection intensity) original feature. After the input, a 3×3×3 submanifold convolution is used to extract the original features of the voxel and perform 1x downsampling, followed by feature extraction using 2 consecutive residual submanifold blocks, and then 3 3×3×3 sparse convolutions are used to perform 2x, 4x, and 8x downsampling along the X, Y, and Z directions, respectively. In this way, multi-scale three-dimensional features are obtained. Each sparse convolution is followed by two consecutive residual sub-manifold blocks for feature extraction. The structure of the residual sub-manifold block consists of two 3×3×3 sub-manifold convolutions, and the input of the first sub-manifold convolution and the output of the second sub-manifold convolution are added. At the same time, each sub-manifold convolution is followed by a BatchNorm layer and a ReLU layer. Finally, a sparse convolution is used to downsample the Z direction by 2 times, and then projected onto the bird's-eye view to convert it into a 256×188×188 pseudo image expression.

[0014] Furthermore, the two-dimensional encoder module includes five parts: a residual structure, a two-dimensional convolution, a self-calibration convolution, a spatial attention and a channel attention mechanism. The module uses a 3×3 two-dimensional convolution to perform 1x, 2x, and 4x downsampling on the 256×188×188 pseudo image expression obtained by the three-dimensional encoder module to obtain multi-scale features. Then, two consecutive self-calibration convolutions are used after each two-dimensional convolution to extract the multi-scale features, and the input of the first self-calibration convolution and the output of the second self-calibration convolution are added. The spatial attention and channel attention mechanisms are added after the self-calibration convolution to enhance useful feature expressions and reduce useless information in the spatial and channel directions; finally, the multi-scale features are returned to the same size through deconvolution and spliced ​​along the channel direction, and then a two-dimensional convolution is used to perform dimensional compression to obtain a 64×188×188 two-dimensional feature expression.

[0015] Furthermore, the candidate box generation module places 10 anchor boxes in each pixel of the 188×188 two-dimensional feature map for the two-dimensional feature expression of size 64×188×188 obtained by the two-dimensional encoder module. They are five categories: cars, trucks, buses, pedestrians, and cyclists, and each category has two directions of 0° and 180°. The two-dimensional feature map is used to perform category prediction and size regression (x, y, z, w, h, l, θ) for each anchor box to obtain 10×188×188 three-dimensional boxes. During training, 9000 three-dimensional boxes with the highest classification scores are selected and sent to the non-maximum suppression module to obtain 512 high-quality candidate boxes.

[0016] Furthermore, the candidate box pooling module randomly samples 128 candidate boxes from the 512 high-quality candidate boxes generated by the candidate box generation module and sends them to the refinement stage, of which 64 are positive samples and 64 are negative samples. The voxel candidate box pooling module is used to extract multi-scale voxel features: first, 6×6×6 grid points are uniformly sampled for each candidate box, and then non-empty voxels within a certain distance around these grid points are queried, and then pointnet, fully connected layers and maximum pooling layers are used to extract the features of these non-empty voxels to obtain pooled features; when querying the voxels around the grid points, the voxels sampled by 2 times, 4 times, and 8 times are queried respectively to obtain multi-scale voxel features.

[0017] Furthermore, the fully connected layer module takes the pooling of 128 candidate boxes randomly sampled by the candidate box pooling module as input, performs confidence prediction and refinement of the regression box through two layers of fully connected layers, and predicts the classification confidence c of each candidate box for the confidence prediction branch. i , its target is obtained by the following formula:

[0018]

[0019] in is the confidence target value of the i-th candidate box; IoU i is the intersection-over-union ratio of the i-th candidate box and its true value box. H ,θ L are the intersection-over-union ratio thresholds for foreground and background, which are 0.75 and 0.25 respectively;

[0020] The intersection-over-union ratio of the predicted box and the true value is associated with the classification execution degree to alleviate the problem of mismatch between classification confidence and positioning accuracy. For the regression box refinement branch, more accurate location information (x, y, z, w, h, l, θ) is predicted through the fully connected layer.

[0021] Furthermore, the dataset used by the model adopts Huawei's ONCE dataset as a training dataset and a validation dataset. During training, a category-balanced sampling enhancement method is added, that is, training scenes with fewer categories are randomly copied and sent to the training dataset to alleviate category imbalance and expand the number of training samples.

[0022] Furthermore, the model training adopts end-to-end training, and five categories of cars, trucks, buses, cyclists and pedestrians are trained simultaneously; the network is trained for 90 rounds on two RTX 2080Ti GPUS, and its optimizer is Adam, the learning rate variation method adopts cosine fire method, the maximum learning rate is 0.003, the frequency division factor is 10, the momentum is from 0.95 to 0.85, the weight decay is 0.02, and the batch size is 6; its total loss L TOTAL Including RPN loss L RPN and RCNN loss L RCNN In the RPN stage, FocalLoss is used to calculate the classification loss L cls , use Smooth L1 Loss to calculate the regression loss L reg1 , in the RCNN stage, the confidence loss L is calculated using the binary cross entropy loss iou , use Smooth L1 Loss to calculate the regression loss L reg2 :

[0023] L TOTAL =L RPN +L RCNN

[0024]

[0025] Where N fg is the number of foreground candidate boxes, s i is the predicted category of the candidate box, is the true value of the candidate box category, α represents the regression loss calculated only for the candidate box of the foreground, is the predicted candidate box regression parameter, is the true value of the candidate box regression parameter;

[0026]

[0027] Where N s is the number of candidate boxes sampled, c i is the confidence of the prediction, is the target value of confidence, β represents the regression loss calculated only for the candidate box of the sampled foreground, is the predicted candidate box regression parameter, is the true value of the candidate box regression parameter.

[0028] Based on the above model, the present invention proposes a method for detecting three-dimensional targets in laser point clouds for complex traffic scenes, which includes the following steps:

[0029] Step 1: Select or create a data set as a training data set and a verification data set for the detection network, and add a class-balanced sampling enhancement method during training.

[0030] Step 2: voxelize the lidar point cloud input to the network.

[0031] Step 3: Use a three-dimensional encoder composed of a residual structure, sparse convolution, and submanifold convolution to extract features from non-empty voxels, and project the obtained voxel features onto a bird's-eye view to generate a pseudo image expression.

[0032] Step 4: Use a two-dimensional encoder composed of a residual structure, two-dimensional convolution, self-calibration convolution, spatial attention and channel attention mechanism to extract features of the pseudo image expression to obtain a two-dimensional feature expression.

[0033] Step 5: Use two-dimensional features to classify and regress the target and generate high-quality candidate boxes.

[0034] Step 6: Perform voxel pooling on the candidate boxes to obtain pooled features.

[0035] Step 7: Based on the pooled features, use the fully connected layer to refine the candidate boxes.

[0036] Step 8: Train the network from step 2 to step 7.

[0037] Step 9: Visualization of detection results.

[0038] The specific implementation process of each step is detailed in the content recorded in the specific implementation method.

[0039] Beneficial effects of the present invention:

[0040] 1. The present invention uses the autonomous driving data set ONCE for network training. The data set collected on different roads, in different weather conditions, and at different times can better represent complex traffic conditions. The network trained with this data set can be applicable to complex traffic conditions.

[0041] 2. The three-dimensional encoder designed by the present invention is composed of a residual structure, sparse convolution, and submanifold convolution. The residual structure can maintain more complete information, which is beneficial to the detection of long-distance targets and small targets. Sparse convolution and submanifold convolution can greatly improve the encoding efficiency of voxel features, which is beneficial to real-time detection.

[0042] 3. The two-dimensional encoder designed by the present invention is composed of a residual structure, a two-dimensional convolution, a self-calibration convolution, spatial attention and channel attention. The residual structure can maintain more complete information, which is beneficial to the detection effect of long-distance targets and small targets, and makes the network easier to optimize. The self-calibration convolution can be used to extract features at the original scale and the self-calibration scale, respectively, which expands the receptive field and can extract more complete and richer features. The spatial attention and channel attention can enhance the expression of useful features in the spatial direction and the channel direction, and suppress useless information.

[0043] 4. During training, the present invention adds a category balanced sampling method, which, on the one hand, alleviates the category imbalance and improves the detection effect of pedestrians and cyclists with a small sample size in the data set; on the other hand, it expands the training samples of the data set and improves the robustness of the algorithm.

[0044] 5. The final detection accuracy of the present invention is 81.88% for vehicles, 47.82% for pedestrians, and 69.81% for cyclists, with an average accuracy of 66.25%, which is 9.9% higher than the original VOXEL RCNN algorithm. 13.8FPS is achieved on the RTX 2080Ti display. The detection accuracy and speed can meet the perception needs of smart cars in complex traffic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is the algorithm flow of the laser point cloud three-dimensional target detection method for complex traffic scenes of the present invention.

[0046] Figure 2 It is a network structure diagram of the three-dimensional encoder of the present invention.

[0047] Figure 3 It is a network structure diagram of the two-dimensional encoder of the present invention.

[0048] Figure 4 It is a detection visualization effect diagram of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described below in conjunction with the accompanying drawings.

[0050] The present invention proposes a laser point cloud 3D target detection method for complex traffic environments. Figure 1 As shown, the specific process includes the following:

[0051] Step 1: Select the Huawei ONCE dataset as the training dataset and verification dataset for the detection network, and add a category-balanced sampling enhancement method during training.

[0052] The Huawei ONCE dataset is collected by 7 cameras and 1 40-line LiDAR. It contains 5 categories, namely cars, trucks, buses, pedestrians and cyclists. The dataset collects scenes in different weather (sunny, cloudy, rainy), different times (morning, Chinese, afternoon, evening), and different road conditions (city center, suburbs, highways, tunnels, bridges), which can better represent complex traffic conditions. The ONCE dataset collects 144 hours of autonomous driving scenes and has about 1 million frames of samples. Among them, 16,000 frames have labeled samples, 5,000 frames are used for training, 3,000 frames are used for verification, and 8,000 frames are used for testing. In addition, since there are more car samples in the training samples and fewer training samples of other trucks, buses, pedestrians and cyclists, the present invention adopts a category balanced sampling enhancement method during training, that is, randomly copying training scenes with fewer categories and sending them into the training data set, thereby alleviating category imbalance and improving the detection accuracy of fewer categories. At the same time, since the number of training samples is expanded from 5000 frames to 16600 frames, which is expanded by 3.3 times, the increase in training samples improves the robustness of the model.

[0053] Step 2: voxelize the lidar point cloud input to the network.

[0054] The input point cloud is pre-processed by voxelization along the X, Y, and Z directions, and the entire point cloud scene is divided into uniform voxels so that the point cloud can be efficiently extracted through sparse convolution and submanifold convolution. The center of the lidar is taken as the origin, the X axis is along the vehicle forward, the Y axis is to the left, and the Z axis is upward. The range of the entire point cloud scene (X, Y, Z) is [(-75.2, 75.2), (-75.2, 75.2), (-5, 3)] meters, and the size of each voxel is set to (0.1, 0.1, 0.2) meters. Therefore, the entire scene is divided into 1504×1504×40 voxels of equal size. If there are more than 5 original points in each voxel, it is downsampled to 5, and the average of the x, y, z, coordinates and reflection intensity of all points in each voxel after downsampling is used as the original feature of the voxel. Through voxelization, the efficiency of point cloud feature extraction is greatly improved, which is conducive to real-time detection in autonomous driving scenarios.

[0055] in, Figure 2The internal form is expressed as SubM(a,b)-c×d×e-(f,g,h), where SubM represents submanifold convolution, a and b are the input and output dimensions respectively, c×d×e represents the convolution kernel size in the x, y, and z directions, and (f,g,h) represents the step size in the x, y, and z directions; Figure 2 The internal form is expressed as SpConv(a,b)-c×d×e-(f,g,h), where SpConv stands for sparse convolution, a and b are the input and output dimensions respectively, c×d×e represents the convolution kernel size in the x, y, z directions, and (f,g,h) represents the step size in the x, y, z directions; Figure 2 The internal form is expressed as Residual SubM Block (a, b), where Residual SubM Block represents the residual submanifold block, a and b are the input and output dimensions respectively, and its specific composition is Figure 2 Shown as dotted box.

[0056] Step 3: Use a three-dimensional encoder composed of a residual structure, sparse convolution, and submanifold convolution to extract features from non-empty voxels, and project the obtained voxel features onto a bird's-eye view to generate a pseudo image expression.

[0057] The input to the 3D encoder is a 40×1504×1504 voxel representation, and each voxel has 4-dimensional (x, y, z, reflection intensity) original features. The network structure of the 3D encoder is as follows Figure 2 As shown in the figure. First, a 3×3×3 sub-manifold convolution is used to extract the original features of the voxels and perform 1-fold downsampling, followed by 2 consecutive residual sub-manifold blocks for feature extraction. Then, 3 3×3×3 sparse convolutions are used to perform 2-fold, 4-fold, and 8-fold downsampling along the X, Y, and Z directions respectively to obtain multi-scale three-dimensional features. Each sparse convolution is followed by 2 consecutive residual sub-manifold blocks for feature extraction. Specifically, the structure of the residual sub-manifold block is composed of 2 3×3×3 sub-manifold convolutions to form a residual sub-manifold block, and the input of the first sub-manifold convolution and the output of the second sub-manifold convolution are added. This residual structure is conducive to maintaining more complete information during the downsampling process and improving the detection effect of small targets and long-distance targets. At the same time, each sub-manifold convolution is followed by a BatchNorm layer and a ReLU layer. The BatchNorm layer and the ReLU layer are conducive to accelerating the convergence speed of the network, preventing gradient explosion and gradient disappearance, and preventing overfitting. Finally, a sparse convolution is used to downsample the Z direction by a factor of 2, and then projected onto the bird's-eye view to convert it into a pseudo image representation of 256 × 188 × 188. In the 3D encoder, the use of a large number of sparse convolutions and submanifold convolutions can increase the speed of extracting non-empty voxel features.

[0058] Step 4: Use a two-dimensional encoder composed of a residual structure, two-dimensional convolution, self-calibration convolution, spatial attention and channel attention mechanism to extract features of the pseudo image expression to obtain a two-dimensional feature expression.

[0059] The 256×188×188 pseudo image representation obtained in step 3 is sent to the two-dimensional encoder for feature extraction. The network structure of the two-dimensional encoder is as follows: Figure 3 As shown. The original pseudo image expression is downsampled by 1, 2, and 4 times using a 3×3 two-dimensional convolution to obtain multi-scale features. Then, two consecutive self-calibration convolutions are used after each two-dimensional convolution to extract multi-scale features, and the input of the first self-calibration convolution and the output of the second self-calibration convolution are added. Self-calibration convolution is used to extract features in the original scale space and the self-calibration scale space, which greatly increases the receptive field and obtains more significant features. In addition, spatial attention and channel attention mechanisms are added after the self-calibration convolution to enhance useful feature expressions in the spatial and channel directions and reduce useless information. Finally, the multi-scale features are returned to the same size through deconvolution and spliced ​​along the channel direction, and then a two-dimensional convolution is used for dimensional compression to obtain a 64×188×188 two-dimensional feature expression.

[0060] in, Figure 3 The internal form is expressed as Conv(a,b)-c×d-(e,f), where Conv represents a two-dimensional convolution, a and b are the input and output dimensions respectively, c×d represents the convolution and size in the x and y directions, and (e,f) represents the step size in the x and y directions; Figure 3 The internal form is expressed as Conv.T(a,b)-c×d-(e,f), where Conv.T represents deconvolution, a and b are the input and output dimensions respectively, c×d represents the convolution and size in the x and y directions, and (e,f) represents the step size in the x and y directions; Figure 3 The internal form is expressed as SC-Conv(a,b), where SC-Conv stands for self-calibrated convolution, and a and b are the input and output dimensions respectively; Figure 3 The internal form is expressed as CBAM(a,b), where CBAM stands for channel attention and spatial attention, and a and b are the input and output dimensions respectively; Figure 3 The internal form is a×b×c, where a represents the feature dimension, b and c represent the dimensions of the feature map in the x and y directions respectively.

[0061] Step 5: Use two-dimensional features to classify and regress the target and generate high-quality candidate boxes.

[0062] The two-dimensional feature representation obtained in step 4 is 64×188×188 in size. Ten anchor boxes (5 categories: cars, trucks, buses, pedestrians, and cyclists, each with two directions of 0° and 180°) are placed in each pixel of the 188×188 two-dimensional feature map. The two-dimensional feature map is used to perform category prediction and size regression (x, y, z, w, h, l, θ) for each anchor box to obtain 10×188×188 three-dimensional boxes. During training, the 9,000 three-dimensional boxes with the highest classification scores are selected and sent to the non-maximum suppression module to obtain 512 high-quality candidate boxes.

[0063] Step 6: Perform voxel pooling on the candidate boxes to obtain pooled features.

[0064] For the 512 candidate boxes generated in step 5, 128 are randomly sampled and sent to the refinement stage, of which 64 are positive samples and 64 are negative samples. The voxel candidate box pooling module is used to extract multi-scale voxel features. First, 6×6×6 grid points are uniformly sampled for each candidate box, and then non-empty voxels within a certain distance around these grid points are queried. Then, pointnet, fully connected layers, and maximum pooling layers are used to extract the features of these non-empty voxels to obtain pooled features. When querying the voxels around the grid points, the voxels sampled 2x, 4x, and 8x are queried respectively to obtain multi-scale voxel features, which is conducive to the detection of targets of different sizes.

[0065] Step 7: Based on the pooled features, use the fully connected layer to refine the candidate boxes.

[0066] For the 128 candidate boxes in step 6, their pooling is used as input, and confidence prediction and regression box refinement are performed through 2 layers of fully connected layers. For the confidence prediction branch, the classification confidence c of each candidate box is predicted i , its target is obtained by the following formula:

[0067]

[0068] in is the confidence target value of the i-th candidate box; IoU i is the intersection-over-union ratio of the i-th candidate box and its true value box. H ,θ L are the intersection-over-union ratio thresholds of foreground and background, which are 0.75 and 0.25 respectively.

[0069] Associating the intersection-over-union ratio of the predicted box and the true value with the classification execution degree can alleviate the problem of mismatch between classification confidence and positioning accuracy. For the regression box refinement branch, more accurate location information (x, y, z, w, h, l, θ) is predicted through the fully connected layer.

[0070] Step 8: Train the network from step 2 to step 7.

[0071] Steps 2 to 7 constitute the network model of the present invention. The network is trained end-to-end for five categories: cars, trucks, buses, cyclists, and pedestrians. The network is trained for 90 rounds on two RTX 2080Ti GPUS. The optimizer is Adam, the learning rate variation method adopts the cosine fire method, the maximum learning rate is 0.003, the frequency division factor is 10, the momentum is from 0.95 to 0.85, the weight decay is 0.02, and the batch size is 6. The total loss L TOTAL Including RPN loss L RPN and RCNN loss L RCNN , In the RPN stage, Focal Loss is used to calculate the classification loss L cls , use Smooth L1 Loss to calculate the regression loss L reg1 , in the RCNN stage, the confidence loss L is calculated using the binary cross entropy loss iou , use Smooth L1 Loss to calculate the regression loss L reg2 .

[0072] L TOTAL =L RPN +L RCNN

[0073]

[0074] Where N fg is the number of foreground candidate boxes, s i is the predicted category of the candidate box, is the true value of the candidate box category, α represents the regression loss calculated only for the candidate box of the foreground, is the predicted candidate box regression parameter, is the true value of the candidate box regression parameter.

[0075]

[0076] Where N s is the number of candidate boxes sampled, c i is the confidence of the prediction, is the target value of confidence, β represents the regression loss calculated only for the candidate box of the sampled foreground, is the predicted candidate box regression parameter, is the true value of the candidate box regression parameter.

[0077] Step 8: Visualization of detection results.

[0078] Input the point cloud scene into the trained network to obtain the refined target category, confidence and candidate box. At this time, there are multiple candidate boxes of the target in the point cloud road scene. First, sort them from large to small according to their classification confidence, filter out the boxes with lower confidence, and then select the box with the highest score from the remaining boxes. At the same time, delete the boxes whose intersection and union ratio with the box is greater than a certain threshold, and then repeat this operation in the remaining boxes (non-maximum suppression) to generate the final detection result and visualize it. The visualization effect is as follows Figure 4 As shown, green represents vehicles (cars, trucks, and buses are all classified as vehicles), yellow represents pedestrians, and blue represents cyclists. Table 1 is a comparison chart of the detection accuracy of the original VOXEL RCNN and the detection model algorithm of the present invention under different distance conditions.

[0079] Table 1 Comparison of detection accuracy between the original VOXEL RCNN and this algorithm at different distances

[0080]

[0081] As shown in Table 1, compared with the original VOXEL RCNN, the detection accuracy of this algorithm for vehicles is improved by 7.58%, the detection accuracy of pedestrians is improved by 12.16%, the detection accuracy of cyclists is improved by 9.96%, and the average accuracy is improved by 9.9%. The detection accuracy can meet the perception needs of smart cars in complex traffic conditions. Secondly, by counting the detection time of each frame of point cloud, 13.8FPS is achieved on the RTX 2080Ti graphics card, which can meet the real-time requirements.

[0082] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A laser point cloud 3D target detection device for complex traffic scenes. It is characterized in that It includes voxel processing module, 3D encoder module, 2D encoder module, candidate box generation module, candidate box pooling module and fully connected layer module; The voxel processing module is used to voxelize the inputted laser radar point cloud data; The three-dimensional encoder module: extracts features from non-empty voxels, and projects the obtained voxel features onto a bird's-eye view to generate a pseudo image expression; The two-dimensional encoder module is used to extract features from the pseudo-image expression to obtain a two-dimensional feature expression; The candidate box generation module uses two-dimensional features to perform target classification and regression to generate high-quality candidate boxes; The candidate box pooling module performs voxel pooling on the candidate box to obtain pooling features; The fully connected layer module: refines the candidate boxes based on the pooled features; The specific process of voxelization processing of the laser radar point cloud data by the voxelization processing module is as follows: The input point cloud is pre-processed into voxels along the X, Y, and Z directions, and the entire point cloud scene is divided into uniform voxels, with the center of the lidar as the origin, the X axis forward along the vehicle, the Y axis to the left, and the Z axis upward; the range of the entire point cloud scene (X, Y, Z) is [(-75.2, 75.2), (-75.2, 75.2), (-5, 3)], and the size of each voxel is set to (0.1, 0.1, 0.2) meters. Therefore, the entire scene is divided into 1504×1504×40 voxels of equal size. If there are more than 5 original points in each voxel, it is downsampled to 5, and the average of the x, y, z, coordinates and reflection intensity of all points in each voxel after downsampling is used as the original feature of the voxel; The 3D encoder module includes three parts: residual structure, sparse convolution, and submanifold convolution. The input of the 3D encoder module is a 40×1504×1504 voxel expression, and each voxel has a 4-dimensional (x, y, z, reflection intensity) original feature. After the input, a 3×3×3 submanifold convolution is used to extract the original features of the voxel and perform 1x downsampling. Then, 2 consecutive residual submanifold blocks are used for feature extraction, and then 3 3×3×3 sparse convolutions are used to perform 2x, 4x, and 8x downsampling along the X, Y, and Z directions respectively. In this way, multi-scale three-dimensional features are obtained. Each sparse convolution is followed by two consecutive residual sub-manifold blocks for feature extraction. The structure of the residual sub-manifold block is composed of two 3×3×3 sub-manifold convolutions, and the input of the first sub-manifold convolution and the output of the second sub-manifold convolution are added. At the same time, each sub-manifold convolution is followed by a BatchNorm layer and a ReLU layer. Finally, a sparse convolution is used to downsample the Z direction by 2 times, and then projected onto the bird's-eye view to convert it into a 256×188×188 pseudo image expression. The two-dimensional encoder module includes five parts: residual structure, two-dimensional convolution, self-calibration convolution, spatial attention and channel attention mechanism. The module uses a 3×3 two-dimensional convolution to perform 1, 2, and 4 times downsampling on the 256×188×188 pseudo image expression obtained by the three-dimensional encoder module to obtain multi-scale features. Then, two consecutive self-calibration convolutions are used after each two-dimensional convolution to extract the multi-scale features, and the input of the first self-calibration convolution and the output of the second self-calibration convolution are added. The spatial attention and channel attention mechanism are added after the self-calibration convolution to enhance the useful feature expression and reduce the useless information in the spatial and channel directions. Finally, the multi-scale features are returned to the same size through deconvolution and spliced ​​along the channel direction, and then a two-dimensional convolution is used to perform dimensional compression to obtain a 64×188×188 two-dimensional feature expression. The candidate frame generation module places 10 anchor frames in each pixel of the 188×188 two-dimensional feature map for the 64×188×188 two-dimensional feature expression obtained by the two-dimensional encoder module. The anchor frames are five categories: cars, trucks, buses, pedestrians, and cyclists. Each category has two directions of 0° and 180°. The two-dimensional feature map is used to perform category prediction and size regression (x, y, z, w, h, l, θ) on each anchor frame to obtain 10×188×188 three-dimensional frames. During training, 9000 three-dimensional frames with the highest classification scores are selected and sent to the non-maximum suppression module to obtain 512 high-quality candidate frames. The candidate box pooling module randomly samples 128 candidate boxes from the 512 high-quality candidate boxes generated by the candidate box generation module and sends them to the refinement stage, of which 64 are positive samples and 64 are negative samples. The voxel candidate box pooling module is used to extract multi-scale voxel features: first, 6×6×6 grid points are uniformly sampled for each candidate box, and then non-empty voxels within a certain distance around these grid points are queried, and then pointnet, fully connected layers and maximum pooling layers are used to extract these non-empty voxel features to obtain pooling features; when querying voxels around grid points, 2x, 4x and 8x downsampled voxels are queried respectively to obtain multi-scale voxel features; The fully connected layer module takes the pooling of 128 candidate boxes randomly sampled by the candidate box pooling module as input, performs confidence prediction and refinement of the regression box through two layers of fully connected layers, and predicts the classification confidence c of each candidate box for the confidence prediction branch. i , its target is obtained by the following formula: in is the confidence target value of the i-th candidate box; IoU i is the intersection-and-union ratio of the i-th candidate box and its true value box; θ H ,θ L are the intersection-over-union ratio thresholds for foreground and background, which are 0.75 and 0.25 respectively; The intersection-over-union ratio of the predicted box and the true value is associated with the classification execution degree to alleviate the problem of mismatch between classification confidence and positioning accuracy. For the regression box refinement branch, more accurate location information (x, y, z, w, h, l, θ) is predicted through the fully connected layer.

2. According to claim 1, a laser point cloud three-dimensional target detection device for complex traffic scenes, It is characterized in that The data set used by the device adopts the Huawei ONCE data set as the training data set and the verification data set. During training, a category-balanced sampling enhancement method is added, that is, training scenes with fewer categories are randomly copied and sent into the training data set to alleviate category imbalance and expand the number of training samples.

3. According to claim 1, a laser point cloud 3D target detection device for complex traffic scenes, It is characterized in that The device training adopts end-to-end training, and five categories of cars, trucks, buses, cyclists and pedestrians are trained simultaneously; the network is trained for 90 rounds on 2 RTX 2080Ti GPUS, and its optimizer is Adam, the learning rate variation method adopts cosine fire method, the maximum learning rate is 0.003, the frequency division coefficient is 10, the momentum is from 0.95 to 0.85, the weight decay is 0.02, and the batch size is 6; its total loss is: TOTAL Including RPN loss L RPN and RCNN loss L RCNN In the RPN stage, FocalLoss is used to calculate the classification loss L cls , use Smooth L1 Loss to calculate the regression loss L reg1 , in the RCNN stage, the confidence loss L is calculated using the binary cross entropy loss iou , use Smooth L1 Loss to calculate the regression loss L reg : L TOTAL =L RPN +L RCNN Where N fg is the number of foreground candidate boxes, s i is the predicted category of the candidate box, is the true value of the candidate box category, α represents the regression loss calculated only for the candidate box of the foreground, is the predicted candidate box regression parameter, is the true value of the candidate box regression parameter; Where N s is the number of candidate boxes sampled, c i is the confidence of the prediction, is the target value of confidence, β represents the regression loss calculated only for the candidate box of the sampled foreground, is the predicted candidate box regression parameter, is the true value of the candidate box regression parameter.

4. A detection method based on a laser point cloud three-dimensional target detection device for complex traffic scenes, It is characterized in that The steps include: S1, voxelizes the lidar point cloud input to the network; The input point cloud is preprocessed along the X, Y, and Z directions to divide the entire point cloud scene into uniform voxels. The range of the entire point cloud scene (X, Y, Z) is [(-75.2, 75.2), (-75.2, 75.2), (-5, 3)], and the size of each voxel is set to (0.1, 0.1, 0.2) meters. Therefore, the entire scene is divided into 1504×1504×40 voxels of equal size. If there are more than 5 original points in each voxel, it is downsampled to 5, and the average of the x, y, z, coordinates and reflection intensity of all points in each voxel after downsampling is used as the original feature of the voxel; S2, using the 3D encoder to extract features from non-empty voxels, and projecting the obtained voxel features onto the bird's-eye view to generate a pseudo image expression; The input of the 3D encoder is a 40×1504×1504 voxel representation. Each voxel has a 4-dimensional (x, y, z, reflection intensity) original feature. After the input, a 3×3×3 sub-manifold convolution is used to extract the voxel original feature and perform 1x downsampling. Then, two consecutive residual sub-manifold blocks are used for feature extraction. Then, three 3×3×3 sparse convolutions are used to perform 2x, 4x, and 8x downsampling along the X, Y, and Z directions respectively to obtain multi-scale 3D features. Each sparse convolution Each of them is followed by two consecutive residual sub-manifold blocks for feature extraction; the structure of the residual sub-manifold block is composed of two 3×3×3 sub-manifold convolutions, and the input of the first sub-manifold convolution and the output of the second sub-manifold convolution are added together. At the same time, each sub-manifold convolution is followed by a BatchNorm layer and a ReLU layer; finally, a sparse convolution is used to downsample the Z direction by 2 times, and then projected onto the bird's-eye view to convert it into a 256×188×188 pseudo image expression; S3, extracting features from the pseudo image expression using a two-dimensional encoder to obtain a two-dimensional feature expression; The 256×188×188 pseudo image representation obtained by the 3D encoder is downsampled by 1, 2, and 4 times using a 3×3 2D convolution to obtain multi-scale features. Then, two consecutive self-calibration convolutions are used after each 2D convolution to extract the multi-scale features, and the input of the first self-calibration convolution and the output of the second self-calibration convolution are added. The spatial attention and channel attention mechanisms are added after the self-calibration convolution to enhance the useful feature expression and reduce the useless information in the spatial and channel directions. Finally, the multi-scale features are returned to the same size through deconvolution and spliced ​​along the channel direction, and then a 2D convolution is used for dimensional compression to obtain a 64×188×188 2D feature expression. S4, uses two-dimensional features to classify and regress targets and generate high-quality candidate boxes; For the 2D feature representation of 64×188×188 obtained by the 2D encoder module, 10 anchor boxes are placed in each pixel of the 188×188 2D feature map, which are 5 categories: cars, trucks, buses, pedestrians, and cyclists. Each category has two directions of 0° and 180°. The 2D feature map is used to perform category prediction and size regression (x, y, z, w, h, l, θ) for each anchor box to obtain 10×188×188 3D boxes. During training, 9000 3D boxes with the highest classification scores are selected and sent to the non-maximum suppression module to obtain 512 high-quality candidate boxes. S5, perform voxel pooling on the candidate box to obtain pooled features; For the 512 high-quality candidate boxes generated by the candidate box generation module, 128 candidate boxes are randomly sampled and sent to the refinement stage, of which 64 are positive samples and 64 are negative samples. The voxel candidate box pooling module is used to extract multi-scale voxel features: first, 6×6×6 grid points are uniformly sampled for each candidate box, and then non-empty voxels within a certain distance around these grid points are queried. Then, pointnet, fully connected layers and maximum pooling layers are used to extract the features of these non-empty voxels to obtain pooled features; when querying the voxels around the grid points, 2x, 4x, and 8x downsampled voxels are queried respectively to obtain multi-scale voxel features; S6, based on the pooled features, the candidate boxes are refined using the fully connected layer; For the 128 candidate boxes randomly sampled by the candidate box pooling module, their pooling is used as input, and confidence prediction and regression box refinement are performed through 2 layers of fully connected layers. For the confidence prediction branch, the classification confidence c of each candidate box is predicted. i , its target is obtained by the following formula: in is the confidence target value of the i-th candidate box; IoU i is the intersection-over-union ratio of the i-th candidate box and its true value box, θ H ,θ L are the intersection-over-union ratio thresholds for foreground and background, which are 0.75 and 0.25 respectively; The intersection-over-union ratio of the predicted box and the true value is associated with the classification execution degree to alleviate the problem of mismatch between classification confidence and positioning accuracy. For the regression box refinement branch, more accurate location information (x, y, z, w, h, l, θ) is predicted through the fully connected layer. S7, visualization of detection effect; The candidate boxes of multiple targets in the refined point cloud road scene are first sorted from large to small according to their classification confidence, and the boxes with low confidence are filtered out. Then, the box with the highest score is selected from the remaining boxes, and the boxes whose intersection and union ratio with the box is greater than the threshold are deleted. Then, this non-maximum suppression operation is repeated in the remaining boxes to generate the final detection results and visualize them.

Images