Road target detection model and detection method based on Voxel RCNN Complex HD

Abstract

The invention discloses a Voxel RCNN Complex HD-based road target detection model and detection method, and proposes an improved three-dimensional feature extraction network and a two-dimensional feature extraction network for the problem that the Voxel R-CNN algorithm is insufficient in point cloud feature extraction. In order to solve the problem of low pedestrian detection precision, a mixed detection head with an anchor frame and a mixed detection head without an anchor frame are provided, and the detection precision of a large target and a small target is improved by using the mixed detection head based on an anchor and a mixed detection head without an anchor. The voxel candidate box self-attention pool module is provided for solving the problem that the difference between the detection precision of buses and trucks and the detection precision of trolleys in vehicle categories is large, local and global features of an object are better obtained, and therefore the size, the position and the course angle of a large target can be more accurately refined. And finally, the problem of insufficient detection accuracy of pedestrians, buses and trucks in a complex traffic scene is solved.

Description

Technical Field

[0001] The present invention belongs to the field of laser radar target detection algorithms, and in particular to a road target detection method based on Voxel RCNN Complex HD. Background Art

[0002] The unmanned driving system is mainly composed of three systems: perception, decision-making, and execution. Among them, the perception system mainly uses sensors such as cameras, lidar, and millimeter-wave radar to perceive the surrounding environment, and the global positioning system (GPS), inertial navigation, and high-precision maps to determine the position of the vehicle in the environment. The decision-making system analyzes and calculates the output information of the perception system, and performs task planning, behavior planning, and motion planning. Through task planning, the vehicle will get the path from the starting point to the end point; behavior planning determines what state the car should be in at each moment, such as normal driving, overtaking, changing lanes, etc.; motion planning determines the state of various actuators of the car, such as steering wheel angle, transmission gear, engine speed, etc. Finally, the execution system receives the output information of the decision-making system and acts on various actuators of the vehicle to achieve lateral and longitudinal control of the vehicle.

[0003] It can be seen that the perception system is the foundation of the driverless car system and ensures the safe operation of the vehicle. Compared with other sensors, LiDAR has the advantages of high precision and strong resistance to light interference. The road target detection algorithm based on LiDAR can accurately and stably perceive the category, size, position, heading angle and other information of obstacles. With the reduction of the cost of automotive-grade LiDAR and the continuous improvement of the computing power of on-board chips, LiDAR and its algorithms have begun to be mass-produced and installed on vehicles. Therefore, the target detection algorithm based on LiDAR is a major focus in the research of driverless car technology.

[0004] Smart vehicles collect information about the surrounding environment through sensors such as cameras and lidar, and then the vehicle system makes decisions and plans to ensure the normal operation of the vehicle. With the advancement of sensor technology and the continuous improvement of computer computing power, deep learning-based object detection algorithms have made significant progress in recent years. Deep learning-based algorithms can enable smart vehicles to accurately and quickly perceive surrounding obstacles such as vehicles, pedestrians, cyclists, etc. After several years of research, camera-based two-dimensional object detection and segmentation have matured to the point where they can achieve performance comparable to that of the human eye. However, camera-based two-dimensional detection is easily affected by environmental factors such as lighting and cannot adapt to complex and changeable autonomous driving scenarios. In addition, compared with 2D object detection, lidar-based 3D object detection can provide information about the position, size, and heading angle of an object relative to the self-vehicle. In addition, lidar has higher accuracy and is less affected by environmental factors. Lidar can provide smart vehicles with more accurate environmental information to ensure the normal operation of the vehicle. With the continuous reduction in the cost of lidar and the development of chip technology, lidar-based 3D object detection algorithms have rapidly improved and begun to be applied to vehicles. VoxelRCNN is a classic two-stage anchor-based 3D object detection algorithm that achieves a balance between speed and accuracy, but performs poorly for pedestrians, buses, and trucks in complex autonomous driving scenarios.

[0005] In unmanned driving scenarios, accurate and rapid detection of road traffic participants plays an important role in the safe operation of unmanned vehicles. Voxel R-CNN is a voxel-based, anchor-frame, two-stage LiDAR 3D target detection algorithm with fast detection speed and high accuracy, which is suitable for unmanned driving scenarios. However, the Voxel R-CNN algorithm is mainly trained and optimized on the KITTI dataset, and its detection scenarios are relatively simple, which is not suitable for the current complex traffic scenarios. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a Voxel R-CNN-Complex-HD algorithm for complex traffic scenes based on the Voxel R-CNN algorithm. Figure 1 The process schematic diagram shown, the main contents of the present invention are as follows:

[0007] (1) To address the problem of insufficient feature extraction of point cloud by the Voxel R-CNN algorithm, the three-dimensional feature extraction network and the two-dimensional feature extraction network were improved. A three-dimensional feature extraction network consisting of sparse convolution, submanifold convolution, and residual structure and a two-dimensional feature extraction network consisting of two-dimensional convolution, self-calibrated convolution, spatial and channel attention mechanism, and residual structure were constructed. These improvements enhance the feature extraction capability of the network and improve the detection effect of the algorithm for long-distance and small targets.

[0008] (2) To address the problem of low pedestrian detection accuracy of existing anchor frame detection heads, we utilize the advantages of high vehicle detection accuracy of anchor frame detection heads and high pedestrian detection accuracy of non-anchor frame detection heads, and propose a hybrid detection head with and without anchor frames. That is, different detection heads are used for different categories of targets. This improves the pedestrian detection accuracy while ensuring the vehicle detection accuracy.

[0009] (3) To address the problem that the voxel candidate box pooling module cannot obtain the global features of very large targets, a voxel candidate box self-attention pooling module is proposed. The self-attention mechanism is introduced to simultaneously obtain the local features and global features of the candidate box, thereby improving the detection accuracy of two types of very large targets, trucks and buses.

[0010] Therefore, to address the problems of low detection accuracy of long-distance and small targets, small sample size and category imbalance in the ONCE dataset, the present invention proposes a Voxel R-CNN-Complex HD algorithm for complex traffic scenes. It mainly proposes improved three-dimensional and two-dimensional feature extraction networks to enhance the feature extraction capability of the algorithm, thereby improving the detection accuracy of the algorithm for long-distance and small targets. It adopts a category balanced sampling strategy to increase the data volume of the dataset, alleviate the imbalance of target categories, and enhance the robustness of the algorithm.

[0011] According to whether anchors are used to enumerate targets, object detection algorithms can be divided into anchor-based algorithms and anchor-free algorithms. On the ONCE dataset, anchor-free algorithms perform better for pedestrian detection, while anchor-based algorithms perform better for vehicle detection. Anchor-based algorithms place two anchors of 0 and 90 degrees at each pixel of the 2D feature map to fit the object. Anchor-based algorithms are easy to train and have fast network convergence. For vehicle detection, since most vehicles in autonomous driving scenarios are perpendicular or parallel to the ego vehicle, two anchors of 0 and 90 degrees can fit the object well through the neural network. However, the direction of roadside pedestrians is usually 360 degrees, which cannot be enumerated by two anchor boxes. When more anchor boxes are set and the angle is larger, the algorithm's inference speed will decrease, and the imbalance of positive and negative samples will be more serious, which will affect the detection results. In addition, it is easier for anchor-based algorithms to match large objects than small objects, resulting in good performance for vehicles but poor performance for pedestrians. Compared with anchor-based algorithms, anchor-free algorithms use heat maps to predict the center point of an object, and then use the features of the center point to directly predict its size, position, and heading angle. However, anchor-free algorithms have several disadvantages: (1) The network converges slowly on datasets with fewer training samples, such as ONCE and KITTI; (2) When faced with large objects (such as vehicles), it is difficult to regress their position and size well using center point features; (3) Since the lidar emits a laser beam to an obstacle, it only returns a point cloud on the surface of the object, but cannot obtain a point cloud at the center of the object, so it loses features near the center of the object. Therefore, anchor-free algorithms cannot accurately predict the center point of larger objects (such as vehicles). There is a bias when using center point features to predict the size of an object. Moreover, pedestrians in autonomous driving scenarios are small in size and usually only occupy one grid on the feature map, so grid features can well predict the center position, size, and heading angle of pedestrians. From the above analysis, it can be seen that anchor-based algorithms perform better for large objects, while anchor-free algorithms perform better for small objects.

[0012] Therefore, based on the advantages and disadvantages of anchor and anchor-free algorithms, the present invention proposes a hybrid detection head based on anchor and anchor-free. The anchor-based detection head is used for four types of larger objects, including cars, buses, trucks and cyclists. The anchor-free detection head is used for pedestrians. This hybrid detection head achieves a balance between large objects and small objects.

[0013] In order to refine the position, size and heading angle of the candidate box, the second-stage network obtains the features of the candidate box generated in the first stage through the candidate box pooling module. The two-stage network has a significant effect on improving the detection accuracy of the object detection algorithm. In the second stage, PV-RCNN and Voxel RCNN sow 6×6×6 uniform grid points along the length, width and height of the candidate box, and then encode the key point features or voxel features around each grid point to the grid point, and finally predict the confidence of the box through the fully connected layer and refine the position, size and angle of the candidate box. For small objects, each grid point encodes the key points or voxel features near it to obtain the global features of the object. For large objects such as trucks and buses, each grid point can only encode partial features of the object. However, global features are crucial for the correction of objects. CT3D encodes the original point cloud and obtains the global features of the candidate box in the two-stage network through the self-attention mechanism. However, as mentioned in Lidar RCNN, encoding features from the original points will lead to the ambiguity of the candidate box, which will lose the size information of the candidate box and is not conducive to the correction of the candidate box. According to the above analysis, the present invention proposes a voxel candidate box self-attention pool module based on Voxel RCNN and CT3D algorithms. On the basis of encoding the surrounding voxels at the grid point and obtaining the local features of the object, the Self-Attention mechanism is used to expand the receptive field of each grid point, encode the local and global features of the candidate box at each grid point, and retain the size information of the candidate box. The proposed voxel candidate box self-attention pool module improves the detection performance of large objects (including buses and trucks).

[0014] The present invention proposes the following technical contents of road target detection based on Voxel RCNN Complex HD:

[0015] A. Detection method flow using Voxel RCNN Complex HD

[0016] Voxel RCNN is an anchor-based two-stage algorithm for 3D object detection. The first stage involves voxelization of the point cloud along the X, Y, and Z axes, followed by a 3D backbone network consisting of sparse convolutions and submanifold convolutions. Submanifold convolutions are used to extract features from non-empty voxels, while sparse convolutions are used to expand the receptive field. The 3D backbone network consisting of sparse and submanifold convolutions significantly improves the processing speed of point clouds. Voxels are downsampled by 1×, 2×, 4×, and 8× through the 3D backbone network to obtain multi-scale 3D voxel features, which are then projected onto a bird's-eye view to form a 2D pseudo image. The 2D backbone network consisting of 2D convolutions is used to extract features from the 2D pseudo image and form a 2D feature map. Two anchor boxes with angles of 0 and 90 degrees are placed on each pixel of the 2D feature map of each category. The input of the detection head is the anchor box and the 2D feature map, and the classification score and box regression parameters of each anchor box are predicted. Then, based on the classification score, high-quality candidate boxes are retained. In the second stage, 6×6×6 grid points are uniformly sown along the length, width and height of each candidate box, and each grid point aggregates the multi-scale voxel features around it. After the voxel candidate box pooling module, each candidate box obtains the pooled features, and then uses the fully connected layer for confidence prediction and box refinement.

[0017] However, when faced with complex traffic conditions, the detection performance of the Voxel RCNN algorithm is insufficient. Specifically, the detection accuracy of pedestrians, trucks, and buses is low. Therefore, the present invention proposes Voxel RCNN Complex HD as a variant of VoxelRCNN. The network structure of Voxel RCNN Complex HD is as follows: Figure 2 As shown, the red rectangle represents the proposed new module. In the first stage of Voxel RCNN Complex HD, a hybrid detection head based on anchors and anchor-free is proposed; each detection head predicts a category, where four anchor-based detection heads predict cars, buses, trucks and cyclists respectively, and one anchor-free detection head predicts pedestrians. This can ensure the detection performance of both pedestrians and vehicles. In the second stage, a voxel candidate box self-attention pooling module is proposed, which introduces a self-attention mechanism to encode the local and global features of each grid point, significantly improving the detection performance of large objects such as trucks and buses.

[0018] The improvements in each part are as follows:

[0019] B. Improved 3D feature extraction network and 2D feature extraction network

[0020] The feature extraction network is the "skeleton" of the entire target classification and detection algorithm, also known as the backbone. It has a very important impact on the algorithm, and its ability to extract features directly determines the accuracy of the algorithm. In the original Voxel R-CNN algorithm, its three-dimensional feature extraction network simply stacked sparse convolutions and sub-manifold convolutions. The present invention has improved it by replacing the sub-manifold module with a residual sub-manifold module to obtain more complete features. The residual sub-manifold module adds a shortcut between the input and output of each convolution, which makes it easier to maintain feature integrity during voxel downsampling and makes the network easier to optimize. In addition, when the sparse convolution is downsampled 8 times, the voxel features are increased from 64 dimensions to 128 dimensions, and richer information is extracted through high-dimensional features, thereby improving the detection performance of the algorithm in complex scenes. The structure of the improved three-dimensional feature extraction network is shown below. Figure 3 As shown in the figure, the structure of the improved 3D feature extraction network is similar to the original 3D feature extraction network, but the original submanifold module is replaced by the residual module, and the voxel features are up-scaled when downsampling by 8 times. The structure of the residual submanifold module-1 is specifically introduced. For the input features of the residual submanifold module-1, two 3×3×3 submanifold convolutions are used for feature extraction. A shortcut branch is added between the input of the first submanifold convolution and the output of the second submanifold convolution, and the shortcut branch is added to the corresponding position elements of the original branch output feature map. The structures of the remaining residual modules are the same as those of the residual module-1.

[0021] For target detection tasks, it is necessary to regress the position and size of the target and classify the target. The bottom-level large-size feature map contains accurate position information and weaker semantic information, and the top-level small-size feature map contains high-dimensional semantic information and a larger receptive field. Therefore, the bottom-level large-size feature map can detect small-size targets or targets with simple features, and the top-level small-size feature map can detect large-size targets or targets with complex features. The two-dimensional feature extraction network of the original Voxel R-CNN algorithm stacks two groups of two-dimensional convolutions, and fuses features of two scales through downsampling, feature extraction, and upsampling. However, this method of directly stacking two groups of two-dimensional convolutions for feature extraction is not conducive to the extraction of complex scene features, and is not conducive to the deep fusion of the position features of the small-size feature map and the semantic features of the large-size feature map, resulting in the algorithm being unable to meet the detection task of five sizes of targets in complex scenes. Therefore, the present invention improves the two-dimensional feature extraction network and designs a two-dimensional feature extraction network composed of two-dimensional convolution, self-calibration convolution, channel attention and spatial attention mechanisms, and a residual structure. Its structure is as follows: Figure 4 shown.

[0022] The original pseudo-image expression is downsampled by 1, 2, and 4 times using three 3×3 two-dimensional convolutions to obtain features of more scales. Then, two consecutive self-calibration convolutions are used after each two-dimensional convolution to extract features in the self-calibration space and the original scale space, respectively. This not only allows feature fusion, but also expands the receptive field. A shortcut branch is added between the input of the first self-calibration convolution and the output of the second self-calibration convolution, which is conducive to maintaining the flow of features and gradients. In addition, spatial attention and channel attention mechanisms are added at the end of each convolution group to filter out useless features and enhance useful feature expressions. Finally, the features of the three scales are upsampled to the same size through transposed convolution for feature fusion. Through these operations, the point cloud features of complex scenes can be fully extracted, which is conducive to the generation of high-quality regression boxes.

[0023] C. A hybrid detection head based on anchor and anchor-free is proposed

[0024] The anchor-based detection head presets a large number of prior frames on the feature map to fit the target, which are also called anchor frames. Compared with directly fitting the target bounding box, it is simpler for the algorithm to fit the target bounding box based on the anchor frame. Since the two-dimensional target detection algorithm Faster-RCNN was proposed, the idea of ​​fitting the target with anchor frames has been widely used in target detection algorithms. Anchor frames are some of the most representative frames among all true value frames. In the two-dimensional target detection algorithm, the size of the anchor frame is obtained using algorithms such as K-Means clustering; in the three-dimensional target detection algorithm, the size of the true value bounding box of each category is averaged to obtain the size of the anchor frame of each category.

[0025] In the present invention, Figure 5 As shown in Figure 1, the anchor-based detection head places multiple anchor boxes in each grid of the feature map, placing one 0-degree and 90-degree anchor box for each category, and then predicts its position, size, heading angle offset and confidence through two-dimensional convolution. Finally, the position, size and heading angle of the predicted candidate box are calculated through the predicted offset and anchor box information, and the candidate boxes with low confidence are filtered out to obtain high-quality predicted candidate boxes. The calculation formula for generating candidate boxes with anchor box detection head is shown in formula (1).

[0026] x p =△x·d a +x a ,y p =△y·d a +y a ,z p =△z·h a +z a ,

[0027] w p =w a ·e△w ,l p = l a ·e △l ,h p =h a ·e △h , (1)

[0028]

[0029] where x p ,y p 、z p is the center coordinate of the predicted candidate box, w p , l p 、h p To predict the length, width and height of the candidate box, θ p is the heading angle of the predicted candidate box, x a ,y a 、z a is the center coordinate of the anchor box, w a , l a 、h a is the length, width and height of the anchor box, θ a is the heading angle of the anchor frame, d a is the diagonal length of the bottom of the anchor frame. △x, △y, △z are the center coordinates of the predicted candidate frame and the anchor frame, △w, △l, △h are the length, width and height of the predicted candidate frame and the anchor frame, and △θ is the offset of the heading angle between the predicted candidate frame and the anchor frame.

[0030] YOLOV1 is an anchor-free detection algorithm, but its accuracy is low. Therefore, in the subsequent improved algorithms, the detection heads of the YOLO series are replaced with anchor-free detection heads. However, since the Centernet algorithm has made a major breakthrough in the field of two-dimensional object detection, its idea of ​​anchor-free detection head based on heat map has been widely used in various algorithms and expanded to the field of three-dimensional object detection, forming algorithms such as Centernet3D, CenterPoints, and AFDet.

[0031] In the anchor-free detection algorithm based on heatmap, generating the true value heatmap is the key step of the algorithm. First, the center point of each target is calculated and rounded, then the Gaussian radius is calculated according to the size of the target, and the Gaussian scattering kernel is generated for each target with the center point as the center and the Gaussian radius as the radius. Finally, the Gaussian scattering kernels of all targets of the same category are projected onto the same map to form the true value heatmap of each category.

[0032] The anchor-free box detection algorithm based on heat map of the present invention has two branches, which are used for heat map prediction and bounding box regression respectively. The heat map prediction branch regards all targets as center points, predicts a heat map for each category, finds the grid where the target center is located by finding the peak of the heat map, and uses the heat value corresponding to the grid as the confidence of the target. The bounding box regression branch predicts the offset of the target center in the X and Y axis directions of the grid, the position in the Z axis direction, the size regression parameters, the direction regression parameters and other information to obtain the final predicted candidate box, and its calculation formula is shown in formula (2).

[0033] x p =x grid +reg x ,y p =y grid +reg y ,z p =reg z ,

[0034]

[0035] where x p ,y p 、z p is the center coordinate of the predicted candidate box, w p , l p 、h p To predict the length, width and height of the candidate box, θ p To predict the heading angle of the candidate box, reg x , reg y , reg z , reg w , reg l , reg h , reg rotc , reg rots is the regression parameter, specifically: reg x , reg y , reg z They represent the regression parameters of the target position with the vehicle as the coordinate origin (x, y, and z represent the x-axis direction, y-axis direction, and z-axis direction, respectively), reg l , reg w , reg h Respectively represent the regression parameters corresponding to the length, width and height of the detection box, reg rotc , reg rots Respectively represent the cosine (cos) and sine (sin) regression parameters of the heading angle, x grid ,y grid The coordinates of the grid where the center point of the predicted candidate box is located.

[0036] On the Huawei ONCE dataset, for complex unmanned driving scenarios, the anchor-free algorithm has a higher detection accuracy for pedestrians and a lower detection accuracy for vehicles, while the anchor-within-a-frame algorithm has a higher detection accuracy for vehicles and a lower detection accuracy for pedestrians.

[0037] The anchor frame algorithm places two anchor frames of 0 degrees and 90 degrees for each category of target at each position in the two-dimensional feature map to fit the target. This type of algorithm is easy to train and the neural network converges quickly. Especially for vehicle detection, most vehicles in unmanned driving scenarios are perpendicular or parallel to the vehicle, so the two anchor frames of 0 degrees and 90 degrees can better fit the target by training the neural network. However, for pedestrians on the roadside, their orientation is usually 360 degrees, and it is impossible to fit them with anchor frames of two angles. If more anchor frames are set, the algorithm's reasoning speed will slow down and the imbalance of positive and negative samples will be aggravated, thus affecting the detection effect. In addition, the anchor frame-based algorithm is easier to match with large targets and less likely to match with small targets, which also results in the anchor frame algorithm having good results for vehicle detection and poor results for pedestrian detection.

[0038] Unlike the anchor-based algorithm, the anchor-free algorithm predicts the center point of the target through the heat map, and directly regresses its size, position and heading angle through the features of the center point. However, the anchor-free algorithm also has its disadvantages. The neural network converges slowly. On datasets with fewer training samples, such as KITTI and ONCE datasets, when facing large targets such as vehicles, the center point features cannot be used to regress their position and size well. However, it performs well on datasets with more training samples such as Waymo and nuScenes. In addition, since the laser radar emits a laser beam to the obstacle, the returned point cloud is only the point on the surface of the target, and the point cloud at the center of the target cannot be obtained, which will cause the loss of features near the center of the target. Therefore, for large targets such as vehicles, the anchor-free algorithm cannot predict the center point of the target well. Then, the prediction of the size, position, heading angle and other information of the target through the features of the center point will also be biased, and the "target spinning" situation will often occur. However, the volume of pedestrians is small, and generally only occupies one grid on the feature map. The center point and size of pedestrians can be better predicted through the feature information of the grid. Therefore, the anchor-free algorithm has a better detection effect on pedestrians, but a poor detection effect on vehicles.

[0039] Anchor-based algorithms use anchor boxes to enumerate possible locations of objects. However, anchor-based algorithms perform poorly for pedestrians due to the flexible and changeable heading angles of pedestrians. Anchor-free algorithms use the features of center points to regress objects, but point clouds only exist on the surface of objects. For large objects such as vehicles, some center point features will be lost. In response to the low detection accuracy of anchor-based algorithms for pedestrians and the low detection accuracy of anchor-free algorithms for vehicles, this algorithm proposes an anchor-based hybrid detection head in the first stage. The hybrid detection head can improve the performance of pedestrian detection while ensuring the accuracy of vehicle detection, while taking advantage of the advantages of anchor-based and anchor-free algorithms.

[0040] like Figure 3 As shown in the figure, the anchor-based and anchor-free hybrid detection head module takes a 2D feature map as input and generates five new feature maps using five 2D convolutions. The five new feature maps are sent to five detection heads, of which the first four are anchor-based detection heads, used to predict cars, buses, trucks, and cyclists, respectively, and the fifth is an anchor-free detection head, used to predict pedestrians. For each anchor-based detection head, two anchor boxes with angles of 0 and 90 degrees are placed at each pixel of the feature map, and there are two branches. One is used to classify the anchor point, whether it belongs to the foreground or the background, and the other is used to regress the position, size, and heading angle parameters. Candidate boxes are created using parameters and anchors, and the anchor points are fine-tuned according to the predicted parameters. The anchor-free detection head predicts heat maps and regresses parameters through feature maps. The location of the peak of the heat map represents the center of the predicted object, and the peak also represents the classification score. At the same time, the location, size, and heading angle of the object are directly predicted by the features of the center point. Finally, the output boxes of the five detection heads are combined and sorted from high to low by the classification score. The algorithm selects the 9,000 boxes with the highest classification scores and sends them to the Non-Maximum Suppression (NMS) module, which generates 512 high-quality proposal boxes for the second stage.

[0041] D. Improved voxel candidate box self-attention pooling module

[0042] The voxel candidate box pooling module evenly sets 6×6×6 grid points along the length, width, and height of each candidate box, and then queries the non-empty voxels within a certain distance around these grid points. For each query grid point, according to its voxel coordinates (i q ,j q ,k q ), translate its voxel coordinates within a certain range, and the translation distances on the three coordinate axes of the voxel coordinate system are △i, △j, and △k respectively. Find the non-empty voxels around the grid point, and the query range is shown in formula (3).

[0043] |△i|+|△j|+|△k|≤d (3)

[0044] Where d is the distance threshold.

[0045] The original voxel candidate pooling module uniformly sows 6×6×6 grid points along the length, width, and height, and each grid point encodes the features of the surrounding voxels. With this encoding method, for small objects, each grid point can obtain the global features of the object. However, the length and height of two large objects, trucks and buses, are mostly around 10 meters and 3 meters. As a result, each grid point can only encode the local features of the object and cannot accurately refine the bounding box. In addition, the small angle deviation of large objects will result in a small intersection over union (IoU) between the predicted box and the ground truth box. To solve this problem, the algorithm uses the Self-Attention mechanism to calculate the relationship between each grid point, and each grid point can encode the features of other grid points. Each grid point can obtain the local and global features of the object through the voxel candidate self-attention pooling module, so that the size, position, and heading angle of large objects can be more accurately refined.

[0046] Figure 6 The structure of the voxel candidate box self-attention pooling module is illustrated. Using the high-quality proposal box generated in the first stage and the voxel features of the 3D backbone network as input, the voxel features around the grid point are encoded onto the grid point through the voxel candidate box pooling module, and each grid point can obtain the local feature F local . Local feature F local Multiply by matrix W q , W k and W v The features Q, K, and V are obtained respectively. The inner product of features Q and K is used to determine the relationship between the grid point features and other grid point features. The weight is then calculated using the Softmax function and multiplied by V, so that each grid point obtains the global feature F of the object. global Finally, the local feature F local and the global feature F global The grid point features F that are combined to form the proposal grid .

[0047] Q=F local ·W q , K=F local ·W k , V=F local ·W v (4)

[0048]

[0049] F grid =F local +F global (6)

[0050] where d is the length of features Q and K.

[0051] E. Design loss function

[0052] The loss function L of the Voxel R-CNN-Complex-HD algorithm TOTAL The one-stage region proposal network loss L RPN and the two-stage regional convolutional neural network loss L RCNN The calculation formula is shown in formula (7).

[0053] L TOTAL =L RPN +L RCNN (7)

[0054] The region proposal network loss L RPN Including the anchor-free detection head loss L AF and the anchor box detection head loss L AB . Regional Convolutional Neural Network Loss L RCNN Including confidence prediction loss L c And the candidate box position correction loss L locref .L RPN , L RCNN , L AF , L AB The calculation formulas are shown in equations (8) to (11) respectively.

[0055] L RPN =λ 1 L AF +L AB (8)

[0056] L RCNN =L c +λ 2 L locref (9)

[0057] L AF =L HM +λ 3 L locaf (10)

[0058] L AB =L cls +λ 4 L locab (11)

[0059] The loss of the anchor-free detection head is L AF The heat map loss L HM And the anchor-free candidate box position loss L locaf Composition, with anchor box detection head loss LAB By the classification loss L cls And the candidate box position loss L with anchor box locab This algorithm uses Gaussian Focal Loss to calculate the heat map loss L HM , Focal Loss calculates the classification loss L cls , binary cross entropy loss calculates the confidence prediction loss L c , smooth l1 Loss calculates the position loss L of the candidate box without anchor box locaf , the candidate box position loss L with anchor box locab And the candidate box position correction loss L locref In the formula, λ 1 , 2 , 3 , 4 They are all loss balance coefficients, which are 0.2, 1, 2, and 2 respectively.

[0060] Beneficial effects of the present invention:

[0061] 1. Based on Voxel RCNN, the present invention proposes an improved algorithm Voxel RCNN Complex HD. Aiming at the problem that the Voxel R-CNN algorithm does not sufficiently extract point cloud features, an improved three-dimensional feature extraction network and a two-dimensional feature extraction network are proposed to enhance the feature extraction capability of the algorithm, thereby improving the detection accuracy of the algorithm for long-distance targets and small targets.

[0062] 2. In order to solve the problem that the detection accuracy of buses and trucks in the vehicle category is quite different from that of cars, the present invention proposes an improved voxel candidate box self-attention pool module to better obtain the local and global features of objects, so as to more accurately refine the size, position and heading angle of large targets.

[0063] 3. In order to solve the problem of low pedestrian detection accuracy, the present invention proposes a hybrid detection head with and without anchor frames, and uses a hybrid detection head based on anchors and without anchors to improve the detection accuracy of large targets such as trucks and small targets such as pedestrians. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a general schematic diagram of the present invention;

[0065] Figure 2 This is the network structure diagram of Voxel RCNN Complex HD;

[0066] Figure 3 To improve the network structure of three-dimensional feature extraction;

[0067] Figure 4An improved two-dimensional feature extraction network;

[0068] Figure 5 This is a diagram of the anchor-based and anchor-free hybrid detection head module;

[0069] Figure 6 Module diagram of the voxel candidate box self-attention pool. DETAILED DESCRIPTION

[0070] The present invention will be further described below in conjunction with the embodiments.

[0071] 1. Road object detection based on Voxel RCNN Complex HD

[0072] A. Dataset

[0073] The dataset used in the present invention is the Huawei ONCE dataset, which was collected in China using HeSai Pandar 40-line laser radar and 7 cameras to capture autonomous driving scenes under different times, road conditions and weather conditions. Compared with the KITTI dataset, the scenes of the ONCE dataset are more complex. The ONCE dataset contains a larger number of objects per frame. The KITTI dataset contains a single forward-looking perspective, while the ONCE dataset contains a 360-degree scene. Therefore, the ONCE dataset is more suitable for the perception needs of intelligent vehicles in autonomous driving scenarios. ONCE contains more than one million unlabeled samples and more than 16,000 labeled samples, which can be used for unsupervised, semi-supervised and supervised learning. The algorithm is classified as supervised learning; it divides 16,000 labeled samples into three groups: 5,000 for training, 3,000 for verification, and 8,000 for testing. According to the evaluation script officially provided by the ONCE dataset, the detection accuracy of five types of objects, cars, trucks, buses, cyclists, and pedestrians, can be evaluated. In addition, cars, trucks, and buses can be combined to evaluate the detection accuracy of three types of objects: vehicles, cyclists, and pedestrians.

[0074] B. Experimental Details and Setup

[0075] On the ONCE dataset, the Voxel RCNN Complex HD algorithm is compared with Voxel RCNN and other classic 3D object detection algorithms to demonstrate its effectiveness. Among them, the mAP of PointPainting, PointRCNN, PointPillars, SECOND, PV-RCNN, and CenterPoints is provided by the ONCE benchmark. Voxel RCNN and Voxel RCNN Complex HD are trained and tested on a workstation equipped with two Nvidia RTX 2080Ti GPUs, i7-9700kCPU, and 64GB memory. It takes about 28 hours to train the network. The platform is Ubuntu 18.04, the framework is Pytorch 1.4, the programming language is python 3.7, the batch size is 4, the epoch is 80, the initial learning rate is 0.003, the learning rate strategy is cosine annealing, and the optimizer is adamW. Focusing on the self-mounted laser radar, the detection range is 75.2 meters in front, behind, left, and right, 3 meters up, and 5 meters down. The whole scene is divided into uniform voxels, each with a size of 0.1×0.1×0.2 meters.

[0076] When training Voxel RCNN Complex HD, the loss balance coefficient λ 1 As shown in Table 1, larger or smaller λ 1 will lead to a significant decrease in mAP. 1 Used to balance the anchor-free and anchor-based detection heads. Larger λ 1 The gradient of the anchor-free branch is larger and the smaller λ 1 Make the loss focus on the anchor-based branch, regardless of whether λ is large or small 1 The network cannot converge well. 1 Set to 0.2, it has the best detection performance. 2 , 3 , 4 The setting follows Voxel-RCNN and Centerpoints, i.e. 1,2,2.

[0077] Table 1 Different λ 1 Results of Voxel RCNN Complex HD

[0078]

[0079] C. Experimental Results

[0080] Table 2 compares the detection accuracy and speed of Voxel RCNN Complex HD with other algorithms on the ONCE dataset. The mAP of PointPainting, PointRCNN, PointPillars, SECOND, PV-RCNN, and CenterPoints are taken from the official benchmark of the ONCE dataset. The speed of all algorithms is trained under the default configuration and calculated on an NVIDIA RTX 2080Ti GPU. As shown in Table 2, anchor-based algorithms such as SECOND, PVRCNN, and VoxelRCNN outperform anchor-free algorithms for vehicles. In contrast, for pedestrians, anchor-free algorithms such as CenterPoints outperform anchor-based algorithms. Compared with the original Voxel RCNN, Voxel RCNN Complex HD improves the accuracy of vehicles, pedestrians, cyclists, and mAP by 1.11%, 7.19%, 3.04%, and 3.78%, respectively. The mAP of Voxel RCNN Complex HD is 9.01%, 38.05%, 22.45%, 14.9%, 13.24% and 6.74% higher than PointPainting, PointRCNN, PointPillars, SECOND, PV-RCNN and CenterPoints. By comparing Voxel RCNN Complex HD with other algorithms, it can be concluded that Voxel RCNN Complex HD can improve the accuracy of pedestrian detection while maintaining the accuracy of vehicle detection, which proves the effectiveness of the algorithm.

[0081] Table 2 Comparison of Voxel RCNN Complex HD and other algorithms on the ONCE dataset

[0082]

[0083] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. The road object detection model based on Voxel RCNN Complex HD is characterized by: include: Improved 3D feature extraction network and 2D feature extraction network for feature extraction; A hybrid detection head with and without anchor frames to detect pedestrians and vehicles at mixed anchor points; Self-attention pooling module using voxel proposals for very large object detection.

2. The road object detection model based on Voxel RCNN Complex HD according to claim 1, characterized in that: The improved three-dimensional feature extraction network uses a residual submanifold module instead of a submanifold module to obtain complete features. The residual submanifold module adds a shortcut between the input and output of each convolution, which can maintain the integrity of the features during voxel downsampling and make the network easy to optimize. In addition, when the sparse convolution is downsampled 8 times, the voxel features are increased from 64 dimensions to 128 dimensions, and more abundant information is extracted through high-dimensional features, thereby improving the detection performance of the algorithm in complex scenes. The residual submanifold module has input features extracted by two 3×3×3 submanifold convolutions, adds a shortcut branch between the input of the first submanifold convolution and the output of the second submanifold convolution, and adds the shortcut branch to the corresponding position elements of the original branch output feature map.

3. The road object detection model based on Voxel RCNN Complex HD according to claim 1, characterized in that: The improved two-dimensional feature extraction network includes two-dimensional convolution, self-calibration convolution, channel attention and spatial attention mechanisms, and residual structure. The original pseudo-image expression is downsampled by 1, 2, and 4 times using three 3×3 two-dimensional convolutions to obtain features of more scales. Then, two consecutive self-calibration convolutions are used after each two-dimensional convolution to extract features in the self-calibration space and the original scale space, respectively, which can not only perform feature fusion but also expand the receptive field. A shortcut branch is added between the input of the first self-calibration convolution and the output of the second self-calibration convolution, which is conducive to maintaining the flow of features and gradients. Spatial attention and channel attention mechanisms are added at the end of each convolution group to filter out useless features and enhance useful feature expressions. Finally, the features of the three scales are upsampled to the same size through transposed convolution for feature fusion. Through these operations, the point cloud features of complex scenes can be fully extracted, which is conducive to the subsequent generation of high-quality regression frames.

4. The road object detection model based on Voxel RCNN Complex HD according to claim 1, characterized in that: The detection head with anchor frames places multiple anchor frames in each grid of the feature map, places one 0-degree and 90-degree anchor frame for each category, and then predicts their position, size, heading angle offset and confidence through two-dimensional convolution. Finally, the position, size and heading angle of the predicted candidate frame are calculated through the predicted offset and anchor frame information, and the candidate frames with lower confidence are filtered out to obtain high-quality predicted candidate frames.

5. The road object detection model based on Voxel RCNN Complex HD according to claim 4, characterized in that: The calculation formula for generating candidate boxes with anchor box detection head is shown in formula (1): where x p ,y p 、z p is the center coordinate of the predicted candidate box, w p , l p 、h p To predict the length, width and height of the candidate box, θ p is the heading angle of the predicted candidate box, x a ,y a 、z a is the center coordinate of the anchor box, w a , l a 、h a is the length, width and height of the anchor box, θ a is the heading angle of the anchor frame, d a is the diagonal length of the bottom of the anchor box. △x, △y, △z, △w, △l, △h, △θ are the offsets of the center coordinates, length, width, height, and heading angle of the predicted candidate box and the anchor box.

6. The road object detection model based on Voxel RCNN Complex HD according to claim 4, characterized in that: The anchor-free box detection head adopts an anchor-free box detection algorithm based on a heat map, which has two branches, one for heat map prediction and the other for bounding box regression. The heat map prediction branch regards all targets as center points, predicts a heat map for each category, finds the grid where the target center is located by finding the peak of the heat map, and uses the heat value corresponding to the grid as the confidence of the target. The bounding box regression branch predicts the offset of the target center in the X and Y axis directions of the grid, the position in the Z axis direction, the size regression parameters, and the direction regression parameters to obtain the final predicted candidate box.

7. The road object detection model based on Voxel RCNN Complex HD according to claim 6, characterized in that: The hybrid detection head with anchor boxes and without anchor boxes takes the two-dimensional feature map as input and generates five new feature maps using five two-dimensional convolutions. The five new feature maps are sent to five detection heads, of which the first four detection heads are anchor-based detection heads, which are used to predict cars, buses, trucks and cyclists respectively, and the fifth detection head is an anchor-free detection head, which is used to predict pedestrians. For each anchor-based detection head, two anchor boxes with angles of 0 and 90 degrees are placed at each pixel of the feature map, and there are two branches, one for classifying the anchor point as belonging to the foreground or the background, and the other for regressing the position, size and heading angle parameters. Candidate boxes are created using the parameters and anchor points, and the anchor points are fine-tuned according to the predicted parameters. The anchor-free detection head is used to predict the heat map and regression parameters through the feature map. The position of the heat map peak represents the center of the predicted object, and the peak also represents the classification score. At the same time, the position, size and heading angle of the object are directly predicted through the features of the center point. Finally, the output frames of the five detection heads are combined and sorted from high to low according to the classification scores. The frames with high scores are selected and sent to the non-maximum suppression (NMS) module to generate 512 high-quality proposal frames.

8. The road object detection model based on Voxel RCNN Complex HD according to claim 7, characterized in that: The self-attention pooling module using the voxel candidate box uses the generated high-quality proposal box and the voxel features of the 3D backbone network as input, and encodes the voxel features around the grid point to the grid point through the voxel candidate box pooling module, and each grid point can obtain the local feature F local , local feature F local Multiply by the weight matrix W q , W k and W v The features Q, K and V are obtained respectively. The inner product of features Q and K is used to determine the relationship between the grid point features and the features of other grid points. Then the weight is calculated using the Softmax function and multiplied by V, so that each grid point obtains the global feature F of the object. global ; Finally, the local feature F local and the global feature F global The grid point features F that are combined to form the proposal grid , the calculation formula of this process is as follows: Q=F local ·W q ,K=F local ·W k ,V=F local ·W v (4) F grid =F local +F global (6) where d is the length of features Q and K.

9. The road object detection model based on Voxel RCNN Complex HD according to claim 1, characterized in that: The loss function L of the detection model TOTAL The one-stage region proposal network loss L RPN and the two-stage regional convolutional neural network loss L RCNN The calculation formula is shown in formula (7) L TOTAL =L RPN +L RCNN (7) Among them, the region proposal network loss L RPN Including the anchor-free detection head loss L AF and the anchor box detection head loss L AB , regional convolutional neural network loss L RCNN Including confidence prediction loss L c And the candidate box position correction loss L locref ; L RPN , L RCNN , L AF , L AB The calculation formulas are shown in equations (8) to (11) respectively. L RPN =λ1L AF +L AB (8) L RCNN =L c +λ2L locref (9) L AF =L HM +λ3L locaf (10) L AB =L cls +λ4L locab (11) Among them, the anchor-free detection head loss L AF The heat map loss L HM And the anchor-free candidate box position loss L locaf Composition, with anchor box detection head loss L AB By the classification loss L cls And the candidate box position loss L with anchor box locab Composition, use Gauss FocalLoss to calculate the heat map loss L HM , Focal Loss calculates the classification loss L cls , binary cross entropy loss calculates the confidence prediction loss L c , smooth l1 Loss calculates the position loss L of the candidate box without anchor box locaf , the candidate box position loss L with anchor box locab And the candidate box position correction loss L locref , λ1, λ2, λ3, and λ4 are all loss balance coefficients.

10. The detection method of the road object detection model based on Voxel RCNN Complex HD according to any one of claims 1 to 9, characterized in that: It includes the following two parts: The first part of the detection adopts a two-stage detection based on anchoring: In the first stage, the point cloud is voxelized along the X, Y, and Z axes, followed by a 3D backbone network consisting of sparse convolution and submanifold convolution, i.e., a 3D feature extraction network. Submanifold convolution is used to extract features from non-empty voxels, and sparse convolution is used to expand the receptive field. The 3D backbone network consisting of sparse and submanifold convolution can improve the processing speed of point clouds. The voxels are downsampled by 1×, 2×, 4×, and 8× through the 3D backbone network to obtain multi-scale 3D voxel features, which are then projected onto the bird's-eye view to form a 2D pseudo image; The 2D backbone network composed of 2D convolutions, i.e., the 2D feature extraction network, is used to extract the features of the 2D pseudo-image and form a 2D feature map. Two anchor boxes with angles of 0 and 90 degrees are placed on each pixel of the 2D feature map of each category. The input of the detection head is the anchor box and the 2D feature map, and the classification score and box regression parameters of each anchor box are predicted; then, high-quality candidate boxes are retained based on the classification score; In the second stage, 6×6×6 grid points are uniformly sown along the length, width and height of each voxel candidate box. Each grid point aggregates the multi-scale voxel features around it. After being processed by the voxel candidate box pooling module, each candidate anchor box obtains the pooled features, and then a fully connected layer is used for confidence prediction and anchor box refinement. The second part of the test consists of two stages: In the first stage, the confidence prediction and anchor box refinement output by the fully connected layer are used. A hybrid detection head with and without anchor boxes is used. Each detection head predicts one category. Four anchor-based detection heads are used to predict cars, buses, trucks, and cyclists, respectively, and one anchor-free detection head is used to predict pedestrians. This ensures the detection performance of both pedestrians and vehicles. In the second stage, the voxel-based candidate box self-attention pooling module introduces a self-attention mechanism to encode the local and global features of each grid point, thereby improving the detection performance of large objects such as trucks and buses.

Images