A 3D target detection method based on original point cloud processing

Abstract

The application provides a 3D target detection algorithm based on original point cloud processing, which is completed based on cuda on a GPU and comprises: first feature extraction processing of original point cloud data, first prediction of the category and the bounding box of a target to be detected, acquisition of an index with the highest category prediction score as the corresponding category of the target to be detected and bounding box prediction information; second feature extraction processing of point clouds in the bounding box obtained in the first feature extraction processing, second prediction of the category and the bounding box, and final output of 3D target information in the whole point cloud scene. The application maintains a high processing speed from point cloud operation to neural network operation, guarantees high precision of the 3D target detection algorithm through feature extraction on original point clouds, and realizes an end-to-end 3D target detection algorithm for point cloud feature extraction.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of laser radar detection, in particular to a 3D target detection method based on raw point cloud processing. BACKGROUND

[0002] Laser radar is one of the indispensable sensors in the field of autonomous driving. It relies on laser beams to scan the surrounding environment continuously to complete the scene reconstruction. Different laser radars can generate different numbers of 3D laser points, and the laser points describe the accurate xyz coordinates and reflectivity of the object surface in the radar coordinate system. The point cloud data is sparse, unordered and unstructured, unlike the regular and ordered image data, so the 2D target detection algorithm of the image cannot be directly applied in the point cloud field.

[0003] In the field of laser point cloud 3D target detection, there are different processing methods: (1) processing point cloud based on 2D image idea, this method can quickly detect but loses the height information of the point cloud, and can only produce a 2D bounding box on the bird's eye view (complex yolo); (2) processing point cloud based on gridding idea, this method also loses part of the original point information, resulting in slow processing speed and low accuracy in the point cloud gridding process (second pointpillar); (3) processing based on raw point cloud, directly extracting point-level features and aggregating, and performing target detection on the original point features, this processing method can maintain the original features of the points, and the 3D target detection accuracy is high, but the calculation amount of the algorithm model is large and the time-consuming is long (pointrcnn).

[0004] In the prior art, the typical target detection algorithm based on raw point cloud is PointRCNN, which is composed of four parts. The first part is the first stage neural network processing part; the second part is the post-processing part of the one-stage network, which is completed in the CPU and takes a long time; the third part is the second stage neural network processing part, which is processed in the GPU; the fourth part is the second stage post-processing part, which is processed in the CPU and takes a long time. Overall, the data processing of the entire two-stage network needs to be transmitted four times in different hardware devices, and each process takes a long time, which causes the raw point-based 3D target detection method to be unable to achieve real-time performance.

[0005] In summary, the existing technology for 3D target detection of point cloud scenes has three major problems. First, the accuracy and speed of the 3D point cloud algorithm are difficult to balance; second, it cannot form a truly end-to-end detection, which will cause a lot of delay in the data processing process, which is very unfriendly to actual application; third, the non-neural network data processing part of the existing technology is complex, time-consuming and occupies a large amount of memory, resulting in high complexity and large calculation amount of the algorithm model. SUMMARY

[0006] Therefore, the application provides a 3D target detection method based on original point cloud processing, which adopts an end-to-end design idea, uses cuda and cudnn to accelerate the point cloud neural network processing part, and uses cuda to accelerate the non-neural network data processing part, so that the processing speed of point cloud operation and neural network operation is maintained at a high level, the feature extraction on the original point cloud ensures the high precision of the 3D target detection method, and the end-to-end point cloud feature extraction is realized. The application solves the problem that the existing detection algorithm model based on original point cloud processing cannot be run in real time in an end-to-end manner, and the problem that the speed and precision of the 3D target detection method are difficult to balance.

[0007] In order to achieve the above object, the application provides the following technical scheme:

[0008] A 3D target detection method based on original point cloud processing, which is completed on a GPU based on cuda, comprising:

[0009] performing first feature extraction processing on original point cloud data, completing first prediction of the category and the bounding box of the target to be detected, obtaining an index with the highest category prediction score as the corresponding category of the target to be detected, and the bounding box prediction information;

[0010] performing second feature extraction processing on the point cloud in the bounding box obtained in the first feature extraction processing, completing second prediction of the category and the bounding box, and finally outputting 3D target information in the entire point cloud scene.

[0011] Further, the first feature extraction processing is used for feature fusion and extraction of the features of the sampling points, and the high-level semantic features of the sampling points are extracted as the high-level semantic features of the original point cloud, and the first prediction of the category and the bounding box of the target to be detected is performed according to the high-level semantic features of the original point cloud.

[0012] Further, the first feature extraction processing specifically comprises: first performing grid downsampling on the original point cloud to reduce the resolution of the point cloud, then aggregating the bottom-level features of the sampled point cloud, and then extracting high-level dimensional features through a neural network processing layer after the aggregated bottom-level features, then performing feature interpolation operation on the down-sampling points generated by the downsampling through an up-sampling layer to restore the low-resolution high-level dimensional features to the point cloud of the original size with high resolution to obtain up-sampling features of the original point, and then performing fusion and extraction of the up-sampling features through a neural network processing layer to obtain the high-level semantic features of the original point cloud.

[0013] Further, the bounding box prediction information includes a relative offset of length, width and height corresponding to a predicted category target anchor, a three-dimensional coordinate absolute value and a target orientation angle.

[0014] Further, the second feature extraction processing includes: performing a downsampling feature extraction and aggregation operation on the points in the bounding box obtained after the first prediction again to obtain high-level semantic features of the points in each bounding box, and then performing a second prediction of the category and the bounding box on the obtained high-level semantic features of the bounding box.

[0015] Further, before the second feature extraction processing, the bounding box prediction information obtained after the first prediction is decoded into target bounding box absolute values, and an nms operation is performed to retain the bounding boxes with a category score higher than a set value, and then a query is performed in the original points, and each retained bounding box is queried for nearby grid points based on the parallelism of cuda, and finally the point cloud in each bounding box is obtained, and the second feature extraction processing is performed.

[0016] Further, in the nms operation, the target bounding box absolute values are sorted according to the category score, and the IOU of each box in each category and the highest score bounding box is calculated, and the bounding boxes with a category score higher than a set value are retained.

[0017] Further, after the second prediction of the category and the bounding box is performed, the category and bounding box prediction information obtained after the second prediction is corrected, and an nms operation is performed again, and finally the 3D category and bounding box information of the existing target in the entire point cloud scene is output.

[0018] Further, the 3D target information includes seven values of the three-dimensional xyz coordinates of the target, the length, width and height of the target, and the orientation angle of the target.

[0019] Further, the original point cloud data includes four values of each point, i.e., three coordinate values of xyz and reflectivity.

[0020] The 3D target detection method for end-to-end processing of original point clouds is realized, and the key point is that each processing operation and neural network layer is combined in the entire algorithm. The processing speed is maintained at a very high level from point cloud operation to neural network operation, and the feature extraction on the original point cloud ensures the high accuracy of the 3D target detection method. The high-precision 3D point cloud algorithm can be put into practical application, and only needs to be deployed and optimized to be applied in the field of automatic driving. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only illustrate some of the embodiments of the present disclosure, and those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0022] Figure 1 A flowchart of a 3D target detection method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0024] The above and other advantages and features of the present disclosure will become apparent from the following description of the embodiments, given by way of example only, with reference to the accompanying drawings.

[0025] It is to be understood that the ranges and limits mentioned herein include all ranges lying between the recited values; they include limits, which can be either included or excluded in the ranges; and they include all values, which lie within the range. It should be noted that, as used in the specification and the appended claims, the singular forms "a," "an" and "the" include their corresponding plural references unless the context clearly dictates otherwise. It should also be noted that, as used in the specification and the appended claims, the term "or" is generally used in the sense of "and / or" unless the context clearly dictates otherwise. In addition, it should be noted that, as used in the specification and the appended claims, the terms "the another", "another" and "an additional" generally mean "one or more than one", "one or more", "at least one", or "one or more than one of the previously described features".

[0026] It is also to be understood that the above description is only illustrative of the application and that modifications and other embodiments of the application will become apparent to those skilled in the art without departing from the spirit and scope of the application. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature and not as restrictive.

[0027] Also in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, one skilled in the relevant art will appreciate that the described aspects can be practiced without these specific details.

[0028] The present application provides a kind of 3D target detection method based on original point cloud processing, the 3D target detection method is completed based on cuda (Compute Unified Device Architecture, general parallel computing architecture) on GPU, including:

[0029] the first feature extraction processing is carried out to original point cloud data, the first prediction of the category and boundary box of the target to be detected is completed, the index with the highest category prediction score is obtained as the corresponding category of the target to be detected, and boundary box prediction information is obtained;

[0030] the second feature extraction processing is carried out to the point cloud in the boundary box obtained in the first feature extraction processing, the second prediction of the category and boundary box is completed, and the information of the 3D category and boundary box of the existing target in the whole point cloud scene is finally output.

[0031] Next, reference is made to the accompanying drawings Figure 1 , further illustrate the 3D target detection method of the present application.

[0032] As shown in the figure Figure 1 It is the 3D target detection network structure diagram based on original point cloud processing of end to end of the embodiment of the present application, input is original point cloud data, wherein each point is represented by three coordinate values and reflectivity, a total of 4 numerical values, from top to bottom, grid sampling layer, adjacent point query and feature aggregation layer, neural network processing layer, up-sampling layer, neural network processing layer, category and boundary box prediction layer, boundary box decoding layer, NMS (Non-Maximum Suppression, non-maximum suppression), boundary box point query layer, grid sampling layer, adjacent point query and feature aggregation layer, neural network processing layer, category and boundary box prediction layer, boundary box decoding, NMS (non-maximum suppression), output is 3D target information, which contains the three-dimensional center coordinates XYZ of the target, the real size length, width and height of the target and the orientation angle of the target, a total of 7 numerical values.

[0033] The embodiment of the present application completes the 3D target detection method of end to end point cloud feature extraction, the whole process is completed based on cuda on GPU, so that the 3D point cloud target detection algorithm model reasoning process has the advantage of high frame rate. Among them, the end to end described above means that the whole 3D point cloud target detection algorithm is completed in GPU from receiving point cloud data to outputting target 3D detection frame, without calling or converting to other computing units to complete the processing of the whole algorithm, including the following steps:

[0034] 1. Grid down-sampling of the received original point cloud data, which is similar to the convolution down-sampling of an image, and the purpose is to reduce the resolution of the point cloud. Since the point cloud is sparse and disordered, down-sampling in the original point cloud with a large amount of data consumes a lot of hardware resources. In order to ensure that the distribution of the sampling points is relatively uniform, we consider dividing a fixed-size grid in the spatial scale, and taking part of the points in each grid, so that parallel sampling can be realized, and the time-consuming caused by sampling can be greatly reduced.

[0035] 2. The sampled point cloud is relatively uniformly distributed in space. In order to extract the semantic information of each point, the features of the point and its nearby points need to be extracted. This requires finding 16 points and 32 points near each point. The neighbor point query is a fast query operator based on cuda completed in this embodiment, and the main idea is to query the points with close distance in the 27 nearest grids or the 125 second nearest grids of the sampling points in each grid in the last step. If the number of neighboring points is less than 16 or 32 points, the nearest points are used to complete. After the neighbor point query, each down-sampled point has a fixed number of neighboring points. The xyz three-dimensional coordinate values and reflectivity of each point are spliced to complete the aggregation of the bottom layer features.

[0036] 3. The features aggregated in the last step are extracted to extract high-level features, including Linear, BN, and Relu layers. These layers belong to neural network processing layers, and the parallel processing speed on GPU is relatively fast. The final extracted feature is a high-dimensional feature of each aggregated sampling point. Here, Maxpool is used to take the maximum value of the feature of each aggregated sampling point as the neighborhood feature of the sampling point.

[0037] 4. The up-sampling layer is a feature interpolation operation on the down-sampled points generated in the last step, which is similar to the up-sampling layer of an image, and the purpose is to restore the low-resolution high-dimensional features to the high-resolution original size point cloud. For each down-sampled point, the embodiment of the application needs to use the parallel processing of cuda to query the 3 points near the original point in the down-sampled point set, take the features of the 3 points as the features of the original point, and normalize the distance from the 3 down-sampled points to the original point to obtain the weights of the 3 points. Multiply the 3 weights by the feature values corresponding to the 3 points to obtain the up-sampled features of the original point.

[0038] 5. Feature fusion and extraction are performed on the up-sampled features, and the features of the original point are extracted through two Linear, BN and ReLU layers to extract high-level semantic features as the high-level semantic features of the original point.

[0039] 6、According to the high-level semantic features of the original point cloud, the category prediction of the to-be-detected target and the regression of the bounding box thereof are performed, the category prediction outputs scores of several categories, and after a sigmoid function, the index with the highest score is taken as the corresponding category; the bounding box prediction information includes the relative offset of the length, width and height corresponding to the predicted category target anchor, the absolute value of the three-dimensional coordinate, and the target orientation angle.

[0040] 7、The bounding box decoding layer mainly converts the bounding box prediction information obtained in the previous step into the absolute value of the target bounding box, which includes the three-dimensional xyz coordinates of the target, the length, width and height of the target, and the orientation angle of the target.

[0041] 8、The non-maximum suppression nms layer sorts the target prediction boxes according to the category scores, calculates the IOU (intersection over union) of each box in each category with the highest score prediction box, and filters out the prediction boxes greater than a certain threshold, so as to ensure that there is only one bounding box with the highest score at each target, and the nms step retains a certain number of bounding boxes with higher scores.

[0042] 9、The bounding box point query layer queries the nearby grid points based on the bounding boxes output by the nms, and the embodiment of the present application queries the nearby grid points based on the parallelism of cuda for each bounding box to ensure low latency in the processing process, and finally obtains a certain number of points in each bounding box.

[0043] 10、The points in the bounding box obtained in the previous step are down-sampled and extracted and aggregated again, and then the step 2.3 is repeated to obtain the high-level semantic features of the points in each bounding box, and on this basis, the high-level semantic features of multiple points in each box are max-pooled to finally retain the high-level semantic features corresponding to each box.

[0044] 11、The semantic features of the bounding box obtained in the previous step are subjected to category prediction and bounding box prediction, wherein the category prediction is to correct the category of the bounding box again, and the bounding box prediction information is also to correct the bounding box based on the step 8.

[0045] 12、The bounding box corrected in the previous step is subjected to nms operation again, and finally the 3D category and bounding box information of the existing target in the whole point cloud scene are output.

[0046] In the embodiment of the present application, the grid sampling layer of the point cloud can be replaced by farthest point sampling or random sampling, which can also achieve the purpose of reducing the resolution of the point cloud. The NMS non-maximum suppression operation can be replaced by DINMS, IOU NMS, SoftNMS, etc., which can also achieve the purpose of filtering overlapping bounding boxes. In the neural network processing layer, Linear, BN, Relu can be replaced by Conv2d, BN, Relu / LeakyRelu / Relu, which can also achieve the purpose of feature extraction. In addition, MaxPool can be replaced by Avgpool, which can also achieve the purpose of pooling the maximum feature.

[0047] In a specific embodiment of the present application, taking Nvidia's 2080 TI GPU as an example, the original point cloud data is transmitted to the GPU, and the entire 3D point cloud algorithm processing can be started. The processing process includes point cloud sampling, neighboring point query and grouping, feature extraction and aggregation, feature interpolation, feature fusion, NMS and coordinate conversion operations. Finally, the GPU outputs the 3D target information. The end-to-end processing mode from the GPU input data to the GPU output data ensures high-speed data processing and real-time and high efficiency of the 3D point cloud algorithm.

[0048] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A 3D object detection method based on raw point cloud processing, wherein the 3D object detection method is implemented on a GPU based on CUDA, characterized in that, include: The original point cloud data is subjected to the first feature extraction process to complete the first prediction of the category and bounding box of the target to be detected. The index with the highest category prediction score is used as the corresponding category of the target to be detected, as well as the bounding box prediction information. A second feature extraction process is performed on the point cloud within the bounding box obtained in the first feature extraction process to complete the second prediction of the category and bounding box, and finally output the 3D target information within the entire point cloud scene. The first feature extraction process is used to perform feature fusion and extraction on the features of the sampling points, extract the high-level semantic features of the sampling points as the high-level semantic features of the original point cloud, and make the first prediction of the category and bounding box of the target to be detected based on the high-level semantic features of the original point cloud. The first feature extraction process specifically includes: firstly, performing grid downsampling on the original point cloud to reduce the resolution of the point cloud; then, aggregating the low-level features of the sampled point cloud; and then processing the aggregated low-level features through a neural network layer to extract high-level dimensional features. Next, performing feature interpolation on the downsampled points generated by the downsampling through an upsampling layer to restore the low-resolution high-level dimensional features to the high-resolution original size point cloud, obtaining the upsampled features of the original points. Finally, the upsampled features are fused and extracted through a neural network layer to obtain the high-level semantic features of the original point cloud. The bounding box prediction information includes the relative offsets of length, width and height of the predicted category target anchor, the absolute values ​​of the three-dimensional coordinates, and the target orientation angle; The second feature extraction process includes: performing downsampling feature extraction and aggregation operations on the points in the bounding boxes obtained after the first prediction to obtain high-level semantic features of the points inside each bounding box; and then performing a second prediction of the category and bounding box based on the obtained high-level semantic features of the bounding boxes.

2. The 3D target detection method based on raw point cloud processing according to claim 1, characterized in that, Before the second feature extraction process, the bounding box prediction information obtained after the first prediction is decoded and converted into the absolute value of the target bounding box. Then, an NMS operation is performed to retain the bounding boxes with class scores higher than a set value. Next, a query is performed on the original points. Based on the parallelism of CUDA, the nearby grid points of each retained bounding box are queried to finally obtain the point cloud in each bounding box, and then the second feature extraction process is performed.

3. The 3D target detection method based on raw point cloud processing according to claim 2, characterized in that, In the NMS operation, the absolute values ​​of the target bounding boxes are sorted by category score, and the IOU between each box in each category and the highest-scoring bounding box is calculated. The bounding boxes are then filtered according to a set value for the category score, retaining the bounding boxes whose category scores are higher than the set value.

4. The 3D target detection method based on raw point cloud processing according to claim 3, characterized in that, After the second prediction of category and bounding box, the process includes correcting the category and bounding box prediction information obtained after the second prediction, performing NMS operation again, and finally outputting the 3D target information in the entire point cloud scene.

5. The 3D target detection method based on raw point cloud processing according to claim 1, characterized in that, The 3D target information includes seven values: the target's three-dimensional xyz coordinates, the target's length, width, and height, and the target's orientation angle.

6. The 3D target detection method based on raw point cloud processing according to claim 1, characterized in that, The raw point cloud data includes four values ​​for each point: x, y, z coordinates and reflectance.

Images