Point cloud 3D target detection method and model based on discrete transformer

Abstract

The application discloses a point cloud 3D target detection method and model based on a discrete Transform, and the method comprises the following steps: S1, collecting point cloud data frames of objects in real time; S2, performing point cloud voxelization on the point cloud data frames to obtain initial voxels; S3, extracting voxel features containing dynamic information and static information from the initial voxels through a 3D backbone network based on a discrete Transform; S4, mapping the voxel features finally output in the step S3 to a BEV space to obtain corresponding 2D BEV features; and S5, sending the 2D BEV features to a 3D target detector through a Neck network to perform 3D target detection, and obtaining object attribute information of objects in a 3D space.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving object perception, in particular to a point cloud 3D target detection method and model based on discrete Transformer. BACKGROUND

[0002] An automatic driving car is a complex unmanned system relying on vehicle-mounted sensors to perceive the environment and make decision control. In order to realize the decision and control of automatic driving, the sensor (usually including laser radar and camera) is first used to perceive the surrounding environment, and the 3D semantic information of the object in the surrounding environment is obtained by processing the sensor data.

[0003] 3D object detection based on laser radar is a key technology to solve the problem of environment perception of automatic driving. Through the real-time obtained point cloud data frame, the 3D semantic information of the object is obtained by encoding and decoding through neural network. In order to realize high efficiency, the point cloud 3D target detection algorithm based on voxel is generally used in the field of automatic driving at present. These methods first quantize the point cloud into voxels, and then use the backbone network based on 3D sparse convolution to extract the features of voxels. However, due to the unordered and unstructured discrete form of point cloud data structure, it is difficult to extract the geometric structure information of the object using fixed weight static convolution. Recently, VOTR proposes to use voxel Transformer to extract the dynamic features of voxels. The algorithm generates a voxel hash table according to the voxel coordinates to index the voxel features, and then pads to the specified length and inputs into the conventional full-attention attention mechanism. However, this Transformer ignores the sparse and discrete nature of point cloud, and for each query voxel feature, a specified number of key voxel features are needed, which greatly increases the amount of calculation and time consumption. And VOTR uses a large receptive field, which is not conducive to detecting small objects. Secondly, VOTR is completely composed of Transformer, which is difficult to extract the static features of point cloud. In order to improve the precision and recall rate of point cloud 3D target detection algorithm, it is urgent to improve the existing 3D backbone network to realize efficient preservation of dynamic and static features of point cloud. SUMMARY

[0004] In order to solve the problem of efficient extraction of dynamic and static features of point cloud in the existing point cloud 3D detection technology, the present application proposes a point cloud 3D target detection method and model based on discrete Transformer, so that the network can efficiently preserve the static and dynamic features of point cloud.

[0005] According to an embodiment of the present application, a point cloud 3D target detection method based on a discrete Transformer is provided, comprising the following steps: S1, collecting a point cloud data frame of an object in real time; S2, voxelizing the point cloud data frame to obtain initial voxels; S3, extracting voxel features containing dynamic information and static information from the initial voxels through a 3D backbone network based on a discrete Transformer; S4, mapping the voxel features finally output by step S3 to a BEV space to obtain corresponding 2D BEV features; and S5, sending the 2D BEV features to a 3D target detector through a Neck network to perform 3D target detection, and obtaining object attribute information of the object in a 3D space.

[0006] According to another embodiment of the present application, a point cloud 3D target detection model based on a discrete Transformer is provided, comprising: a point cloud voxelization module, configured to voxelize a point cloud data frame of an object to output initial voxels; a 3D backbone network based on a discrete Transformer, connected to an output end of the point cloud voxelization module, configured to extract voxel features containing dynamic information and static information from the initial voxels; a voxel feature mapping module, connected to the 3D backbone network based on a discrete Transformer, configured to map the voxel features containing dynamic information and static information to a BEV space to obtain corresponding 2D BEV features; and a Neck network, connected to an output end of the voxel feature mapping module, configured to send the 2D BEV features to a 3D target detector to perform 3D target detection, and obtain object attribute information of the object in a 3D space.

[0007] The present application proposes a general grid point cloud feature extraction backbone network which can be applied to all existing grid-based point cloud 3D detectors. Compared with the prior art (CenterPoint, PV-RCNN, Focals, Voxel-RCNN, SST, PillarNet, PointPillar and other algorithms), the detection method proposed by the present application can effectively extract dynamic features and static features of the point cloud, retain more rich 3D geometric information, and thus greatly improve the precision and recall rate of the 3D target detection algorithm and improve the perception ability of the automatic driving car to the surrounding environment. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a process schematic diagram of the point cloud 3D target detection model based on a discrete Transformer of the embodiment of the present application.

[0009] Figure 2 is a process schematic diagram of the discrete Transformer module processing voxels of the embodiment of the present application.

[0010] Figure 3 is a flowchart of a process of a multi-scale discrete Transformer module processing a voxel according to an embodiment of the present application.

[0011] Figure 4 is a schematic diagram of a discrete attention mechanism according to an embodiment of the present application. DETAILED DESCRIPTION

[0012] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the purpose of providing the embodiments is merely to illustrate, but not to limit the protection scope of the present application.

[0013] An embodiment of the present application provides a point cloud 3D target detection model based on a discrete Transformer. The network architecture of the model is shown in Figure 1 , and the model comprises, in sequence from an input end to an output end: a point cloud voxelization module, a sub-manifold 3D sparse convolution 10, a discrete Transformer module 20, a multi-scale discrete Transformer module 30, a multi-scale discrete Transformer module 40, a voxel feature BEV mapping module, a Neck network, and a 3D target detector. BEV stands for Bird’s Eye View, which is a bird’s eye view.

[0014] The sub-manifold 3D sparse convolution 10, the discrete Transformer module 20, the multi-scale discrete Transformer module 30, and the multi-scale discrete Transformer module 40 constitute a 3D backbone network of the model, and are mainly responsible for extracting voxel features containing dynamic information and static information from the voxelized point cloud. The 3D backbone network based on the discrete Transformer extracts the static information and the dynamic information of the voxelized point cloud using the discrete Transformer module. The discrete Transformer module first obtains down-sampled voxels by down-sampling 3D sparse convolution, and uses the down-sampled voxels as query features of the discrete attention mechanism. Then, the discrete Transformer module extracts static features of the voxels using the sub-manifold 3D sparse convolution, extracts dynamic features of the voxels using the discrete attention mechanism, and concatenates the dynamic features and the static features along the channel dimension as output features.

[0015] The discrete Transformer module 20, the multi-scale discrete Transformer module 30 and the multi-scale discrete Transformer module 40 are all composed of down-sampling 3D sparse convolution, sub-manifold 3D sparse convolution and 3D discrete attention mechanism, and the main difference between the multi-scale discrete Transformer module (30, 40) and the discrete Transformer module 20 is that: 1) the input of the multi-scale discrete Transformer module (30, 40) is two different scales of voxels; 2) the difference in attention calculation (to be described later).

[0016] Another embodiment of the present application provides a point cloud 3D target detection method based on a discrete Transformer, and a flowchart of the method is as shown in Figure 1 The method comprises the following steps: collecting a point cloud data frame of an object in real time by using a laser radar; performing point cloud voxelization on the point cloud data frame by using mean voxelization or dynamic voxelization based on a multi-layer perception machine to obtain an initial voxel V0 (the voxel contains voxel features and voxel coordinates); the initial voxel V0 is subjected to sub-manifold 3D sparse convolution 10 to increase a receptive field, and the output voxel is denoted as V1; the voxel V1 is sent to a discrete Transformer module 20 to obtain a voxel V2; the voxels V1 and V2 are sent to a multi-scale discrete Transformer module 30 to obtain a voxel V3; the voxels V2 and V3 are sent to a multi-scale discrete Transformer module 40 to obtain a voxel V4; the voxel V4 is mapped to a BEV space to obtain corresponding 2D BEV features, denoted as F bev ; finally, F bev is sent to a 3D target detector through a Neck network to perform 3D target detection, and the position, the three-dimensional size of a bounding box, the object heading angle and other object attribute information of the object in the 3D space are obtained.

[0017] In some embodiments, the convolution kernel size of the sub-manifold 3D sparse convolution 10 can be, for example, 3x3x3.

[0018] The specific steps of sending the voxel V1 to the discrete Transformer module 20 to obtain the voxel V2 are as follows: first, refer to Figure 2 According to the voxel coordinates of the voxel V1 (corresponding to the voxel v0 in Figure 2 , the voxel feature dimension is denoted as MxC), a voxel hash table is generated, and each row of the hash table stores the value of a voxel (the calculation process of the voxel value: the i-th voxel coordinate is denoted as (x i ,y i ,z i ), and the maximum value of all voxel coordinates is (x max ,y max ,z max ), so the value of the i-th voxel is xi *y max *z max +y i *z max +z i ) and the index id of the voxel, then the voxel V1 (corresponding to the voxel v0 in Figure 2 ) is obtained by a down-sampling 3D sparse convolution, and the voxel v1 with a feature dimension of N x 2C is obtained by a sub-manifold 3D sparse convolution 11, and the down-sampled voxel v2 (with a feature dimension of N x C) is obtained. Next, referring jointly to Figure 4 , the attention calculation is performed using the voxel V1 (that is, v0 in Figure 2 ) and v2: according to the coordinates of v2, the voxels in the corresponding range in the hash table of the voxel V1 are searched as the key voxels for attention calculation in a search space of 3*3*3, to obtain a key index table and a query index table (for the i-th voxel in v2, the coordinates are (x i ,y i ,z i ), and the voxels in the range of 2x i -1≤x≤2x i +1, 2y i -1≤y≤2y i +1, 2z i -1≤z≤2z i +1 in the voxel V1 are searched, and then the query index table and the key index table are obtained); continue to refer to Figure 4 , the query feature (with a dimension of K x C) is obtained from the index in v2 according to the query index table, and the key feature (with a dimension of K x C) is obtained from the index in the voxel V1 according to the key index table; then the dot product of the query feature and the key feature is performed, and the sum along the feature dimension is obtained to obtain a voxel feature with a dimension of K x 1, then the discrete Softmax is performed according to the query index table to obtain a K x 1 dimensional attention score, then the dot product of the attention score and the query feature is performed to obtain a feature with a dimension of K x C, and then the discrete sum is performed according to the query index table to obtain a feature F attention with a dimension of N x C, which is finally concatenated with the voxel v3 obtained by the sub-manifold 3D sparse convolution 12 along the feature dimension to obtain the output voxel V2 (corresponding to the voxel v4 in Figure 2 ).

[0019] The voxels V1 and V2 are input into the multi-scale discrete Transformer module 30 to obtain the voxel V3, and the specific steps are as follows: first refer to Figure 3 , Figure 3is a processing schematic inside the multi-scale discrete Transformer module, and voxel V2 is taken as Figure 3 voxel v0 in Figure 3 , the processing procedure of voxel v0 (V2) is the same as Figure 2 , which will not be repeated here; and voxels V1 and V2 are taken as the “multi-scale voxels” in Figure 3 , the internal network architecture of the multi-scale discrete Transformer module 30, 40 (such as Figure 3 ) is the same as that of the discrete Transformer module 20 (such as Figure 2 ), and the main difference in processing voxels is that the input of the discrete attention mechanism calculation process also contains multi-scale voxels. The difference from the attention calculation in the aforementioned step of “sending voxel V1 into the discrete Transformer module 20 to obtain voxel V2” is that: two key index tables and two query index tables are calculated for V1 and V2, then two key features with dimensions of K1xC and K2xC and two query features with dimensions of K1xC and K2xC are obtained, then the two key features are spliced and the two query features are spliced to obtain key features and query features with dimensions of (K1+K2)xC, at this time, the dimensions of the “key features” and “query features” in Figure 4 are (K1+K2)xC, which is equivalent to replacing K in the dimensions of the “key features” and “query features” in Figure 4 with (K1+K2), and then repeating the subsequent attention calculation steps to obtain the feature F attention with dimensions of NxC. Continuing to refer to Figure 3 , the feature F attention obtained through the discrete attention mechanism is spliced with voxel v3 along the feature dimension to obtain an output voxel V3 with dimensions of Nx2C (corresponding to voxel v4 in Figure 3 ).

[0020] The voxels V2 and V3 are sent into the multi-scale discrete Transformer module 40 to obtain voxel V4. The specific steps are as follows: since this step and the previous step are both implemented by using a multi-scale discrete Transformer module, the processing steps and principles are the same, and still refer to Figure 3 , but in this step: voxels V2 and V3 are taken as the “multi-scale voxels” in Figure 3 , voxel V2 is taken as voxel v0 in Figure 3 , and the output feature after the discrete attention mechanism is a feature F attention with dimensions of NxC, and voxel V4 is equivalent to voxel v4 in Figure 3 .

[0021] Mapping the voxel V4 to the BEV space obtains the corresponding 2D BEV feature F bev The specific steps are: the voxel V4 is further down-sampled by a 3D sparse convolution, and then the height dimension feature is spliced to the channel dimension, the mapping of the 3D feature to the BEV feature is completed, and the 2D feature F of the BEV space is obtained bev .

[0022] Finally, F bev is sent into the 3D target detector through the Neck network for 3D target detection, and the position of the object in the 3D space, the three-dimensional size of the bounding box, the object heading angle, the object category and other object attributes are obtained. It should be understood that the 3D target detector of the embodiment of the application can be a CenterHead, a PV-RCNNHead, a Voxel-RCNNHead and the like, and the application does not limit this. Different 3D detection heads can output different object attributes.

[0023] In the detection method and model of the embodiment of the application, the use of the discrete attention mechanism can enable the efficient application of the Transformer in the point cloud; the detection method and model of the embodiment of the application can efficiently extract the dynamic features and static features of the point cloud through the discrete Transformer; and the detection method and model of the embodiment of the application can be applied to all grid-based point cloud 3D target detection algorithms.

[0024] The detection method and model proposed in the embodiment of the application can be applied to the automatic control scene such as the automatic driving car and the robot to perceive the environment, improve the performance of the existing point cloud 3D target detection algorithm without increasing additional time-consuming, and improve the perception ability of the automatic driving car to the environment and the accuracy and recall rate of target detection.

[0025] The above is a further detailed description of the application in combination with a specific preferred embodiment, and the specific implementation of the application cannot be limited to these descriptions. For those skilled in the art to which the application belongs, without departing from the concept of the application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the application.

Claims

1. A point cloud 3D target detection method based on Discrete Transformer, characterized in that, Includes the following steps: S1. Real-time acquisition of point cloud data frames of objects; S2. Perform point cloud voxelization on the point cloud data frame to obtain initial voxels; S3. Extract voxel features containing dynamic and static information from the initial voxels using a 3D backbone network based on discrete Transformer; The 3D backbone network based on Discrete Transformer uses Discrete Transformer modules to extract static and dynamic information from voxelized point clouds. The Discrete Transformer module first obtains downsampled voxels through downsampled 3D sparse convolution, which serve as the query features for the discrete attention mechanism. Then, it uses submanifold 3D sparse convolution to extract the static features of the voxels, uses the discrete attention mechanism to extract the dynamic features of the voxels, and concatenates the dynamic features and static features along the channel dimension as the output features. S4. Map the voxel features finally output in step S3 to the BEV space to obtain the corresponding 2D BEV features. S5. The 2D BEV features are fed into the 3D target detector through the Neck network to perform 3D target detection and obtain the object attribute information of the object in 3D space.

2. The point cloud 3D target detection method based on discrete Transformer as described in claim 1, characterized in that: In step S2, the initial voxels are obtained through mean voxelization or dynamic voxelization based on a multilayer perceptron.

3. The point cloud 3D target detection method based on discrete Transformer as described in claim 1, characterized in that, Step S3 specifically includes: S31. Input the initial voxel into a first submanifold 3D sparse convolution to obtain the first voxel; S32. Input the first voxel into a discrete Transformer module to obtain the second voxel; S33. Input the first voxel and the second voxel into a first multi-scale discrete Transformer module to obtain the third voxel; S34. Input the second voxel and the third voxel into a second multi-scale discrete Transformer module to obtain a fourth voxel, which is used as the voxel feature of the final output of step S3.

4. The point cloud 3D target detection method based on discrete Transformer as described in claim 3, characterized in that: The discrete Transformer module, the first multi-scale discrete Transformer module, and the second multi-scale discrete Transformer module are all composed of downsampled 3D sparse convolution, second submanifold 3D sparse convolution, and 3D discrete attention mechanism.

5. The point cloud 3D target detection method based on discrete Transformer as described in claim 1, characterized in that, Step S4 specifically includes: The voxel features output in step S3 are then processed by a 3D sparse convolution, and the height dimension features are concatenated to the channel dimension to complete the mapping from 3D space to BEV space, thus obtaining the 2D BEV features.

6. The point cloud 3D target detection method based on discrete Transformer as described in claim 1, characterized in that: The object attribute information includes the object's position in 3D space, the 3D size of the bounding box, the object's heading angle, and its category.

7. A point cloud 3D target detection model based on Discrete Transformer, characterized in that, include: The point cloud voxelization module is used to voxelize the point cloud data frame of an object and output the initial voxel. A 3D backbone network based on Discrete Transformer is connected to the output of the point cloud voxelization module. It is used to extract voxel features containing dynamic and static information from the initial voxels. The 3D backbone network based on Discrete Transformer uses the Discrete Transformer module to extract static and dynamic information from the voxelized point cloud. Specifically, the Discrete Transformer module first obtains downsampled voxels through downsampled 3D sparse convolution, which serves as the query features for the discrete attention mechanism. Then, it uses submanifold 3D sparse convolution to extract the static features of the voxels, uses the discrete attention mechanism to extract the dynamic features of the voxels, and concatenates the dynamic and static features along the channel dimension as the output features. The voxel feature mapping module, connected to the discrete Transformer-based 3D backbone network, is used to map the voxel features containing dynamic and static information to the BEV space to obtain the corresponding 2D BEV features. The Neck network, connected to the output of the voxel feature mapping module, is used to send the 2D BEV features into the 3D target detector for 3D target detection, thereby obtaining the object attribute information of the object in 3D space.

8. The point cloud 3D target detection model based on discrete Transformer as described in claim 7, characterized in that: The 3D backbone network based on Discrete Transformer includes a first submanifold 3D sparse convolution, a discrete Transformer module, a first multi-scale discrete Transformer module, and a second multi-scale discrete Transformer module connected in sequence. The first submanifold 3D sparse convolution takes the initial voxel as input and outputs a first voxel; the discrete Transformer module takes the first voxel as input and outputs a second voxel; the first multi-scale discrete Transformer module takes the first voxel and the second voxel as input and outputs a third voxel; the second multi-scale discrete Transformer module takes the second voxel and the third voxel as input and outputs a fourth voxel.

9. The point cloud 3D target detection model based on discrete Transformer as described in claim 8, characterized in that: The discrete Transformer module, the first multi-scale discrete Transformer module, and the second multi-scale discrete Transformer module are all composed of downsampled 3D sparse convolution, second submanifold 3D sparse convolution, and 3D discrete attention mechanism.

Images