Transform-based three-dimensional target detection for time-series point cloud

Abstract

The application discloses a three-dimensional target detection of time sequence point cloud based on a transformer, and comprises the following steps: establishing a K-frame time sequence point cloud sequence, converting each frame of point cloud to a unified description under a current frame coordinate system; voxelizing each frame of point cloud and using 3D sparse convolution to extract spatial features, generating a pseudo 2D feature map under a bird's-eye view; further extracting features and fusing multi-scale features of each frame of bird's-eye view feature map to generate a feature map; using a TSA center point prediction network to generate space-time attention, weighting and fusing multi-frame feature maps to generate a center point prediction result of a target; using a transformer multi-frame fusion network to mine inter-frame target correlation features of the time sequence point cloud; using a regression detection head to extract and transform target features to obtain various regression parameters of a target frame, and decoding the regression parameters of the target frame and the center point prediction result together to obtain a final detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving and deep learning, and particularly relates to a three-dimensional target detection of time-series point cloud based on a Transformer. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, it penetrates into various fields including automatic driving, robots, etc. Automatic driving is crucial for improving road safety and enhancing traffic efficiency. The typical automatic driving vehicle system architecture can be divided into three modules: environment perception, decision planning and motion control. The perception module is mainly responsible for collecting and processing the environment information of the vehicle and its surroundings. Three-dimensional target detection, as a crucial task in perception, needs to obtain the three-dimensional information of vehicles, pedestrians and other obstacles in the surrounding environment, including the center point position, size and orientation of the target.

[0003] Using single-frame data for detection inevitably has the problem of limited perception angle and occlusion between targets. Due to the movement of vehicles and the continuous acquisition of sensors, the same scene will be captured at different times and angles, forming a sequence of multi-frame time-series point clouds. The multi-frame point clouds across time can provide information of different perspectives of three-dimensional targets. Therefore, it is of great significance to combine historical information and current frame data and explore multi-frame time-series fusion networks to better understand the surrounding environment.

[0004] The most intuitive fusion method is to convert the point clouds in multiple frames to a unified coordinate system and then splice them together to form a dense point cloud frame, and then complete target detection based on a single-frame detector. This method directly performs time-series fusion at the data layer, also known as frame stacking. It can help static or slowly moving objects to construct a more complete three-dimensional profile, but when the target moves quickly or a longer time range is considered, it will cause serious trailing problems, which is not conducive to the regression of the detection box, and the feasibility will be greatly reduced. Traditional recurrent neural network models such as LSTM and GRU usually need to input different time-series data into the network respectively, extract their own features, and then perform stage-by-stage fusion. Time-series information faces the problem of feature weakening or loss during transmission, and cannot be processed in parallel. The Transformer structure can directly capture the relevance between time-series frames and perform targeted fusion while preserving the features of each frame. In addition, the Transformer also has the ability to process time-series data in parallel, thereby improving the fusion speed. These outstanding performances show that using the Transformer for time-series fusion to improve the accuracy and stability of target detection has great research value.

[0005] In the existing timing model, 3DVID proposes an attentive spatiotemporal transformer GRU (ASTGRU) to aggregate spatiotemporal information, which enhances the traditional ConvGRU through an attention memory gating mechanism. However, its defect is that it needs to be fused in series in multiple frames, and cannot process data in parallel, so the fusion efficiency is low.

[0006] The classic algorithm CenterPoint for three-dimensional target detection directly uses the method of stacking frames in the timing processing. Although this method (Center-based 3D Object Detection and Tracking Tianwei Yin Xingyi ZhouPhilipp Krahenbiihl) can help static or slowly moving objects to construct a more complete three-dimensional profile, when the target moves quickly or a longer time range is considered, a serious trailing problem will occur, which is not conducive to the regression of the detection frame. SUMMARY

[0007] The purpose of the present application is to make up for the above-mentioned defects in the prior art research, and to provide a timing point cloud three-dimensional target detection based on Transformer, which utilizes the advantage of parallel processing data of Transformer to make the multi-frame timing fusion more efficient, and aims to fully utilize the timing information to improve the target detection performance.

[0008] The present application is realized at least by one of the following technical solutions.

[0009] The timing point cloud three-dimensional target detection based on Transformer constructs and trains a multi-frame timing fusion model, including the following steps:

[0010] S1, a K-frame timing point cloud sequence is established, and each frame of point cloud is converted to a unified description in the current frame coordinate system;

[0011] S2, each frame of point cloud is voxelized and spatial feature extraction is performed using 3D sparse convolution to generate a pseudo-2D feature map in the bird's eye view;

[0012] S3, a multi-scale feature fusion network is used to further extract features and fuse multi-scale features of the bird's eye view feature map of each frame to generate a feature map combining high-level semantic information and low-level spatial information;

[0013] S4, a TSA center point prediction network is used to generate spatiotemporal attention, and the multi-frame feature map is weighted and fused to generate a center point prediction result of the target;

[0014] S5, using a Transformer multi-frame fusion network to mine the inter-frame target correlation features of K-frame time sequence point clouds, to realize time sequence feature enhancement of the target;

[0015] S6, using a regression detection head to extract and transform the target features, to obtain various regression parameters of the target frame, and to decode the regression parameters of the target frame and the target center point prediction results obtained by the TSA network in step S4 to obtain the final detection results;

[0016] S7, defining a loss function of the model, using a data set to train the model and evaluate the detection results.

[0017] Further, in step S1, according to the real-time global pose of the ego vehicle and the real-time extrinsic parameters of the sensor, the K-1 frame sequence point clouds other than the current frame are converted through the following formula:

[0018]

[0019] In the above formula, P i represents the point cloud coordinate system of the i-th frame, C i represents the ego vehicle coordinate system of the i-th frame, represents the laser radar extrinsic parameter transformation matrix from the i-th frame point cloud coordinate system to the ego vehicle coordinate system; G i represents the vehicle global coordinate system of the i-th frame, represents the global pose of the ego vehicle of the i-th frame; G K represents the vehicle global coordinate system of the K-th frame, represents the global coordinate system transformation matrix from the i-th frame to the K-th frame, which is an identity matrix; C K represents the ego vehicle coordinate system of the K-th frame, represents the global pose of the ego vehicle of the K-th frame; P K represents the point cloud coordinate system of the K-th frame, represents the transformation matrix from the K-th frame ego vehicle coordinate system to the point cloud coordinate system, i.e., the K-th frame laser radar extrinsic parameter, represents the point cloud coordinate system change matrix from the i-th frame to the K-th frame.

[0020] Further, the specific process of step S2 is as follows:

[0021] S21, point cloud voxelization: voxelize the multi-frame point clouds respectively, wherein the voxelization coordinate range is x∈[-54m, 54m], y∈[-54m, 54m], z∈[-5m, 3m], and the voxelization resolution is 0.075m×0.075m×0.2m, thereby generating a three-dimensional voxel feature with a size of 16×1440×1440×40, wherein 16 is the number of feature channels, and m represents meters;

[0022] S22, using a 3D sparse convolution network to perform feature extraction on the voxelized features.

[0023] Further, the 3D sparse convolution network includes six layers of subnetworks in series, uses two 3D sparse convolution operations, submanifold 3D sparse convolution SubMConv3d and spatial 3D sparse convolution SparseConv3d, SubMConv3d maintains the sparsity of voxels, and SparseConv3d expands the receptive field, and finally stacks the features along the height dimension to generate a pseudo 2D feature map in the bird's eye view.

[0024] Further, step S3 uses a multi-scale feature fusion network to upsample and downsample the bird's eye pseudo 2D features in step S2, obtains multi-scale feature maps and performs fusion, and in the process of downsampling and upsampling, a channel spatial attention mechanism CBAM module is connected in series.

[0025] Further, the CBAM module is an attention module, the CBAM module is embedded into the multi-scale feature fusion network, attention features are generated in the channel and spatial dimensions respectively, and the original input feature map is multiplied to perform adaptive feature correction to generate the final feature map.

[0026] Further, the TSA center point prediction network in step S4 adaptively learns the feature contribution degree of different spatial positions of the sequence frame and performs feature weighted fusion to generate a heat map representing the distribution of the center point position of the current frame target Where H and W are the heat map sizes, and Cls represents the number of target categories.

[0027] Further, the Transformer multi-frame fusion network in step S5 includes a self-attention module, a deformable cross-attention module, and a feedforward network; the self-attention module is used for interaction between different targets in the same frame, the deformable cross-attention module is used for feature association of the same target between different frames, and the feedforward network is used to improve the nonlinear ability of the network.

[0028] Further, in step S6, regression detection heads are used to obtain various regression parameters of the target box, the regression detection heads include a shared feature extraction part and four types of regression parameter detection parts, and the final parameters of the network prediction true value and the detection box are generated through parameter encoding and decoding.

[0029] Further, a loss function Loss is defined, and the loss function includes a classification loss L cl s, a regression loss L reg , and a key point loss L key :

[0030] Loss=λ1L cls +λ2Lreg + lambda3L key

[0031] Wherein, lambda1, lambda2, lambda3 are weights of each type of loss respectively; the classification loss acts on the heat map of the TSA center point prediction network output For the imbalance problem of positive and negative samples, FocalLoss is used for classification loss calculation; the regression loss acts on the parameter output of the regression detection head, and L1 loss is used to calculate the error between the regression parameters of the positive sample and the true value; the key point loss is used as auxiliary supervision, the key point is defined as the four midpoints of the target bounding box in the bird's eye view and the target midpoint, and the key point heat map is also generated as the training true value, and L2 loss is used to calculate the error.

[0032] The present application has the following advantages and beneficial effects relative to the prior art:

[0033] (1) The present application discloses a kind of based on Transformer's time sequence point cloud three-dimensional target detection, consider the continuity of space-time, design a kind of Transformer multi-frame time sequence fusion model MTF, utilize the self-attention and cross-attention of Transformer to excavate the target association features between time sequence frames, to enhance the feature expression of target, improve the accuracy and stability of three-dimensional target detection

[0034] (2) The present application discloses a kind of based on Transformer's time sequence point cloud three-dimensional target detection, in the design of Transformer fusion mechanism, propose a kind of deformable cross-attention mechanism, can autonomously learn to sample key position in space, better capture the feature dependence relationship of long distance between frames.

[0035] (3) The present application discloses a kind of based on Transformer's time sequence point cloud three-dimensional target detection, propose a kind of TSA time sequence fusion center point prediction module, can combine time feature and space feature organically by space-time attention, produce more accurate center point prediction result by means of time sequence information, and provide more high-quality feature input for Transformer module.

[0036] (4) The present application discloses a kind of based on Transformer's time sequence point cloud three-dimensional target detection, propose to use the target center point of current frame as the initialization of Transformer decoder query object, this strong prior information reduces the learning difficulty of network, so that only using one layer decoder structure can replace traditional six-layer decoding architecture, reduce the parameter quantity and calculation amount of network. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1A dynamic and static target timing superimposed frame diagram in an embodiment of the present application;

[0038] Figure 2 A structure diagram of a three-dimensional target detection system based on a multi-frame point cloud timing fusion Transformer in an embodiment of the present application;

[0039] Figure 3 A multi-frame point cloud coordinate system conversion process diagram in an embodiment of the present application;

[0040] Figure 4 A 3D sparse convolution network structure diagram in an embodiment of the present application;

[0041] Figure 5 A SubMResBlock sub-module network structure diagram in an embodiment of the present application;

[0042] Figure 6 A multi-scale feature fusion network structure diagram in an embodiment of the present application;

[0043] Figure 7 A TSA center point prediction network structure diagram in an embodiment of the present application;

[0044] Figure 8 A Transformer multi-frame fusion network structure diagram in an embodiment of the present application;

[0045] Figure 9 A multi-head self-attention network structure diagram in an embodiment of the present application;

[0046] Figure 10 A deformable multi-head cross-attention network structure diagram in an embodiment of the present application;

[0047] Figure 11 A feedforward network structure diagram in an embodiment of the present application;

[0048] Figure 12 A regression detection head network structure diagram in an embodiment of the present application;

[0049] Figure 13 A result diagram of various errors of different modes of an MTF model in an embodiment of the present application;

[0050] Figure 14 A flowchart of a three-dimensional target detection process based on a timing point cloud Transformer in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0052] Figure 1 For the dynamic and static target timing superimposed frame diagram in the embodiment of the application, it can be seen that the method of directly splicing multiple frames of point clouds can help to construct a more complete three-dimensional profile for static or slowly moving objects, but when the target moves quickly or a longer time range is considered, a serious tailing problem will occur, which is not conducive to the regression of the detection box, and the feasibility of the method will be greatly reduced. Therefore, an effective timing fusion mechanism needs to be designed to deal with targets in different motion states and different scenes.

[0053] As shown in Figure 2 , a three-dimensional target detection system based on a Transformer multi-frame point cloud timing fusion system includes a multi-frame timing fusion model (MTF model). The model includes a 3D feature extraction module, a multi-scale feature fusion network, a TSA center point prediction network, a Transformer multi-frame timing fusion network, and a regression detection head. The 3D feature extraction module is used for voxelization and 3D feature extraction of point cloud data. The multi-scale feature fusion network fuses high-level abstract semantic features and low-level spatial features to improve feature expression from a bird's eye view. The TSA center point prediction network combines timing information to improve the prediction accuracy of the target center point by generating spatial attention. The Transformer multi-frame timing fusion network captures long-range feature dependencies between frames and uses inter-frame timing information to form a more complete feature expression for the detected target. The regression detection head produces the final regression parameters of the detection box.

[0054] As shown in Figure 14 , a method for implementing a three-dimensional target detection system based on a Transformer multi-frame point cloud timing fusion system is implemented, which includes constructing and training a multi-frame timing fusion model (MTF model), specifically including the following steps:

[0055] S1, for continuous multiple frames of point cloud data, set K frames as a group of timing frame sequence, the first frame as the starting frame, the Kth frame as the key frame, and perform multiple spatial transformations on all historical point cloud frames according to the real-time pose of the vehicle and the sensor extrinsic parameters.

[0056] As an embodiment, as shown in Figure 3 , perform multiple spatial transformations on all historical point cloud frames according to the real-time pose of the vehicle and the sensor extrinsic parameters, and convert to a unified description in the current key frame coordinate system:

[0057]

[0058] Where, P i represents the point cloud coordinate system of the ith frame, C i represents the ego coordinate system of the ith frame, Li represents the LiDAR extrinsic transformation matrix from the point cloud coordinate system of the i-th frame to the ego vehicle coordinate system. i G represents the vehicle global coordinate system of the i-th frame, P represents the ego vehicle global pose of the i-th frame. K G represents the vehicle global coordinate system of the K-th frame, G represents the global coordinate system transformation matrix from the i-th frame to the K-th frame, which is an identity matrix. K G represents the ego vehicle coordinate system of the K-th frame, P represents the ego vehicle global pose of the K-th frame. K G represents the point cloud coordinate system of the K-th frame, G represents the transformation matrix from the ego vehicle coordinate system of the K-th frame to the point cloud coordinate system, i.e., the LiDAR extrinsic parameter of the K-th frame. G represents the point cloud coordinate system transformation matrix from the i-th frame to the K-th frame.

[0059] S2, the 3D feature extraction module voxelizes each frame of point cloud, and then uses 3D sparse convolution to extract features, and stacks the features along the height dimension to generate a 2D feature map in the bird's eye view perspective;

[0060] S21, point cloud voxelization

[0061] For multiple frames of point cloud, first, voxelize each frame of point cloud, in this embodiment, the voxelization range is x∈[-54m, 54m], y∈[-54m, 54m], z∈[-5m, 3m], the voxelization resolution is 0.075m×0.075m×0.2m, m represents the unit, so as to generate a three-dimensional voxel feature with a size of 16×1440×1440×40, wherein 16 is the number of feature channels.

[0062] S22, 3D sparse convolution feature extraction

[0063] The 3D sparse convolution network as shown in Figure 4 is used to perform feature extraction operation on the voxelized features. The 3D sparse convolution network mainly extracts features step by step by six sub-networks, wherein two kinds of 3D sparse convolution operations, sub-manifold 3D sparse convolution SubMConv3d and spatial 3D sparse convolution SparseConv3d, are used, SubMConv3d maintains the sparsity of the voxel, and SparseConv3d expands the receptive field. The first layer of the network is SubMConv3d followed by a BN layer and a ReLU activation function, and the second layer is two SubMResBlock sub-modules connected in series. The network structure of the sub-module is as shown in Figure 5As shown, SubMConv3d maintains the sparsity of voxels, SparseConv3d expands the receptive field, and the residual block idea of ResNet is adopted, wherein one path is SubMConv3d followed by a BN layer and a ReLU activation function, and then followed by another SubMConv3d with only a BN layer, and the other path is directly merged with the output of the one path through a jump connection, and the element-wise addition is followed by ReLU activation as the output, and the module uses the residual connection to make the network have the ability to efficiently extract features.

[0064] The third, fourth and fifth layers of the feature extraction module are all SparseBlock modules, each of which is in the form of SparseConv3d followed by a BN layer and a ReLU activation function, and two layers of SubMResBlock in series, and the sixth layer is SparseConv3d followed by a BN layer and a ReLU activation function. Finally, the non-empty voxels in the input are re-integrated into the entire voxel grid after being extracted by the 3D sparse convolution network, generating a feature map with a size of 128x180x180x2, where 2 represents the height dimension. Then the features are stacked along the height dimension to generate a feature map with a size of 256x180x180, realizing dimension reduction of the features for further processing by 2D convolution.

[0065] In the entire feature extraction process, two kinds of 3D sparse convolution are mainly used to extract three-dimensional spatial features. Among them, SubMConv3d only calculates the convolution output when the non-empty voxels are covered in the center of the convolution kernel, and the output voxel features are the same as the input voxel features, which maintains a reasonable amount of calculation. SparseConv3d is similar to ordinary 2D convolution, and as long as the non-empty voxels are covered in the entire convolution kernel range, the output will be calculated, which can expand the receptive field, but at the same time will weaken the sparsity of voxel features, resulting in an increase in calculation amount. Therefore, in combination with the convolution stride of stride=2, the feature map size will be halved and the number of feature channels will be doubled every time SparseConv3d operation is performed. The 3D feature extraction module proposed in the present application combines the use of the two types of 3D sparse convolution, which maintains the sparsity of voxels while expanding the receptive field, so as to achieve the purpose of efficiently extracting features. All input time series frames are preliminarily spatially extracted by the network, and the network parameters are shared between different frames.

[0066] S3, use Figure 6 The multi-scale feature fusion network shown further extracts features from the 2D feature maps of each frame, obtains multi-scale feature maps and fuses them to fuse high-level abstract semantic features and low-level fine-grained spatial features.

[0067] As an embodiment, the multi-scale feature fusion network first uses a layer of convolution to perform feature transformation on the bird's eye view feature with an input size of 180x180x256, to obtain a feature x with the same size, which is 8 times down-sampling size relative to the original point cloud voxelization resolution (1440x1440). Then, a convolution with a stride of 2 is used to complete further down-sampling operation, to obtain a bird's eye view down-sampling feature x_down which is 16 times down-sampling size relative to the original voxelization resolution, which is rich in high-level semantic information. Then, the x_down is up-sampled to obtain a feature map with the same size as the input feature x, and the two are added and then passed through a layer of convolution to fully fuse the mixed features x_mix of the two scales.

[0068] In order to ensure the high resolution of the center point prediction in the following, one time up-sampling processing is performed on the mixed features, and finally a feature map x_up with a size of 360x360x256 is obtained. After the down-sampling and up-sampling processing of the multi-scale features, a channel spatial attention mechanism CBAM (Convolutional Block Attention Module) module is connected in series, so as to guide the network to learn more attention features in the process of generating multi-scale features.

[0069] The CBAM module is a lightweight attention module that can be embedded into any backbone network to improve performance. It generates attention features in the channel and spatial dimensions respectively, and multiplies the original input feature map to perform adaptive feature correction and generate the final feature map. The addition here can guide the network to learn more attention features in the process of generating multi-scale features by up-sampling and down-sampling.

[0070] S4, for K frame point cloud feature maps, a TSA center point prediction network is used to generate spatio-temporal attention and to weight and fuse the temporal features and spatial features, the TSA center point prediction network can adaptively learn the feature contribution degree of different spatial positions of the historical frames and perform feature weighting fusion, to generate a heat map representing the distribution of the center point position of the target in the current frame, and to generate the center point prediction result of the target.

[0071] The network structure is as follows Figure 7As shown, for the input K-frame time point cloud features, first, a convolution kernel size of 1x1 is used for dimension reduction operation to reduce the parameter quantity, and a feature map with a size of HxWx1 is obtained, and the convolution parameters of this layer are shared between frames. Then the feature maps are stacked along the channel dimension to obtain a feature map with a size of HxWxK. Then, after a layer of convolution and Sigmoid activation function, a feature map with the same size is obtained, and its physical meaning is the importance of the features of each frame point cloud at different spatial positions. The attention weight is applied to the input K-frame point cloud features to obtain the weighted multi-frame feature map. Finally, after two layers of convolution, and activated by the Sigmoid activation function, a heat map representing the distribution of target center point positions is generated where Cls represents the number of target categories. All positions in the heat map with probability values exceeding the threshold value are regarded as candidate center points of the target, and the features at the corresponding positions are taken out as the input of the subsequent Transformer decoder.

[0072] S5, using a Transformer multi-frame fusion network to mine the target correlation features of K-frame time point clouds, capturing the long-distance feature dependency between frames, so as to fully utilize the inter-frame temporal information to form a more complete feature expression of the detected target, and realizing the temporal feature enhancement of the target.

[0073] The overall network architecture is as shown in Figure 8 The Transformer multi-frame fusion network includes self-attention, deformable cross-attention, and feedforward network parts. The self-attention module is used for the interaction between different targets in the same frame, the deformable cross-attention module is used for the feature correlation of the same target between different frames, and the feedforward network is used to improve the non-linear ability of the network. In order to pass information deeper and enhance the fitting ability of the MTF model, the network uses a residual structure, and uses layer normalization (LN) to normalize the calculated features of each layer.

[0074] As a preferred embodiment, the temporal feature enhancement of the target is realized, and the specific process is as follows:

[0075] S51, the multi-head self-attention module is composed of h self-attention heads, h=6, the input feature of each attention head is 256 dimensions, and the output feature is 64 dimensions, and the network structure is as shown in Figure 9 First, the Top N predicted confidence higher target center point P corresponds to the bird's eye view feature f in is taken out to form an Nx256 feature vector. At the same time, a linear layer with an input feature dimension of 2 (i.e. the bird's eye view coordinate dimension of the center point) is used to encode the center point position, generating a position encoding vector with an output feature of 256 dimensions, which can be represented as:

[0076] E pos =We ·P pos

[0077] where W e is the learnable parameter of the linear position encoding layer, P pos is the N x 2 center point position vector, E pos is the generated N x 256 position encoding vector.

[0078] The center point feature fi in is then added to the position encoding vector E pos , and after LN layer normalization, the first l attention head full connection layer is used to obtain the query matrix Q, the key matrix K and the value matrix V with feature dimension of 64 respectively:

[0079]

[0080]

[0081]

[0082] In the above formula, the subscripts i, j represent different index positions of the N center points, LN(·) represents the LN normalization operation, are the parameters of the Query, Key, Value three-layer full connection layer of the first l attention head respectively. is the query vector, the key vector and the value vector of the first l attention head with index position i; is the position encoding vector with index positions i, j;

[0083] Then the query matrix Q and the key matrix K are related, and the feature channel number d q is scaled, and the normalization is realized by the Softmax function to obtain an attention matrix A representing the correlation between targets, representing the attention value of the target with index position i to the target with index position j under the first l attention head:

[0084]

[0085] The attention matrix A acts on the value matrix V to obtain the 64-dimensional feature output matrix V out_l of the first l attention head, representing the feature vector of the target with index position i

[0086]

[0087] The features of the 6 attention heads are spliced to generate 384-dimensional features, which are then transformed by a fully connected layer to obtain 256-dimensional features again, and added to the input features of the module through a residual connection. The final output is obtained through a Dropout layer with a dropout rate of 0.3:

[0088]

[0089] In the above formula, W o is the parameter of the output fully connected layer, f out is the output feature of the self-attention module.

[0090] Through the above operation, the features between all target center points are interacted, which helps the target to obtain global information and learn the position dependence between each other, and the corrected target center point feature is obtained. The center point feature will be used as the input of the subsequent deformable multi-head cross-attention module.

[0091] S52, the deformable multi-head cross-attention module is composed of h=6 deformable cross-attention heads, and the network structure is as shown in Figure 10 The network generates multiple sampling points of the target on the BEV feature map of the K frame sequence from the query end, and only needs to calculate the cross-attention of a limited number of sampling points. After training, the network will learn the important sampling points closely related to the current frame query in the sequence frame. This not only avoids the defect that the MTF model cannot capture long-distance target dependence due to the limited size of the manually preset attention calculation window, but also avoids the problem of explosive increase in memory and calculation caused by blindly increasing the size of the attention calculation window.

[0092] Specifically, each attention head will perform the following operations in parallel. It should be noted that the letters l and m in the following formula represent the lth attention head and the mth frame in the sequence frame, respectively.

[0093] First, the bird's eye view feature f in is normalized by the LN layer, and then a query matrix Q l is generated by the fully connected layer. At the same time, another fully connected layer generates the sampling point offset P on the bird's eye view of each sequence frame. Each target center is set with 15 sets of sampling points on each sequence frame. The offset is added to the position of each target center to obtain the absolute position of all sampling points on the bird's eye view.

[0094]

[0095]

[0096]

[0097] In the above formula, parameters of the fully connected layer representing Query, parameters of the fully connected layer for generating the offsets of the sampling points, ΔP l,m offsets of the sampling points of the bird's eye view of the sequence frames, absolute coordinates of the bird's eye view of each sampling point.

[0098] Then the features of each sampling point are extracted from the bird's eye view features F m of each sequence frame. Among them, the features of all non-integer sampling points are obtained by bilinear interpolation using the features of their adjacent four pixel points. Next, the key matrix K l,m and the value matrix V l,m are generated respectively via the fully connected layer. Dot product operation is performed on Q l and K l , and feature scaling is performed on the feature channel number d q , and then normalized by the Softmax layer to generate an attention weight matrix A l , which represents the degree of association between each target center and all the sampling points generated by it. The attention matrix acts on the value matrix V l,m , and the weighted sum of the sampling point V values in all K frames of the sequence frame is obtained to obtain a 64-dimensional feature output fused with the time sequence features.

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] In the above formula, are the parameters of the Key and Value two fully connected layers of the lth attention head, respectively.

[0105] The features of multiple attention heads are spliced to obtain 384-dimensional features, which are then transformed by a one-level fully connected layer to obtain a 256-dimensional feature output with the same dimension as the input, and are added element by element with the input. Via a Dropout layer with a dropout rate of 0.3, the final output of the module is obtained.

[0106]

[0107] The cross-attention mechanism can adaptively find the correlation strength of features at different positions between the front and back frames according to the correlation of the features, give higher weights to the features at the positions with high correlation, weaken the features at the positions with low correlation, and make the network pay more attention to the features in the target area in the temporal frame fusion, so as to achieve better fusion effect. In addition, compared with the fixed attention calculation window of the conventional Transformer cross-attention mechanism, the deformable cross-attention mechanism can make the network learn to sample and weight fuse only some key feature positions. In real scenes, targets with different sizes, orientations and motion states often occupy different areas in the feature space. Therefore, the deformable cross-attention mechanism is more robust in dealing with various actual scenes.

[0108] S53, the feedforward network is composed of two full connection layers, and the network structure is as shown in Figure 11 . The feature input and output dimensions of the two full connection layers are both 256 dimensions, and both use Dropout with a decay rate of 0.3. The output of the first full connection layer is activated by the GELU activation function. Since the self-attention and cross-attention are basically linear transformations, the addition of the feedforward network here provides more nonlinearity for the network, improving the expression ability of the MTF model.

[0109] S6, the regression detection head is used to extract and transform the target features, obtain various regression parameters of the target box, and decode the real parameters of the detection box together with the target center point prediction results obtained by the TSA network in step S4.

[0110] The network structure of the regression detection head is as shown in Figure 12 . First, a shared detection head is used to extract and transform the features, a 1D convolution with a convolution kernel size and a step size of 1 is used to convert the 256-dimensional input center point features into 64-dimensional intermediate features, and a BN layer and a ReLU activation function are used for activation. Then, four detection heads are used to predict four types of regression parameters, i.e., center point offset, height, size, and orientation angle. Each detection head first uses a 1D convolution with a convolution kernel size and a step size of 1 to extract features, and keeps the feature dimension unchanged at 64. After a BN layer and a ReLU activation function, a 1D convolution is used to obtain various regression parameters.

[0111] The parameters of the detection box are composed of the center point position predicted by the TSA network and the regression parameters predicted by the regression detection head. In the process of generating the regression true value of the positive sample, the center point coordinates (x g ,y g ,z g ), the length, width and height w g 、h g 、l gThe rotation angle θ of the frame g The annotation box is used to label parameters (x) g ,y g ,z g ,w g ,h g ,l g ,θ g Instead of directly mapping the information of the ground truth bounding boxes to the corresponding locations on the feature map and encoding them, the TSA network and a regression detection head are used to predict the encoded values. Among them, the grid position of the target center point The calculation formula is:

[0112]

[0113] In the formula, P x P y V represents the starting value of the left boundary of the point cloud frame space along the x and y axes. x V y S represents the length and width of the voxel, respectively. x S y These represent the scaling ratios of the feature map along the x-axis and y-axis, respectively. This represents the coordinates of the center point on the heatmap. The true value of the heatmap generated at the center point is Y∈[0,1]. Cls×H×W At that time, not only is the grid value of the target center point set to 1, but all areas covered by the Gaussian radius centered on that point will be assigned a non-zero value.

[0114] In the regression parameter encoding, to reduce the difficulty of regression and keep the range of each regression value within a relatively low variance, the horizontal offset of the center point is normalized to (Δx, Δy) using the heatmap pixel size, while the vertical height Δz of the center point is directly regressed. The length, width, and height of the target box are encoded using logarithmic functions as (Δw, Δh, Δl), and the orientation angle is encoded using sine and cosine functions as (θ). x ,θ y The specific coding formula is as follows:

[0115]

[0116] For the output prediction parameters of the detection head, the following decoding formula is used to restore the prediction results to the true position (x,y,z), length, width, and height (w,h,l) of the detection box and the actual rotation angle θ of the box, thus obtaining the actual parameters of the detection box:

[0117]

[0118] S7. Define the loss function, and use the public dataset nuScenes to train the proposed model and evaluate the detection results.

[0119] The loss function consists of classification loss L cls Regression loss L reg and key point loss L key The three components are: classification loss, regression loss, keypoint loss, and regression parameters, respectively. Classification loss is applied to the target center point prediction heatmap in step S4; regression loss is used for the regression parameters predicted by the regression detection head in step S6; and keypoint loss is used as an auxiliary supervision term.

[0120] Loss=λ1L cls +λ2L reg +λ3L key

[0121] Where λ1, λ2, and λ3 are the weights of various types of losses, respectively.

[0122] In a preferred embodiment, λ1, λ2, and λ3 are set to 1, 0.25, and 1, respectively.

[0123] The heatmap output by the TSA centroid prediction module is affected by the classification loss. Where H and W are the heatmap dimensions, and Cls is the number of target classes. To address the imbalanced positive and negative sample problem, FocalLoss is used to calculate the classification loss:

[0124]

[0125] In the formula y represents the network's prediction result for the index position (i,j) on the heatmap of the center point of the c-th category. ijc N represents the true value of the Gaussian heatmap at that location. pos γ represents the number of positive samples. γ is the attenuation coefficient in FocalLoss that controls the difficulty of classifying samples; in this embodiment, it is set to 2. The β exponent term is introduced to suppress the loss of pixels near the center point in the Gaussian heatmap ground truth; these regions' y... ijc ∈[0,1], and the value of β is set to 4 in this embodiment.

[0126] In a preferred embodiment, γ is set to 2 and β is set to 4.

[0127] The regression loss is applied to the parameter output of the regression detector head. L1 loss is used to calculate the error between the regression parameters of positive samples and the true values. The loss calculation formula is:

[0128]

[0129] In the formula, b represents the regression quantity. b represents the regression prediction result of the network on the sample. gt N represents the true value of the sample.pos is the number of positive samples. In addition, it should be noted that in the training stage, in order to accelerate the convergence speed of the MTF model, the embodiment does not select the top N target center points with high confidence of the center point prediction heat map as the input of the Transformer multi-frame fusion module, but directly calculates the position of the center point of the data set annotation box under the BEV feature map and takes its feature as the input, which makes the regression target of the subsequent detection head directly correspond to the label target, without the need to bind the regression target by performing the Hungarian matching algorithm. In the prediction stage, the top N center points are selected from the generated center point heat map as the input of the Transformer module.

[0130] In addition to the above center point heat map classification loss and regression loss, a key point loss is introduced as auxiliary supervision to improve the performance of the model. Inspired by the two-stage feature extraction of the Centerpoint model, the key points are defined as the four midpoints of the target bounding box in the bird's eye view and the target midpoint. In order to realize the key point loss, a key point heat map prediction branch is added beside the center point heat map detection head to predict the position information of these key points For each detection box, the Gaussian heat map supervision true value of the target key point is generated using the same method as calculating the center point heat map, and the key point loss is calculated using the mean square error at the position where the Gaussian heat map value true value is greater than zero:

[0131]

[0132] In the formula, represents the prediction result of the network at the index position (i, j) on the key point heat map of the cth class, k ijc represents the Gaussian heat map true value at this position, N pos is the number of positive samples. The introduction of this loss helps the MTF model to better focus on and mine the features of the target region, improve the accuracy of the center point prediction of the model, and thus improve the performance of the entire three-dimensional target detection model.

[0133] Then the method proposed in the application is experimentally verified on the automatic driving public data set nuScenes, which is collected by two electric vehicles with the same sensor configuration. Each vehicle is equipped with a 32-line-beam lidar with 360° field of view perception capability. The data collection sites include different scenes such as urban, residential, suburban, industrial areas in Boston and Singapore, and also cover different time periods and weather conditions such as day, night, sunny, rainy, cloudy, etc. Each scene in the data set is a 20-second sequence. On the target detection task, a frame is annotated every 0.5 seconds, which is called a key frame. The labeled objects include ten categories of targets such as small cars, pedestrians, motorcycles, etc., and include dynamic and static motion attributes. The data set also provides pose estimation information for each frame of each sensor and vehicle. Here, 700 scenes are used for training, and 150 scenes are used for verification, corresponding to 28130 and 6019 key frame samples respectively.

[0134] In terms of evaluation indicators, the data set proposes two main evaluation indicators: mean average precision (mAP) and nuScenes detection score (NDS). Among them, the mAP indicator uses 0.5m, 1m, 2m, and 4m bird's eye center distance matching thresholds to define positive and negative samples. The mAP indicator is calculated as follows:

[0135]

[0136] Among them, represents the category set, represents the distance threshold set, AP c,d represents the detection accuracy of category c and distance threshold d. NDS focuses on evaluating the prediction accuracy of all true positive samples TP. These indicators include average translation error (ATE), average scale error (ASE), average orientation error (AOE), average velocity error (AVE), and average attribute error (AAE). NDS is the average of these five types of indicators, calculated as:

[0137]

[0138] Among them, represents the error indicator set, TP c represents the error value of category c.

[0139] The experimental results of the present embodiment on the nuScenes dataset are shown in Table 1 below. The main indicators to be investigated are mAP and NDS, as well as the AP indicators of the eight common obstacle categories. The English abbreviations in the table are: Pedstrian (Ped), Traffic Cone (T.C.), and Construction vehicle (C.V.). The symbol '-' in the table indicates that the model literature source lacks the corresponding data. The experimental results show that the overall indicators of the proposed multi-frame temporal fusion model MTF all surpass these classic temporal algorithms, and compared with the 3D target detection algorithm CenterPoint under the overlap frame condition, MTF achieves an indicator improvement of 5.4 mAP and 3.3 NDS. The experimental results show that, compared with static obstacles such as traffic cones and barrels, MTF has a greater improvement in dynamic categories such as vehicles and pedestrians, which also demonstrates the effectiveness and superiority of the method in temporal fusion.

[0140] Table 1 Performance comparison results of classic temporal algorithms

[0141] Method Mode mAP NDS Car Ped Bus T.C. Truck Motor C.V. Bicycle 3DVID Timeline 45.4 — 79.7 76.5 47.1 58.8 33.6 40.7 18.1 7.9 TCTR Timeline 50.5 — 83.2 74.9 63.7 52.5 51.5 54.0 15.6 22.6 STGNN Timeline 59.0 66.7 86.2 85.5 67.2 71.3 57.2 58.1 14.6 37.0 CenterPoint Overlay 58.0 65.5 84.6 83.4 60.2 76.7 51.0 53.7 17.5 28.7 MTF Timeline 63.4 68.8 86.8 88.2 71.6 76.9 61.4 75.9 22.0 44.8

[0142] In order to more fairly verify the effectiveness of the designed time sequence fusion mode of the Transformer in the MTF model, three different time sequence data processing modes are used in the experiment. Including single frame, stacking frame and fusion. Among them, single frame refers to inputting only the feature map of the current frame as the query object in the cross attention module of the Transformer multi-frame fusion. Stacking frame refers to splicing the sequence frames into a dense point cloud single frame, and then the model uses the same processing method as the single frame. Fusion refers to inputting the feature maps of the current frame and the historical frame as the query object in the cross attention module, that is, the processing method introduced earlier. For the three different processing methods, the mAP and NDS and the AP index of the three common targets (car, pedestrian and motorcycle) in target detection are mainly analyzed. Since the nuScenes dataset is very large, it takes a long time to train each time, so the subsequent experiments are all carried out on the training subset of 100 scenes sampled from the full training set. The experimental results are shown in Table 2, it can be seen that the method of using stacking frame can improve 0.54 mAP and 8.19 NDS, and the method of using fusion can further improve 1.25 mAP and 4.87 NDS compared with stacking frame. And compared with pedestrians, the time sequence fusion method proposed by the MTF model has higher detection accuracy for vehicles with faster moving speed, because the stacking frame method does not establish the concept of point cloud time sequence, and all sequence frames are treated equally. The tailing phenomenon caused by the fast moving target will not be conducive to the regression of the detection box. And the time sequence fusion method proposed by the present application fully excavates the feature correlation of the previous and subsequent frames through the deformable cross attention mechanism of the Transformer, forms the strong features of the target combined with the time sequence, and effectively improves the accuracy of the final parameter regression.

[0143] Table 2 Performance comparison results of different modes of MTF model

[0144]

[0145]

[0146] In order to further analyze the performance improvement ability of the time sequence fusion on the regression parameters of the detection box, the regression performance indicators and classification indicators of single frame, stacking frame and fusion are counted, including average translation error (ATE), average scale error (ASE), average direction error (AOE) and average attribute error (AAE). The results are as follows Figure 13It can be seen that the use of the method of superimposition can reduce the error value of each index, and the time sequence fusion processing method using the MTF model can further reduce each error index, among which the average direction error (AOE) index is the most obvious, and the error value is reduced by 46.62% compared with the single frame case. This is because the multi-frame fusion model can obtain the point cloud information of different perspectives of the target from the time sequence, and excavate the more complete contour and orientation characteristics of the target, which is very important for the estimation of the rotation angle of the target.

[0147] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and are included in the protection scope of the present application.

Claims

1. A Transformer-based temporal point cloud three-dimensional object detection, characterized in that, The multi-frame time sequence fusion model is constructed and trained, including the following steps: S1, a K-frame time sequence point cloud sequence is established, and each frame of point cloud is converted to a unified description in a current frame coordinate system; S2, each frame of point cloud is voxelized and spatial feature extraction is performed using a 3D sparse convolution to generate a pseudo 2D feature map in a bird's eye view; S3, a multi-scale feature fusion network is used to further extract features and fuse multi-scale features of the bird's eye view feature map of each frame to generate a feature map combining high-level semantic information and low-level spatial information; the bird's eye view pseudo 2D feature in step S2 is up-sampled and down-sampled using a multi-scale feature fusion network to obtain multi-scale feature maps and fuse them; a channel spatial attention mechanism CBAM module is connected in series in the process of up-sampling and down-sampling; the CBAM module is an attention module; the CBAM module is embedded into the multi-scale feature fusion network to generate attention features in the channel and spatial dimensions respectively, and the original input feature map is multiplied to perform adaptive feature correction to generate a final feature map; S4, using the TSA center point prediction network to generate a spatio-temporal attention, weighting and fusing multi-frame feature maps to generate a target center point prediction result; the TSA center point prediction network adaptively learns the feature contribution degree of different spatial positions of the sequence frame through the constructed spatio-temporal attention and performs feature weighting fusion, and is used to generate a heat map representing the distribution of the current frame target center point position wherein and are the heat map sizes, respectively, represents the number of target categories; S5, a Transformer multi-frame fusion network is used to mine the inter-frame target correlation features of the K-frame time sequence point cloud to realize time sequence feature enhancement of the target; the Transformer multi-frame fusion network includes a self-attention module, a deformable cross-attention module and a feedforward network; the self-attention module is used for interaction between different targets in the same frame, the deformable cross-attention module is used for feature correlation of the same target between different frames, and the feedforward network is used to improve the nonlinear ability of the network; S6, a regression detection head is used to extract and transform target features to obtain various regression parameters of the target box, and the regression parameters of the target box and the target center point prediction results obtained by the TSA network in step S4 are decoded together to obtain the final detection results; S7, a loss function of the model is defined, and the model is trained and the detection results are evaluated using a data set.

2. The Transformer-based timing point cloud three-dimensional target detection according to claim 1, wherein, In step S1, the K-1 frame sequence point clouds other than the current frame are converted through the following formula according to the real-time global pose of the ego vehicle and the real-time extrinsic parameters of the sensor: In the above formula, represent the first i frame point cloud coordinate system, represent the first i frame ego vehicle coordinate system, represent the first i frame point cloud coordinate system to the laser radar extrinsic transformation matrix of the ego vehicle coordinate system; represent the first i frame global coordinate system of the vehicle, represent the first i frame global pose of the ego vehicle; represent the Kth frame global coordinate system of the vehicle, i represent the Kth frame global coordinate system transformation matrix of the Kth frame global pose of the ego vehicle; represent the Kth frame point cloud coordinate system, represent the Kth i frame point cloud coordinate system to the Kth 3. The Transformer-based timing point cloud three-dimensional target detection according to claim 1, wherein, The specific process of step S2 is as follows: S21, point cloud voxelization: the multi-frame point clouds are voxelized respectively, wherein the coordinate range of voxelization is x∈[54m, 54m], y∈[54m, 54m], z∈[5m, 3m], and the resolution of voxelization is 0.075m×0.075m×0.2m, thereby generating a three-dimensional voxel feature with a size of 16×1440×1440×40, wherein 16 is the number of feature channels, and m represents meters; S22, a 3D sparse convolution network is used to perform feature extraction operation on the voxelized features.

4. The Transformer-based timing point cloud three-dimensional target detection according to claim 3, wherein, The 3D sparse convolution network includes six sub-networks connected in series, uses two kinds of 3D sparse convolution operations, sub-manifold 3D sparse convolution SubMConv3d and spatial 3D sparse convolution SparseConv3d, SubMConv3d maintains the sparsity of voxels, and SparseConv3d expands the receptive field, and finally the features are stacked along the height dimension to generate a pseudo 2D feature map in a bird's eye view.

5. The Transformer-based timing point cloud three-dimensional target detection according to claim 1, wherein, In step S6, each type of regression parameter of the target frame is obtained using a regression detection head, the regression detection head including a shared feature extraction part and four types of regression parameter detection parts, and the final parameters of the detection frame and the network prediction true value are generated through parameter coding and decoding.

6. The Transformer-based time point cloud three-dimensional target detection according to any one of claims 1-5, wherein, defining a loss function Loss , the loss function including a classification loss , a regression loss , and a keypoint loss : wherein, , , are the weights of each type of loss; the classification loss acts on the heat map output of the TSA center point prediction network , to address the imbalance between positive and negative samples, FocalLoss is used to calculate the classification loss; the regression loss acts on the parameter output of the regression detection head, L1 loss is used to calculate the error between the regression parameters of the positive samples and the true value; the key point loss is used as auxiliary supervision, the key point is defined as the four midpoints of the target bounding box and the target midpoint in the bird's eye view, and a key point heat map is also generated as the training true value, and L2 loss is used to calculate the error.

Images