A 3D point cloud object tracking method combining voxel and bird's eye network

Abstract

The application discloses a 3D point cloud target tracking method combining voxels and bird's eye network. The position of a previous frame of the target is taken as the origin of the coordinate system, a subset of the region of interest of the point cloud of the previous frame and the current frame is selected, is divided into a regular voxel grid and is coded; the three-dimensional spatial features of the coded grid are extracted through a voxel network sharing weights, the height information of the three-dimensional spatial features is compressed to the channel dimension to obtain bird's eye features, and then the bird's eye network outputs a feature map of a multi-scale bird's eye view; after the spatial and temporal feature fusion of the bird's eye features of different frames, a high-dimensional feature vector containing rich target motion information is obtained, the relative motion of the target between frames is obtained through a multilayer perception machine regression, and the position of the target in the current frame is predicted. Experimental results show that the method not only reaches a leading level in reasoning speed, but also far exceeds other existing methods in tracking accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to target tracking based on 3D point clouds, and more specifically to a 3D point cloud target tracking method that combines voxels and bird's-eye view networks. Background Technology

[0002] Target tracking is an important branch of computer vision and is widely used in various scenarios such as autonomous driving, video surveillance, and military guidance. In recent years, with the development of LiDAR sensors and the robustness of point clouds obtained by radar scanning to complex environments, target tracking methods based on 3D point clouds have shown great potential.

[0003] Existing point cloud target tracking methods primarily rely on point set representations, such as using networks like PointNet++ and DGCNN for feature extraction. This involves calculating the distances between points within the point cloud to obtain a relatively rudimentary feature representation, resulting in a computational complexity that reaches the square of the number of scattered points, leading to slow inference speed. Voxel and bird's-eye view representations can normalize unordered point clouds without requiring the calculation of distances between points, significantly improving inference speed. Bird's-eye view representation, in particular, better represents the target's motion on a horizontal surface. Especially in autonomous driving scenarios, where targets mostly move horizontally, bird's-eye view representation can effectively reduce noise from altitude and improve tracking accuracy; however, it has some limitations in extracting spatial structure information. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a 3D point cloud target tracking method that combines voxels and bird's-eye view networks. By utilizing the complementary characteristics of voxels and bird's-eye view networks in terms of spatial structure and horizontal motion, this method not only improves inference speed but also achieves high-precision target tracking.

[0005] A 3D point cloud target tracking method combining voxels and bird's-eye view networks specifically includes the following steps:

[0006] S1. Using the estimated target position in frame t-1 as the origin of the coordinate system, select subsets of the region of interest from the point clouds in frames t-1 and t, denoted as P respectively. t-1 and P t .

[0007] S2. Subsets of the region of interest P in frame t-1 and frame t respectively. t-1 and P t The grid is divided into regular voxel grids. Grids that do not contain point clouds are discarded. Non-empty grids are encoded, and the average 3D coordinates of all point clouds in the grid are used as the initial feature of the grid.

[0008] S3. The mesh encoded in S2 is passed through a voxel network to extract sparse features. Then, its height information is compressed to the channel dimension to obtain dense bird's-eye view features. These dense features are then input into the bird's-eye view network, and multi-scale bird's-eye view feature maps are obtained from the outputs of different downsampling layers. and {F t 1 ,F t 2 ,F t 3}

[0009] S4. Use the spatiotemporal feature fusion module to perform a bird's-eye view of the features of frame t-1 and frame t at the largest scale. F t 1 Integrate and extract motion features M 1 :

[0010]

[0011]

[0012] Where [·,·] represents concatenation along the feature dimension, Conv 3×3 This represents a convolutional module of size 3×3 with a stride of 1.

[0013] Then, regarding motion feature M i The scale and feature dimensions are downsampled to the concatenated features of the (i+1)th layer using a 3×3 convolutional module with a stride of 2. Consistent, with After summing, a convolution operation is performed to obtain the motion features M of the next layer. i+1 , i = 1, 2. The motion feature M 3 After global max pooling, we obtain the high-dimensional feature vector F. out .

[0014] S5, the high-dimensional feature vector F out Containing rich target motion information, multilayer perceptron regression is used to obtain the target's motion offsets in the X, Y, and Z axes, as well as its orientation angle offsets (Δx, Δy, Δz, Δθ) between frame t-1 and frame t. Based on these offsets, the target's three-dimensional center point (x, y, Δz, Δθ) in frame t is predicted. t ,y t ,z t ):

[0015]

[0016] The present invention has the following beneficial effects:

[0017] This method, within a simple, single-stage framework, leverages the advantages of both voxel and bird's-eye view representations to effectively extract multi-scale bird's-eye view features from point cloud data. It then utilizes spatiotemporal feature fusion to model different target motion patterns at multiple scales, ultimately generating high-quality motion estimates, thereby effectively improving tracking accuracy. Attached Figure Description

[0018] Figure 1 A schematic diagram of a 3D point cloud target tracking method that combines voxels and bird's-eye view networks. Detailed Implementation

[0019] The present invention will be further explained below with reference to the accompanying drawings;

[0020] like Figure 1 As shown, a 3D point cloud target tracking method combining voxels and bird's-eye view networks specifically includes the following steps:

[0021] S1. Using the estimated target position in frame t-1 as the origin of the coordinate system, select subsets of the region of interest from the point clouds in frames t-1 and t, denoted as P respectively. t-1 and P t .

[0022] S2. Subsets of the region of interest P in frame t-1 and frame t respectively. t-1 and P t The grid is divided into regular voxel meshes. Meshes that do not contain point clouds are discarded. Non-empty meshes are encoded, and the average 3D coordinates of all point clouds within the mesh are used as the initial feature of the mesh.

[0023]

[0024] Among them, v k This represents the k-th non-empty grid, where H is the number of point clouds within that grid, and p k (j) represents the three-dimensional coordinates (x, y, z) of the j-th point cloud within the grid.

[0025] S3. To learn discriminative features, the mesh encoded in S2 is first processed through a voxel network. The voxel network has a four-layer structure, with each layer containing two sub-stream sparse convolutional modules of size 3×3×3 and stride 1. A 3D sparse convolutional module of size 3×3×3 and stride 2 is inserted between each layer. After four layers of downsampling, the spatial resolution of the mesh becomes smaller, the features become denser, and the information becomes richer.

[0026] Then, the high-level information of the sparse features output by the voxel network is compressed to the channel dimension:

[0027] (N,C,Z,Y,X)→(N,C*Z,Y,X)

[0028] Where N, C, Z, Y, and X represent the batch size, channel dimension, and the size of the feature in the Z, Y, and X axes, respectively.

[0029] Finally, the compressed, dense bird's-eye view features are passed through a bird's-eye view network composed of a series of stacked 2D standard convolutional modules of size 3×3. Multi-scale bird's-eye view feature maps are obtained at different downsampling layers, denoted as... and {F t 1 ,F t 2 ,F t 3}

[0030] S4, Multi-scale bird's-eye view features obtained from S3 and {F t 1 ,F t 2 ,F t 3 The spatiotemporal feature fusion module first performs a bird's-eye view analysis of the features of frame t-1 and frame t at the largest possible scale. F t 1 Integrate and extract motion features M 1 :

[0031]

[0032]

[0033] Where [·,·] represents concatenation along the feature dimension, Conv 3×3 This represents a convolutional module of size 3×3 with a stride of 1.

[0034] Then, regarding motion feature M i The scale and feature dimensions are downsampled to the concatenated features of the (i+1)th layer using a 3×3 convolutional module with a stride of 2. Consistent, with After summing, a convolution operation is performed to obtain the motion features M of the next layer. i+1 :

[0035]

[0036] Where i = 1, 2, and Down represents the downsampling module.

[0037] The motion features M output from the last layer 3After global max pooling, we obtain the high-dimensional feature vector F. out .

[0038] S5, the high-dimensional feature vector F out Containing rich target motion information, the relative motion of the target between frames is obtained through multilayer perceptron regression:

[0039] (△x,△y,△z,△θ)=MLP(F out )

[0040] MLP stands for Multilayer Perceptron, which includes a fully connected neural network and a nonlinear function. (△x, △y, △z, △θ) represents the motion offset of the tracked target in the X-axis, Y-axis, and Z-axis directions, as well as the orientation angle offset, between frame t-1 and frame t.

[0041] Applying the above offset to the target's position in frame t-1 yields the target's location in the current frame:

[0042]

[0043] Where (x) t ,y t ,z t ) represents the three-dimensional center point of the target in frame t.

[0044] This embodiment conducts tracking tests on the publicly available KITTI dataset for four classes: Car, Pedestrian, Van, and Cyclist. The experimental environment included an RTX 4090 GPU with 24GB of RAM, Python version 3.9, CUDA version 11.8, and the deep learning frameworks used: PyTorch 2.0.1 and MMEngine 0.7.4. Success and Precision were used as evaluation metrics. Success represents the area under the curve (AUC) of the proportion of frames where the joint intersection (IOU) of the predicted and ground truth boxes is greater than a threshold, ranging from 0 to 1. Precision represents the AUC of the proportion of frames where the distance between the centers of the predicted and ground truth boxes is within a threshold, ranging from 0 to 2 meters. The experimental results are shown in the table below.

[0045]

[0046]

[0047] As can be seen from the data in the table above, this method is significantly better than M2Track in both Success and Precision metrics, effectively improving target tracking accuracy.

Claims

1. A 3D point cloud target tracking method combining voxels and bird's-eye view networks, characterized in that: Specifically, the following steps are included: S1. Select subsets of the region of interest from the current frame t and the previous frame (t-1)th frame, denoted as P. t and P t-1 ; S2, P t-1 and P t The grid is divided into regular voxel grids. Non-empty grids are encoded, and the mean of the three-dimensional coordinates of all point clouds within the grid is calculated as the feature of the grid. S3. Input the grid features encoded in S2 into a voxel network to extract sparse features. Then, compress the height information to the channel dimension to obtain dense bird's-eye view features. Input these dense features into the bird's-eye view network and obtain multi-scale bird's-eye view feature maps from the outputs of different downsampling layers. and {F t 1 ,F t 2 ,F t 3 }; S4. Use the spatiotemporal feature fusion module to process the bird's-eye view feature map. F t 1 Integrate and extract motion features M 1 : Where [·,·] represents concatenation along the feature dimension, Conv 3×3 This represents a convolutional module of size 3×3 with a stride of 1; Then, regarding motion feature M i The scale and feature dimensions are downsampled to the concatenated features of the (i+1)th layer using a 3×3 convolutional module with a stride of 2. Consistent size, with After summing, a convolution operation is performed to obtain the motion features M of the next layer. i+1 , i = 1, 2; The motion feature M 3 After global max pooling, we obtain the high-dimensional feature vector F. out ; S5, Based on high-dimensional feature vector F out The positional offset of the tracked target between frame t-1 and frame t is obtained by multilayer perceptron regression, thereby predicting the three-dimensional center point coordinates of the target in frame t.

2. The 3D point cloud target tracking method combining voxels and bird's-eye view networks as described in claim 1, characterized in that: In S1, the target position estimated in the (t-1)th frame is used as the origin of the coordinate system to select a subset of the region of interest in the point cloud.

3. The 3D point cloud target tracking method combining voxels and bird's-eye view networks as described in claim 1, characterized in that: The voxel network has a four-layer structure, with each pair of layers connected by a 3D sparse convolution module with a convolution size of 3×3×3 and a stride of 2. Each layer of the network includes two sub-stream sparse convolution modules with a convolution size of 3×3×3 and a stride of 1.

4. The 3D point cloud target tracking method combining voxels and bird's-eye view networks as described in claim 1, characterized in that: The method to compress the height information of the sparse features output by a voxel network to the channel dimension is as follows: (N,C,Z,Y,X)→(N,C*Z,Y,X) Where N, C, Z, Y, and X represent the batch size, channel dimension, and the size of the feature in the Z, Y, and X axes, respectively.

5. The 3D point cloud target tracking method combining voxels and bird's-eye view networks as described in claim 1, characterized in that: The bird's-eye view network comprises a series of cascaded 2D standard convolutional modules with a convolutional size of 3×3.

6. The 3D point cloud target tracking method combining voxels and bird's-eye view networks as described in claim 1, characterized in that: Down indicates the downsampling module.

7. The 3D point cloud target tracking method combining voxels and bird's-eye view networks as described in claim 1, characterized in that: (△x,△y,△z,△θ)=MLP(F out ) MLP stands for Multilayer Perceptron, which includes a fully connected neural network and a nonlinear function; (△x, △y, △z, △θ) represents the motion offset of the tracked target in the X-axis, Y-axis, and Z-axis directions and the orientation angle offset between frame t-1 and frame t.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1 to 7.

Images