Laser radar point cloud registration method for urban dynamic environment

Abstract

The present invention provides a laser radar point cloud registration method for an urban dynamic environment. The method comprises: first, establishing a data set of a point cloud registration network; then, constructing a point cloud registration network based on multi-task learning; next, training the designed point cloud registration network; and finally, using the trained network to estimate a transformation matrix of two frames of point clouds. According to the method, a three-dimensional point cloud is converted into a two-dimensional distance image, improving the point cloud processing efficiency. According to the method, two tasks of point cloud segmentation and registration are executed, and due to a shared feature extraction module, point cloud registration can be carried out on the basis of reliable features of static reference targets, reducing errors caused by dynamic targets.

Description

A LiDAR point cloud registration method for dynamic urban environments Technical Field

[0001] The present invention relates to a point cloud registration method, in particular to a laser radar point cloud registration method for urban dynamic environments, belonging to the technical field of vehicle positioning. Background Art

[0002] Positioning is a key component of smart cars, providing accurate location information for perception, decision-making, and control modules. In open urban environments, a combined navigation system integrating a global navigation satellite system (GNSS) and an inertial navigation system (INS) can meet the high-precision positioning needs of smart cars. However, in urban canyons, on tree-lined avenues, and under overpasses, satellite signals are subject to varying degrees of obstruction, making it difficult for the combined navigation system to maintain high positioning accuracy. In urban satellite-denied environments, existing vehicle positioning technologies primarily rely on wireless sensors, cameras, and LiDAR. Compared to wireless sensors and cameras, LiDAR offers advantages such as high measurement accuracy and unaffected by lighting conditions, making it the preferred choice of smart car companies such as Google and Baidu. As production technology matures, LiDAR costs have significantly decreased, enabling its widespread application in smart cars.

[0003] Point cloud registration is a key step in lidar positioning technology, used to estimate the transformation matrix between two point cloud frames. Initially, researchers conducted research on point cloud registration from an optimization perspective. Optimization-based methods iteratively perform correspondence search and transformation estimation to obtain the optimal transformation matrix. With the development of deep learning, researchers have proposed learning-based methods. First, correspondences are obtained through extracted features, and then the transformation matrix is ​​estimated using a neural network. Unlike optimization-based methods, learning-based methods do not require iterative operations. Whether based on optimization or learning, most point cloud registration methods assume that the positions of all objects in the environment are unchanged, that is, the static environment assumption. However, urban road environments contain many dynamic objects, such as pedestrians and moving vehicles. Therefore, dynamic objects can reduce the accuracy of point cloud registration.

[0004] Summary of the Invention

[0005] To address the problem of dynamic targets reducing the accuracy of point cloud registration, this paper proposes a LiDAR point cloud registration method for dynamic urban environments. This method converts three-dimensional point clouds into two-dimensional range images, improving the efficiency of point cloud processing. This method performs both point cloud segmentation and registration. By sharing a feature extraction module, point cloud registration can be performed based on reliable features of static reference targets, minimizing errors caused by dynamic targets.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a lidar point cloud registration method for urban dynamic environments, comprising the following steps:

[0007] S1: Dataset for building point cloud registration network;

[0008] S2: Constructing a point cloud registration network based on multi-task learning: The point cloud registration network based on multi-task learning includes two parts: a static reference target feature extraction module and a point cloud segmentation and registration module; wherein,

[0009] 1) Static reference target feature extraction module

[0010] Use spherical projection to convert the 3D point cloud into a 2D range image, and the point cloud coordinates (x, y, z) into image coordinates (u, v):

[0011] Where w and h represent the width and height of the distance image, r represents the distance of each point, f represents the vertical viewing angle of the laser radar, f up Represents the upward angle of f; the coordinates (x, y, z), distance r, and reflectivity i of each frame of the point cloud are stored in a distance image of size [w×h×5];

[0012] The input of the static reference target feature extraction module is the range image at time k-1 and k, and the output is the static reference target feature;

[0013] The two range images are decomposed into n image blocks, The size of the image block is Each image block is expanded into a J-dimensional vector, Two distance images are represented as a sequence

[0014] The Transformer encoder is used to extract static reference target features. The vector dimension processed by the Transformer encoder is K, and a learnable linear projection is used. will sequence Convert to Sequence A learnable registration vector Add to sequence In the sequence

[0015] The learnable position vector Add to sequence Get Transformer encoder input Q0=[q0,q1E,q2E,…,q n E]+P;

[0016] The Transformer encoder has L modules, each of which consists of a multi-head self-attention layer (MSA) and a multi-layer perceptron layer (MLP). Layer normalization (LN) is performed before the data enters each layer, and the output of each layer is fused with the input using residual connections. The feature calculation process of module l is: Q l ′=MSA(LN(Q l-1 ))+Q l-1 ,l=1,2,…,LQ l =MLP(LN(Q l ′))+Q l ′,l=1,2,…,L

[0017] Static reference target feature Q L Q0 is obtained through the Transformer encoder;

[0018] 2) Point cloud segmentation and registration module

[0019] Static reference target feature Q L It is a sequence of n+1 K-dimensional vectors, and the vector with index 0 is used to estimate the transformation matrix Z from time k-1 to time k k-1,k =[R k-1,k t k-1,k ], the vector with index 1 to 0.5n is used to obtain the static reference target segmentation result at time k-1, and the vector with index 0.5n+1 to n is used to obtain the static reference target segmentation result at time k. The segmentation at the two moments performs the same operation, that is, parameter sharing;

[0020] Transformation matrix Z k-1,k It is obtained by designing a neural network consisting of an input layer, two hidden layers, and an output layer, and the Tanh activation function is added to the neurons in the hidden layers;

[0021] The process of obtaining the static reference object segmentation result is divided into three steps: first, 0.5n K-dimensional vectors are converted into 0.5n J-dimensional vectors through linear projection, then the 0.5n J-dimensional vectors are transformed and sorted to obtain a feature map of size [w×h×5], and finally, the feature map is processed using a 1*1 convolution kernel and a Softmax function to obtain the static reference object segmentation result at time k-1 and k.

[0022] S3: training the designed point cloud registration network;

[0023] S4: Use the trained network to estimate the transformation matrix of the two frames of point cloud.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0025] 1. The present invention converts three-dimensional point clouds into two-dimensional distance images, thereby improving the efficiency of point cloud processing.

[0026] 2. The present invention performs two tasks, point cloud segmentation and registration. Due to the shared feature extraction module, point cloud registration can be performed based on the reliable features of static reference targets, reducing the error caused by dynamic targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a diagram of the overall design of the lidar point cloud registration method;

[0028] Figure 2 is a diagram of the LiDAR point cloud registration network;

[0029] Figure 3 is a diagram of a static reference target feature extraction module;

[0030] Figure 4 is a diagram of the point cloud segmentation and registration module. DETAILED DESCRIPTION

[0031] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0032] Example 1

[0033] The technical solutions provided by the present invention will be described in detail below with reference to specific examples. It should be understood that the following specific implementation methods are only for illustration and are not intended to limit the scope of the present invention.

[0034] The present invention proposes a lidar point cloud registration method for urban dynamic environments. First, a data set of a point cloud registration network is established, and then a point cloud registration network based on multi-task learning is constructed. Then, the designed point cloud registration network is trained, and finally, the transformation matrix of the two frames of point clouds is estimated using the trained network. This method converts a three-dimensional point cloud into a two-dimensional range image, improving the efficiency of point cloud processing. This method performs two tasks, point cloud segmentation and registration. Due to the shared feature extraction module, point cloud registration can be performed based on the reliable features of static reference targets, reducing the errors caused by dynamic targets. The overall design scheme is shown in Figure 1. The specific steps include:

[0035] (1) Establishing a dataset for point cloud registration network

[0036] The KITTI dataset is currently the world's largest dataset for evaluating artificial intelligence algorithms in autonomous driving scenarios (URL: https: / / www.cvlibs.net / datasets / kitti / index.php), containing real images and point cloud data collected in urban, rural, and highway scenarios.

[0037] The KITTI dataset provides 11 point cloud sequences with ground-truth transformation matrices and the semantic category of each point in these 11 point cloud sequences, for a total of 19 categories. In urban traffic environments, tree trunks and buildings are static reference objects, as their positions do not change. Therefore, the 19 categories in the KITTI dataset are reclassified into three: columns and trunks, buildings, and other.

[0038] Sequences 00-05 are used for training, sequences 06-07 are used for validation, and sequences 08-10 are used for testing.

[0039] (2) Constructing a point cloud registration network based on multi-task learning

[0040] The point cloud registration network based on multi-task learning consists of two parts: a static reference target feature extraction module and a point cloud segmentation and registration module, as shown in Figure 2.

[0041] 1) Static reference target feature extraction module

[0042] In order to improve the processing efficiency of point clouds, spherical projection is used to convert three-dimensional point clouds into two-dimensional distance images, and point cloud coordinates (x, y, z) are converted into image coordinates (u, v):

[0043] Where w and h represent the width and height of the distance image, r represents the distance of each point, f represents the vertical viewing angle of the laser radar, f up represents the upward angle of f. The coordinates (x, y, z), distance r, and reflectivity i of each frame of the point cloud are stored in a range image of size [w × h × 5]. The input of the static reference target feature extraction module is the range image at time k-1 and k, and the output is the static reference target feature, as shown in Figure 3.

[0044] The two range images are decomposed into n image blocks, The size of the image block is Each image block is expanded into a J-dimensional vector, Therefore, the two distance images are represented as the sequence

[0045] The Transformer encoder is used to extract the static reference target features. The vector dimension processed by the Transformer encoder is K, so a learnable linear projection is used. will sequence Convert to Sequence A learnable registration vector Add to sequence In the sequence

[0046] After the image is decomposed, the image blocks lose their position information and the learnable position vector Add to sequence Get Transformer encoder input Q0=[q0,q1E,q2E,…,q n E]+P.

[0047] The Transformer encoder has L modules, each of which consists of a multi-head self-attention layer (MSA) and a multi-layer perceptron layer (MLP). Layer normalization (LN) is performed before the data enters each layer, and the output of each layer is fused with the input using residual connections. The feature calculation process of module l is: Q l ′=MSA(LN(Q l-1 ))+Q l-1 ,l=1,2,…,LQ l =MLP(LN(Q l ′))+Q l ′,l=1,2,…,L

[0048] Static reference target feature Q L It is Q0 obtained through the Transformer encoder.

[0049] 2) Point cloud segmentation and registration module

[0050] The point cloud segmentation and registration module is designed based on multi-task learning, as shown in Figure 4. Static reference target feature Q L It is a sequence of n+1 K-dimensional vectors used for point cloud segmentation and registration.

[0051] The vector with index 0 is used to estimate the transformation matrix Z from time k-1 to time k k-1,k =[R k-1,k t k-1,k ], the vector with index 1 to 0.5n is used to obtain the static reference target segmentation result at time k-1, and the vector with index 0.5n+1 to n is used to obtain the static reference target segmentation result at time k. The segmentation at the two moments performs the same operation, that is, parameter sharing.

[0052] Transformation matrix Z k-1,k It can be obtained by designing a neural network, which consists of an input layer, two hidden layers and an output layer. The Tanh activation function is added to the neurons in the hidden layers to improve the generalization performance of the neural network.

[0053] The process of obtaining the static reference object segmentation results is divided into three steps: first, 0.5n K-dimensional vectors are converted into 0.5n J-dimensional vectors through linear projection, and then the 0.5n J-dimensional vectors are transformed and sorted to obtain a feature map of size [w×h×5]. Finally, the feature map is processed using a 1*1 convolution kernel and a Softmax function to obtain the static reference object segmentation results at time k-1 and k.

[0054] (3) Training the designed point cloud registration network

[0055] The designed point cloud registration network was trained using the KITTI dataset sequences 00-05. The network parameters were iteratively optimized using gradient descent. The training process consisted of two phases: forward propagation and backpropagation. The network was validated using the KITTI dataset sequences 06-07 to select the optimal network model.

[0056] (4) Use the trained network to estimate the transformation matrix of the two frame point clouds

[0057] The KITTI dataset sequence 08-10 is used to test the performance of the point cloud registration network. In order to fully illustrate the advantages of the method of the present invention, representative point cloud registration algorithms are selected for comparison.

[0058] ICP (PJBesl and NDMcKay, "A method for registration of 3-D shapes," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 2, pp. 239-256, Feb. 1992.)

[0059] RANSAC (MAFischler and RCBolles, "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography," Communications of the ACM, vol. 24, no. 6, pp. 381-395, Jun. 1981.)

[0060] DGR (C.Choy, W.Dong and V.Koltun, "Deep Global Registration," in 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp.2511-2520.)

[0061] HRegNet (F.Lu et al., "HRegNet: A Hierarchical Network for Large-scale Outdoor LiDAR Point Cloud Registration," in 2021 IEEE / CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021, pp.15994-16003.)

[0062] The comparison results are shown in the table below. It can be seen that the method of the present invention greatly improves the accuracy of point cloud registration in dynamic environments while ensuring real-time performance.

[0063] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A laser radar point cloud registration method for urban dynamic environments, characterized in that: The steps include: S1: Dataset for building point cloud registration network; S2: Constructing a point cloud registration network based on multi-task learning: The point cloud registration network based on multi-task learning includes two parts: a static reference target feature extraction module and a point cloud segmentation and registration module; wherein, 1) Static reference target feature extraction module The 3D point cloud is converted into a 2D distance image using spherical projection, and the point cloud coordinates (x, y, z) are converted into image coordinates (u, v): Where w and h represent the width and height of the distance image, r represents the distance of each point, f represents the vertical viewing angle of the laser radar, f up Indicates the upward angle of f; the coordinates (x, y, z), distance r, and reflectivity i of each frame of the point cloud are stored in a distance image of size [w×h×5]; The input of the static reference target feature extraction module is the range image at time k-1 and k, and the output is the static reference target feature; The two distance images are decomposed into n image blocks. The size of the image block is Each image block is expanded into a J-dimensional vector, Two distance images are represented as a sequence The Transformer encoder is used to extract static reference target features. The vector dimension processed by the Transformer encoder is K, and a learnable linear projection is used. General rank Convert to sequence A learnable registration vector Add to sequence In the sequence The learnable position vector Add to sequence Get the Transformer encoder Input Q0 = [q0,q1E,q2E,…,q n E]+P; The Transformer encoder has L modules, each of which consists of a multi-head self-attention layer (MSA) and a multi-layer perceptron layer (MLP). Layer normalization (LN) is performed before the data enters each layer, and the output of each layer is fused with the input using residual connections. The feature calculation process of module l is: Q′ l =MSA(LN(Q l-1 ))+Q l-1 ,l=1,2,…,L Q l =MLP(LN(Q′ l ))+Q′ l ,l=1,2,…,L Static reference target feature Q L It is obtained by Q0 through the Transformer encoder; 2) Point cloud segmentation and registration module Static reference target feature Q L It is a sequence of n+1 K-dimensional vectors. The vector with index 0 is used to estimate the transformation matrix Z from time k-1 to time k. k-1,k =[R k-1,k t k-1,k ], the vector with index 1 to 0.5n is used to obtain the static reference target segmentation result at time k-1, and the vector with index 0.5n+1 to n is used to obtain the static reference target segmentation result at time k. The segmentation at the two moments performs the same operation, that is, parameter sharing; Transformation Matrix Z k-1,k It is obtained by designing a neural network consisting of an input layer, two hidden layers, and an output layer, and the Tanh activation function is added to the neurons in the hidden layers; The process of obtaining the static reference object segmentation result is divided into three steps: first, 0.5n K-dimensional vectors are converted into 0.5n J-dimensional vectors through linear projection, and then the 0.5n J-dimensional vectors are transformed and sorted to obtain a feature map of size [w×h×5]. Finally, the feature map is processed using a 1*1 convolution kernel and a Softmax function to obtain the static reference object segmentation results at time k-1 and k. S3: training the designed point cloud registration network; S4: Use the trained network to estimate the transformation matrix of the two frames of point cloud.

Images