A point cloud plane segmentation method based on multi-frame data fusion

Abstract

The present invention is a point cloud plane segmentation method based on multi-frame data fusion, which can be used in 16-line laser radar. In the scenario where the laser radar can only provide sparse point clouds, a segmentation result with a high segmentation accuracy can be obtained, and it can be applied to real-time segmentation. This method is a laser radar point cloud plane segmentation algorithm. In view of the problem that the three-dimensional data acquired by the 16-line laser radar is relatively sparse, we use a point cloud registration algorithm to fuse multi-frame point clouds together into a dense point cloud, and then use the dense point cloud to perform plane segmentation and establish a priori map. For the current frame, we align the current frame with the priori map, convert the current frame point cloud to the priori map, match the plane in the priori map and the neighboring relationship of the current frame point cloud data, quickly determine the plane points, and realize the rapid plane segmentation of the point cloud data.

Description

Technical Field

[0001] The present invention relates to the field of point cloud plane segmentation, and in particular to a point cloud plane segmentation method based on multi-frame data fusion. Background Art

[0002] With the emergence of low-cost depth cameras and 3D sensors on the market, sensors that can directly obtain three-dimensional point cloud data have become a popular choice for various robotics and computer vision applications. Among them, the processing algorithm of the three-dimensional data collected by the sensor is particularly important. As a key link in applications such as autonomous driving and robotics, the processing and analysis of three-dimensional scanning data has received increasing attention in the field of computer vision and deep learning in recent years.

[0003] Detecting planes in unordered point clouds is one of the key technologies in several emerging fields, such as self-driving cars and robotic navigation. It is also essential for more traditional applications, such as reverse engineering, camera calibration, object recognition, 3D reconstruction, point cloud denoising, augmented reality, and point cloud segmentation. Since point clouds are inherently noisy and cannot provide the semantics of the scene, using raw 3D point cloud data for concise semantic modeling has become a hot issue in 3D point cloud data processing. Among them, plane is one of the most important semantics, because man-made buildings are mainly composed of planes. Datasets containing millions of samples are now common, so detecting planar regions in point clouds requires both robustness to noise and scalability in performance. For applications such as reverse engineering, accurate reconstruction is also a key aspect. Unfortunately, most available techniques for detecting planar structures in point clouds are either sensitive to noise or computationally expensive. In addition, most previous techniques use parameters that need to be tuned for different datasets, which is undesirable and time-consuming.

[0004] On the other hand, point cloud plane segmentation is essentially in its infancy and remains an open research problem. It is mainly driven by LiDAR point cloud processing for applications in self-driving cars, autonomous robot navigation, and environmental mapping, all of which typically require real-time processing. Compared to traditional 3D data in computer vision, LiDAR point clouds are irregularly sampled in 3D space. These differences in the nature of 3D data, the need for real-time processing, and the level of accuracy required for safety assurance (such as self-driving cars) pose new challenges to the processing of 3D point cloud data. These challenges motivate us to look for a more suitable architecture that takes into account the unique characteristics of LiDAR data, effectively solves the plane segmentation problem, and improves the accuracy and speed of plane segmentation as much as possible.

[0005] Plane detection technology in point clouds has been widely studied, and most of the technologies can be divided into three categories according to their methods: plane segmentation based on Hough transform, plane segmentation based on random sampling consensus (RANSAC) algorithm, and plane segmentation based on region growing.

[0006] Hough transform is often used for line detection in images. Plane segmentation based on Hough transform converts the original 3D point cloud data collected by the sensor into some feature space. The feature space is subdivided into small cubes, and the number of 3D point clouds mapped to the feature space in each cube is recorded. The plane model represented by the cube with the most point clouds is used as the point cloud plane we want to estimate. However, Hough transform is very sensitive to noise in point clouds, and its segmentation speed is also slower than other methods.

[0007] RANSAC is another popular shape detection technique for multidimensional data. It is widely used in this field due to its simplicity and robustness. The RANSAC algorithm first randomly selects the minimum number of points required to calculate the plane model, estimates the plane model with these point sets, and calculates how many points belong to the plane model, and uses these points as inliers. Repeat this operation, and finally select the plane model with the most inliers as the result of plane segmentation. The RANSAC algorithm is highly robust to noise in the point cloud, but it needs to estimate and analyze a large number of random plane models to obtain good results, so its processing speed is very slow and is not suitable for point clouds with millions of samples.

[0008] Region growing algorithms are very popular in 2D and 2.5D image applications. However, their application in unorganized point clouds is still not fully reflected. This is mainly due to the nature of such datasets, which contain occlusions, noise and lack of neighborhood information. Existing region growing algorithms for plane segmentation use features such as normals and curvatures of LiDAR point clouds to calculate the curvature of all points in the point cloud, select the point with the smallest curvature as the plane point, and then calculate the angle difference between the surrounding neighboring points and the normal of the point. If it is less than a threshold, the point is attributed to the current plane. In this way, the plane point set is continuously expanded to obtain the final estimated plane model. However, when the point cloud is sparse, the effect of this algorithm is not stable enough.

[0009] The above three methods have poor effects on plane detection in sparse point clouds. When the number of LiDAR point clouds is small or the LiDAR is far away from the plane, it is easy to have problems of misdetection and missed detection. However, the cost of existing high-line-count LiDARs is extremely high, and most civilian fields use cheap 16-line LiDARs. Therefore, the research on highly robust plane segmentation methods for sparse LiDAR point clouds is of great significance to the application of LiDAR.

[0010] Many related point cloud segmentation methods have been proposed at home and abroad. Hough transform for point cloud segmentation requires a lot of computer resources and its data processing speed is very slow. The results of RANSAC algorithm fitting itself have certain uncertainties, and its accuracy is not high when the amount of data is small. The segmentation method based on region growing is prone to obtain wrong segmentation results. In the case of improper parameter settings, over-segmentation and under-segmentation are prone to occur. The above algorithms have poor segmentation effects in sparse point clouds and are mainly suitable for dense point cloud segmentation. Summary of the invention

[0011] In order to solve the above technical problems, the present invention proposes a point cloud plane segmentation method based on multi-frame data fusion. Considering that the existing algorithm is difficult to detect the plane in the sparse point cloud, a data fusion method based on Normal Distribution Transformation (NDT) is proposed to transform the sparse point cloud into a dense point cloud, which is convenient for plane segmentation. Considering the problem of slow calculation speed of the existing algorithm, a method is proposed to use multi-frame data to fuse the point cloud to establish a priori map, and then use the plane in the priori map to align with the current point cloud for fast point cloud segmentation. It can perform more accurate plane segmentation on sparse point clouds and improve the point cloud processing speed.

[0012] A point cloud plane segmentation method based on multi-frame data fusion, the specific steps are as follows, and the characteristics are:

[0013] S1.16-line laser radar obtains 3D point cloud data;

[0014] S2. Initialize the current map, read the data of the first 10 frames of the laser radar, use the point cloud registration algorithm to find the pose between each frame, convert the point clouds of other frames to the coordinate system of the first frame, and obtain a dense map; Step S2 includes the following steps:

[0015] S21. Use NDT algorithm for point cloud registration;

[0016] S22. Point cloud fusion;

[0017] The step S21 includes the following steps: dividing the point cloud into cubic grids, calculating the probability density function of each grid, and estimating the pose between two frames of point clouds based on the probability density function information.

[0018] The step S22 includes the following steps: converting the point cloud of the last 9 frames into the map of the first frame and saving it;

[0019] S3. perform point cloud plane segmentation on the dense map, and save the obtained point cloud plane information in the dense map as a priori map;

[0020] Step S3 includes the following steps:

[0021] S31. Calculate the point cloud curvature;

[0022] The step S31 includes the following steps: using the PCA algorithm to calculate the surface curvature of each point;

[0023] S32. Point cloud plane segmentation;

[0024] The step S22 includes the following steps: performing point cloud plane segmentation on the dense map using a growing region growing algorithm according to the curvature, and saving the obtained point cloud plane information in the dense map as a priori map;

[0025] S4. Pair the current frame point cloud with the prior map to estimate the plane segmentation result of the current frame;

[0026] Step S4 includes the following steps:

[0027] S41. Perform NDT registration of the current frame with the prior map to obtain the pose of the current frame and the prior map;

[0028] S42. Point cloud pairing estimation plane segmentation result;

[0029] The step S42 includes the following steps: converting the point cloud of the current frame into the prior map, and estimating the plane segmentation result of the current frame according to the distance between the point cloud of the current frame and the plane model in the prior map.

[0030] As a further improvement of the present invention, the NDT algorithm is specifically as follows:

[0031] First, the two frames of data used for registration are gridded, and the three-dimensional map is divided into small cubes. For each cube, the probability density function is calculated based on the points inside it:

[0032]

[0033] where y k＝1，...，m represents all three-dimensional points in a grid, and x represents the coordinates of any position in the grid;

[0034] The other parameters in the function are calculated as follows:

[0035]

[0036]

[0037] In this way, the point cloud data space is transformed into small cubes. The probability density function f(x) of each cube can be described as the approximate result of the local surface of the point cloud in the area, which contains information such as the position and direction of the surface. The goal of NDT transformation is to find the pose of the currently scanned point cloud so as to maximize the possibility that the current point cloud is located on the reference point cloud surface. Assuming that the point cloud of the current scan is X={x 1 , ..., x n}, the motion transformation parameter between the reference point cloud is T, and the pose change function g(T, x k ) represents the transformation from the current point to the reference point cloud map, and the maximum likelihood estimate of the motion transformation is obtained:

[0038]

[0039] Take the logarithm of the maximum likelihood estimate to obtain Log L, and calculate the pose change parameter T when the derivative of Log L is equal to zero as the registration result between the current frame and the reference frame.

[0040] As a further improvement of the present invention, the specific steps of point cloud plane segmentation are as follows:

[0041] When the laser radar starts working, it collects 10 frames of data. i＝1，...，10 Initialize and use NDT registration to get the motion transformation T between the last 9 frames and the first frame 1，j＝2，...，10 , will use T 1，j＝2，...，10 Convert the last 9 frames of data into the first frame of data to form a dense point cloud X.

[0042] Assume that there are m neighboring points q near point q i＝1，...，k The normal vector of point q is obtained by principal component analysis (PCA), and the following covariance matrix is ​​calculated:

[0043]

[0044] Where q 0 is the centroid of all neighboring points of q. Perform eigenvalue decomposition on the covariance matrix M, and the eigenvector corresponding to its minimum eigenvalue is the required normal vector. Perform eigenvalue decomposition on M to obtain three eigenvectors, which are sorted into λ 0 ≤λ 1 ≤λ 2 , then the surface curvature of point q can be expressed as:

[0045]

[0046] Get the normal vector and curvature of each point, perform plane segmentation of the point cloud using the growing region algorithm, sort the dense point cloud X from small to large in curvature, and place the point x with the smallest curvature iAs the initial point of the plane, search for x i For all neighboring points around, calculate the normal angle difference between each neighboring point and the current initial point. If the difference is less than the set threshold, perform curvature detection on the neighboring point. If the curvature of the point is less than the set threshold, the point is considered to be a plane point, and the point is marked with the serial number of the current plane. Then search for the neighboring points of the newly added plane point, and perform normal difference detection and curvature detection in the same way until no qualified points are detected. All the previously qualified points are used as the point set of the current plane. Then re-sort the points without marked plane serial numbers, repeat the above steps, and obtain all plane segmentation results. The dense point cloud and plane sequence markers obtained in the previous steps are used as a priori maps to achieve subsequent fast plane segmentation.

[0047] Beneficial effects:

[0048] This method can use a low-line-count LiDAR to achieve fast ground plane segmentation in unknown environments. To address the problem that sparse point clouds are difficult to identify planes, we consider fusing multiple frames of data to obtain a dense point cloud, and performing plane segmentation on the dense point cloud to ensure the accuracy of the plane segmentation results. To address the problem that the existing plane segmentation algorithm takes a long time to calculate, we consider first fusing 10 frames of data to obtain a dense map, using the growing region algorithm to segment the dense map, using the segmentation result as a priori map, aligning the current frame with the priori map, and then matching the point cloud of the current frame with the plane in the priori map to obtain the segmentation result of the current frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of the plane segmentation method based on multi-frame fusion. DETAILED DESCRIPTION

[0050] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0051] like Figure 1 As shown in this method, the mobile carrier collects 3D point cloud data of the surrounding environment through a 16-line laser radar during movement, and the processor reads the point cloud data collected by the laser radar. The first 10 frames of point cloud data collected are point cloud registered, and these 10 frames of data are point cloud fused to obtain a dense point cloud map. The dense point cloud is segmented using the growing region algorithm, and the segmentation results and the dense map are used as the prior map. After that, the collected data are all laser radar point cloud registered with the prior map to obtain the posture change of the current frame and the prior map, and then the current frame point cloud is converted to the prior map, and the plane segmentation result of the current frame is estimated according to the position of the current frame point cloud in the prior map and the distance relationship of the plane model in the prior map.

[0052] (a) LiDAR point cloud registration;

[0053] The laser radar uses Velodyne V:P-16 laser radar.

[0054] Point cloud registration based on normal distribution transform (NDT). The NDT algorithm first grids the two frames of data used for registration, divides the 3D map into small cubes, and calculates the probability density function of each cube based on the points inside it:

[0055]

[0056] where y k＝1，...，m represents all three-dimensional points in a grid, and x represents the coordinates of any position in the grid.

[0057] The other parameters in the function are calculated as follows:

[0058]

[0059]

[0060] In this way, the point cloud data space is transformed into small cubes. The probability density function f(x) of each cube can be described as an approximation of the local surface of the point cloud in the area, which contains information such as the position and direction of the surface. The goal of NDT transformation is to find the pose of the currently scanned point cloud so as to maximize the possibility that the current point cloud is located on the reference point cloud surface. Assume that the point cloud of the current scan is X = {x 1 , ..., x n}, the motion transformation parameter between the reference point cloud is T, and the pose change function g(T, x k ) represents the transformation from the current point to the reference point cloud map. The maximum likelihood estimate of the motion transformation can be obtained:

[0061]

[0062] Take the logarithm of the maximum likelihood estimate to obtain Log L, and calculate the pose change parameter T when the derivative of Log L is equal to zero as the registration result between the current frame and the reference frame.

[0063] (ii) Dense point cloud plane segmentation based on multi-frame data fusion;

[0064] When the laser radar starts working, it collects 10 frames of data. i＝1，...，10 Initialize. Use NDT registration to get the motion transformation T between the last 9 frames and the first frame 1，j＝2，...，10 , will use T 1，j＝2，...，10 Convert the last 9 frames of data into the first frame of data to form a dense point cloud X.

[0065] Assume that there are m neighboring points q near point q i＝1，...，k The normal vector of point q is obtained by principal component analysis (PCA), and the following covariance matrix is ​​calculated:

[0066]

[0067] Where q 0 is the centroid of all neighboring points of q. Perform eigenvalue decomposition on the covariance matrix M, and the eigenvector corresponding to its minimum eigenvalue is the required normal vector. Perform eigenvalue decomposition on M to obtain three eigenvectors, which are sorted into λ 0 ≤λ 1 ≤λ 2 , then the surface curvature of point q can be expressed as:

[0068]

[0069] Get the normal vector and curvature of each point, and perform plane segmentation of the point cloud using the growing region algorithm. Sort the dense point cloud X from small to large in curvature, and put the point x with the smallest curvature i As the initial point of the plane. Search x i For all neighboring points around, calculate the normal angle difference between each neighboring point and the current initial point. If the difference is less than the set threshold, perform curvature detection on the neighboring point. If the curvature of the point is less than the set threshold, the point is considered to be a plane point and the point is marked with the serial number of the current plane. Then search for the neighboring points of the newly added plane point, and perform normal difference detection and curvature detection in the same way until no qualified points are detected. All the previously qualified points are used as the point set of the current plane. Then re-sort the points without marked plane serial numbers, repeat the above steps, and obtain all plane segmentation results. The dense point cloud and plane sequence markers obtained in the previous steps are used as a priori maps to achieve subsequent fast plane segmentation.

[0070] (iii) Fast point cloud plane segmentation based on prior maps;

[0071] After obtaining the prior map, the point cloud scanned by the current laser radar is aligned with the prior map through the NDT algorithm, and the point cloud of the current frame is converted from the current laser radar coordinate system to the prior map coordinate system. For each point in the current frame, the nearest neighbor search algorithm of the KD tree is used to find the point closest to it in the prior map for matching. If the matched point is a plane point, the point in the current frame is marked as a plane point. Repeat the above operation, traverse all the points in the current frame, and you can get the plane segmentation result of the current frame, completing the fast plane segmentation of the point cloud.

[0072] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A point cloud plane segmentation method based on multi-frame data fusion, the specific steps are as follows, characterized in that: S1.16-line laser radar obtains 3D point cloud data; S2. Initialize the current map, read the data of the first 10 frames of the laser radar, use the point cloud registration algorithm to find the pose between each frame, convert the point clouds of other frames to the coordinate system of the first frame, and obtain a dense map; Step S2 includes the following steps: S21. Use NDT algorithm for point cloud registration; S22. Point cloud fusion; The step S21 includes the following steps: dividing the point cloud into cubic grids, calculating the probability density function of each grid, and estimating the position and posture between two frames of point clouds according to the probability density function information; The step S22 includes the following steps: converting the point cloud of the last 9 frames into the map of the first frame and saving it; S3. perform point cloud plane segmentation on the dense map, and save the obtained point cloud plane information in the dense map as a priori map; Step S3 includes the following steps: S31. Calculate the point cloud curvature; The step S31 includes the following steps: using the PCA algorithm to calculate the surface curvature of each point; S32. Point cloud plane segmentation; The step S22 includes the following steps: performing point cloud plane segmentation on the dense map using a growing region growing algorithm according to the curvature, and saving the obtained point cloud plane information in the dense map as a priori map; S4. Pair the current frame point cloud with the prior map to estimate the plane segmentation result of the current frame; Step S4 includes the following steps: S41. Perform NDT registration of the current frame with the prior map to obtain the pose of the current frame and the prior map; S42. Point cloud pairing estimation plane segmentation result; The step S42 includes the following steps: converting the point cloud of the current frame into the prior map, and estimating the plane segmentation result of the current frame according to the distance between the point cloud of the current frame and the plane model in the prior map.

2. The point cloud plane segmentation method based on multi-frame data fusion according to claim 1, characterized in that: The NDT algorithm is as follows: First, the two frames of data used for registration are gridded, and the three-dimensional map is divided into small cubes. For each cube, the probability density function is calculated based on the points inside it: where y k＝1，...，m represents all three-dimensional points in a grid, and x represents the coordinates of any position in the grid; The other parameters in the function are calculated as follows: In this way, the point cloud data space is transformed into small cubes. The probability density function f(x) of each cube can be described as the approximate result of the local surface of the point cloud in the area, which contains the position and direction information of this surface. The goal of NDT transformation is to find the pose of the currently scanned point cloud so as to maximize the possibility that the current point cloud is located on the reference point cloud surface. Assume that the point cloud of the current scan is X = {x1, ..., x n }, the motion transformation parameter between the reference point cloud is T, and the pose change function g(T, x k ) represents the transformation from the current point to the reference point cloud map, and the maximum likelihood estimate of the motion transformation is obtained: Take the logarithm of the maximum likelihood estimate to obtain Log L, and calculate the pose change parameter T when the derivative of Log L is equal to zero as the registration result between the current frame and the reference frame.

3. The point cloud plane segmentation method based on multi-frame data fusion according to claim 1, characterized in that: The specific steps of point cloud plane segmentation are as follows: When the laser radar starts working, it collects 10 frames of data. i＝1，...，10 Initialize and use NDT registration to get the motion transformation T between the last 9 frames and the first frame 1，j＝2，...，10 , will use T 1，j＝2，...，10 Convert the last 9 frames of data into the first frame of data to form a dense point cloud X; Assume that there are m neighboring points q near point q i＝1，...，k , the normal vector of point q is obtained by principal component analysis PCA, and the following covariance matrix is ​​calculated: Where q0 is the centroid of all neighboring points of q. The covariance matrix M is decomposed by eigenvalue, and the eigenvector of its minimum eigenvalue is the required normal vector. M is decomposed by eigenvalue to obtain three eigenvectors, which are sorted as λ0≤λ1≤λ2. The surface curvature of point q can be expressed as: Get the normal vector and curvature of each point, perform plane segmentation of the point cloud using the growing region algorithm, sort the dense point cloud X from small to large in curvature, and place the point x with the smallest curvature i As the initial point of the plane, search for x i For all neighboring points around, calculate the normal angle difference between each neighboring point and the current initial point. If the difference is less than the set threshold, perform curvature detection on the neighboring point. If the curvature of the point is less than the set threshold, the point is considered to be a plane point, and the point is marked with the serial number of the current plane. Then search for the neighboring points of the newly added plane point, and perform normal difference detection and curvature detection in the same way until no qualified points are detected. All the previously qualified points are used as the point set of the current plane, and then the points without marked plane numbers are re-sorted. Repeat the above steps to obtain all plane segmentation results. The dense point cloud and plane sequence markers obtained in the previous steps are used as a priori maps to achieve subsequent fast plane segmentation.

Images