A cross-source point cloud registration method for street scene buildings based on graph matching

Abstract

The present invention discloses a method for cross-source point cloud registration of streetscape buildings based on graph matching. First, the two obtained cross-source point cloud sets are normalized to the same scale, and the geometric characteristics of the three-dimensional point cloud are used to segment the streetscape building point cloud into supervoxels, and the feature descriptors of these supervoxels are extracted. The extracted supervoxels and their feature descriptors are used as nodes of the graph, and the adjacent relationships of the supervoxels are used as edges to construct a graph, and the three-dimensional point cloud registration is converted into graph structure matching. The streetscape building point cloud registration is performed according to the corresponding relationship obtained from the graph matching result, and finally a rigid transformation registration result between the two cross-source point cloud sets is obtained. The present invention adopts the extraction of microstructure and macrostructure for the cross-source point cloud, converts the three-dimensional point cloud registration into graph matching, and optimizes the graph matching method, which greatly improves the registration accuracy and efficiency of the cross-source point cloud.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional point clouds, and in particular relates to a cross-source point cloud registration method for streetscape buildings. Background Art

[0002] Point cloud registration involves taking two sets of point clouds with overlapping regions but different poses and outputting a rigid transformation (translation and rotation) from one set to the other, so that the pose of the overlapping region of one set of point clouds is the same as that of the other set after the rigid transformation. It is a fundamental task in graphics, vision, and robotics.

[0003] Point cloud registration is of great practical significance. Due to the limitations of the measurement method and the shape of the object set being measured, 3D scanning measurement equipment can only scan a limited range of point cloud data of the object being measured at a time. This requires multiple scans at multiple perspectives. However, the point cloud data obtained at each perspective has an independent coordinate system and cannot be directly spliced. Therefore, it is necessary to perform coordinate conversion on the point cloud obtained at each perspective and unify it into a global coordinate system. The specific process is to realize the conversion between the point set and the point set coordinate system through the relationship between each point in a point set (target) and the corresponding point in another point set (original point set) to achieve registration.

[0004] Point cloud registration is widely used in fields such as 3D reconstruction and SLAM, and is a fundamental task for tasks such as autonomous driving and dynamic tracking. Therefore, solving the point cloud registration problem is crucial.

[0005] Cross-source point cloud registration refers to the registration between point clouds from two different sensors. This task is also of practical significance in actual application scenarios. Currently, tasks such as 3D reconstruction and autonomous driving mostly use more than one sensor, but instead collect data from a mixture of multiple sensors.

[0006] In practical applications, streetscape buildings are important targets for point cloud construction. These point clouds can be used in areas such as autonomous driving positioning and detection. Currently, numerous methods exist for acquiring streetscape building point clouds, including SFM (Structure from Motion) and LiDAR scanning. Each point cloud acquisition method has its own advantages and disadvantages. SFM analyzes multiple consecutive frames of imagery and determines the spatial and geometric relationships of streetscape building point clouds through camera movement. This method requires only RGB images and can construct a relatively complete streetscape building image, with color-coded point clouds. However, since the point clouds are calculated, errors in scale and geometry may occur. LiDAR scanning can capture real data, but the scanning process can create blind spots that are difficult to scan. To combine the advantages of different streetscape building point cloud acquisition methods, the point clouds acquired by the two methods need to be registered.

[0007] Compared with the registration of homologous point clouds, the registration of cross-source point clouds faces many challenges and problems. For example, the density and scale of point clouds from different sources are different. Due to different sensor types, the noise and anomalies between point clouds will be different. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this paper proposes a graph-matching-based method for cross-source point cloud registration of streetscape buildings. This method can utilize streetscape point clouds acquired using different methods, combining the advantages of each method and improving the efficiency and accuracy of cross-source point cloud registration.

[0009] This method registers streetscape building point clouds generated by the SFM method with those acquired by LiDAR scanning. The macrostructure and microstructure extracted from the two point clouds are converted into graphs, and the three-dimensional point cloud matching is converted into matching between graph structures.

[0010] Cross-source point cloud matching is a difficult challenge for computer vision, but the human eye and brain can easily distinguish the same regions and corresponding relationships between point clouds. This is because despite differences in scale, density, and noise across point clouds, the structure of point clouds for the same region and object remains the same. Based on this, this paper proposes a method for extracting and aligning macroscopic and microscopic structures.

[0011] The macrostructure refers to the overall structure that maintains a similar geometric layout across point clouds, and the microstructure is the component of the macrostructure.

[0012] A cross-source point cloud registration method for street scene buildings based on graph matching, specifically comprising the following steps:

[0013] Step 1: Obtain a video of a street view building and use the SFM method to process the continuous images of the street view building to obtain a point cloud set A of the street view building. Use a LiDAR to scan the same street view building and obtain a LiDAR point cloud set B.

[0014] Step 2: Normalize the two obtained cross-source point cloud sets A and B to the same scale.

[0015] Step 3: Using the geometric properties of the 3D point cloud, the street view building point cloud is segmented into supervoxels and feature descriptors of these supervoxels are extracted. Each segmented supervoxel and its corresponding feature descriptor are microscopic structures.

[0016] Step 4: Use the supervoxels extracted in step 3 and their feature descriptors as nodes of the graph, and the neighboring relationships of the supervoxels as edges to construct a graph. The resulting graph is the macro structure of the two cross-source point clouds.

[0017] Step 5: Optimize the existing graph matching algorithm for point cloud registration, transforming 3D point cloud registration into graph structure matching. Specifically, based on the requirements of cross-source point cloud matching, weights are set for point matching and edge matching in the graph matching process, and the graph matching weights for cross-source point cloud registration are determined through optimization.

[0018] Step 6: Perform street view building point cloud registration based on the correspondence relationship obtained from the graph matching result obtained in step 5, and finally obtain the rigid transformation registration result between the two cross-source point cloud sets.

[0019] The advantages and beneficial effects of the method of the present invention are:

[0020] 1. This invention proposes the idea of ​​using cross-source point clouds to collect street view building point clouds, combining the advantages of different point cloud acquisition methods to obtain excellent street view building point clouds.

[0021] 2. The microstructure and macrostructure of cross-source point clouds are extracted to transform the 3D point cloud registration into graph matching, and the graph matching method is optimized, which greatly improves the registration accuracy and efficiency of cross-source point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The radar scan point cloud of street view building A;

[0023] Figure 2 The radar scan point cloud of street view building B;

[0024] Figure 3 Point cloud constructed by the SFM method for street scene building A;

[0025] Figure 4 Point cloud constructed by the SFM method for street scene building B;

[0026] Figure 5 、 6 is the SFM method point set of street scene building C and its supervoxel segmentation result;

[0027] Figure 7 、 8 The radar scanning method point set of the street scene building C and its supervoxel segmentation result. DETAILED DESCRIPTION

[0028] The present invention is further described in detail below with reference to examples.

[0029] The present invention proposes a cross-source point cloud registration method. The two parts of the point cloud to be registered are obtained by different point cloud sensors, and a robust registration method is performed on the point clouds obtained by different sensors. Figure 1-8 As shown, the method of the present invention includes the following steps.

[0030] Step 1: Use a camera to capture dynamic video of the streetscape building. Use the SFM method to process the acquired continuous images of the streetscape building to obtain a point cloud set A of the streetscape building. Use a lidar to scan the same streetscape building to obtain another point cloud set B of the streetscape building.

[0031] Step 2: Scale normalize the two cross-source point clouds. These two point clouds originate from different sensors and therefore have different scales. Based on the sensor parameters of the point clouds' sources, the Growing Least Squares descriptor (GLS) method is used to convert the scales of the two cross-source point clouds to the same. The LiDAR scanning method generates real-world point clouds, so they have the same scale as actual streetscape buildings. Point cloud A is normalized to the same scale as point cloud B.

[0032] A'=F(A)

[0033] B'=B

[0034] Where A', B' are normalized point cloud data, and function F is the scale mapping from point cloud set A to B.

[0035] Step 3: Using the geometric properties of the 3D point cloud, split the cross-source point cloud set A', B' into supervoxels. Specifically, use the geometric properties of points in 3D space to cluster similar points in the point set, and then form a surface (mesh) from three similar points. Each surface is a supervoxel. A is the set of supervoxels obtained by segmenting the point cloud set A', V B is the set of supervoxels obtained by segmenting the point cloud set B'. In the process of segmenting supervoxels, the geometric characteristics are used to cluster the points, so the density, noise and other issues existing in different source point clouds will not affect the segmentation. At the same time, the segmented supervoxel set V A ,V B Based on the description of the surface distance, angle and area distribution of the partial point cloud, the shape function is used to obtain the ESF descriptor (Ensemble of shape functions shape descriptor) V EA ,V EB .

[0036] Step 4: Use the supervoxel set and the corresponding EFS descriptor as the nodes of the graph and the adjacent relationship as the edge of the graph to construct the graph C of the two cross-source point cloud sets. A and C B Define the description of the graph C = {P, Q, H}, where P is the set of points in the graph, that is, the set of supervoxels and shape descriptors, Q is the set of edges, and H is the adjacency matrix between points.

[0037] Step 5: Match the constructed graph. This invention improves the existing graph matching algorithm FGM (Factorized graph matching). Specifically, graph matching requires A and C B Find the corresponding relationship between them so that the global consistency J is maximized.

[0038]

[0039] Where x∈{0,1} represents the corresponding relationship between the nodes represented by its subscript, such as C A The i1th node and graph C B Corresponding to the i2th node in , then is the matrix k p The element at row i1 and column i2 in is the matrix k q The element in row c1 and column c2 of the matrix k p Figure C A and C B The similarity matrix of the point set P, matrix k q Figure C A and C B The similarity matrix of the edge set Q.

[0040] Since Figure C A and C B The graph is converted from a point cloud set. Graph matching is for the final point cloud registration, so the graph matching algorithm is optimized. In cross-source point cloud registration, the global structural information of the point cloud is more important than the local information of the point cloud. Excessive emphasis on local information of the point cloud will lead to local fitting. Therefore, in graph matching, more emphasis is placed on matching between edges. The graph matching algorithm is modified as follows:

[0041]

[0042] Where α and (1-α) are the weights between the point set and the edge set. Through experimental comparison in the present invention, the best effect is achieved when α is 0.1 in cross-source point cloud matching.

[0043] Finally, we get Figure C A and C B The corresponding relationship T between them, when the corresponding relationship is T, Figure C A and C B The global consistency J is the largest.

[0044] Step 6: Calculate the transformation matrix of the cross-source point cloud sets A and B based on the graph matching result T in step 5. Specifically, the graph C is obtained in the graph matching result T A and C BThe correspondence between points and edges in the figure, where the nodes are supervoxel sets V A ,V B ESF Descriptor V EA and V EB , so V EA and V EB The corresponding relationship can get the corresponding point p between the cross-source point cloud set A', B' a , p b , and then calculate the transformation matrix R and translation vector U from the corresponding points of the two point clouds. a p b The corresponding calculation formula is as follows:

[0045] p a =Rp b +U

[0046] The two cross-source point clouds are registered using the transformation matrix R and the translation vector U. First, the 3D RANSAC algorithm is used to remove outliers from the matching results between point cloud set A and radar scan point cloud set B. The iterative closest point (ICP) algorithm is then used to refine the point cloud registration, reducing the reprojection error between point cloud sets A and B. Finally, a well-registered street view building point cloud set is obtained.

[0047] Table 1 Comparison of cross-source point cloud registration between the proposed method and other point cloud registration methods

[0048] method WHU-TLS KITTI ICP 0.963 0.974 GO-ICP 1.992 2.161 RANSAC -0.067 -0.041 GMMReg 1.764 1.813 JR-MPC 0.257 0.311 4PCS 0.941 0.975 Method of the present invention -0.751 -0.722

[0049] Table 1 compares the proposed method with other registration methods for cross-source point cloud registration. The evaluation metric is log(RMSE), with lower values ​​indicating better registration. The datasets used are the WHU-TLS and KITTI datasets, along with cross-source point clouds generated using the SFM method using the corresponding images. The data in the table demonstrates that traditional point cloud registration methods such as ICP, GO-ICP, and 4PCS exhibit relatively low performance in cross-source point cloud registration, while the proposed method achieves superior performance.

[0050] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0051] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A street view building cross-source point cloud registration method based on graph matching, characterized by: The following steps are involved: Step 1: Obtain a video of a street view building and use the SFM method to process the continuous images of the street view building to obtain a point cloud set A of the street view building. Use a LiDAR to scan the same street view building to obtain a LiDAR point cloud set B. Step 2: Normalize the two cross-source point cloud sets A and B to the same scale; Step 3: Using the geometric characteristics of the 3D point cloud, the street view building point cloud is segmented into supervoxels and feature descriptors of these supervoxels are extracted; Step 4: Combine the supervoxels extracted in step 3 and Its feature descriptors are used as nodes of the graph, and the neighboring relationships of supervoxels are used as edges to construct the graph; the constructed graph is the macro structure of two cross-source point clouds; Step 5: Optimize the existing graph matching algorithm for point cloud registration, transforming 3D point cloud registration into graph structure matching. Specifically, according to the requirements of cross-source point cloud matching, weights are set for point matching and edge matching in graph matching, and the graph matching weights in cross-source point cloud registration are obtained through optimization. Step 6: Perform street view building point cloud registration based on the correspondence relationship obtained from the graph matching result obtained in step 5, and finally obtain the rigid transformation registration result between the two cross-source point cloud sets.

2. A street view building cross-source point cloud registration method based on graph matching according to claim 1, characterized in that: Step 2: Normalize the scales of two cross-source point clouds. These two point clouds come from different sensors and therefore have different scales. Based on the sensor parameters of the point clouds, use the growing least squares descriptor method to convert the scales of the two cross-source point clouds to the same. The real-world point clouds scanned by the lidar scanning method have the same scale as the actual streetscape buildings. Normalize the scale of point cloud A to the same as point cloud B. A'=F(A) B'=B Where A', B' are normalized point cloud data, and function F is the scale mapping from point cloud set A to B.

3. The method for cross-source point cloud registration of street scene buildings based on graph matching according to claim 2, characterized in that: Step 3: Using the geometric properties of the 3D point cloud, the cross-source point cloud set A', B' is divided into supervoxels respectively; specifically, the geometric properties of the points in the 3D space are used to cluster the similar points in the point set, and then a surface mesh is formed by three similar points, and each surface is a supervoxel; V A is the set of supervoxels obtained by segmenting the point cloud set A', V B is the set of supervoxels obtained by segmenting the point cloud set B'; at the same time, the supervoxel set V A ,V B Based on describing the surface distance, angle and area distribution of part of the point cloud, the ESF descriptor V is obtained using the shape function. EA ,V EB .

4. The method for cross-source point cloud registration of street scene buildings based on graph matching according to claim 3, characterized in that: The specific method of step 4 is as follows: the supervoxel set and the corresponding EFS descriptor are used as the nodes of the graph, and the adjacent relationship is used as the edge of the graph to construct the graph C of the two cross-source point cloud sets. A and C B ; Define the description of the graph C = {P, Q, H}, where P is the set of points in the graph, that is, the set of supervoxels and shape descriptors, Q is the set of edges, and H is the adjacency matrix between points.

5. The method for cross-source point cloud registration of street scene buildings based on graph matching according to claim 4, characterized in that: Step 5: Match the constructed graph; improve the existing graph matching algorithm FGM algorithm; specifically, graph matching requires C A and C B Find the corresponding relationship between them so that the global consistency J is maximized; Among them, x∈{0,1} represents the corresponding relationship between the nodes represented by its subscript, is the matrix k p The element at row i1 and column i2 in is the matrix k q The element in row c1 and column c2 of the matrix k p Figure C A and C B The similarity matrix of the point set P, matrix k q Figure C A and C B The similarity matrix of the edge set Q; Since Figure C A and C B The graph is converted from a point cloud set. Graph matching is for the final point cloud registration work, so the graph matching algorithm is optimized; in graph matching, more emphasis is placed on matching between edges, and the graph matching algorithm is modified: Where α and (1-α) are the weights between the point set and the edge set; Finally, we get Figure C A and C B The corresponding relationship T between them, when the corresponding relationship is T, Figure C A and C B The global consistency J is the largest.

6. The method for cross-source point cloud registration of street scene buildings based on graph matching according to claim 5, characterized in that: α is set to 0.

1.

7. A street view building cross-source point cloud registration method based on graph matching according to claim 5 or 6, characterized in that: Step 6: Calculate the transformation matrix of the cross-source point cloud sets A and B according to the graph matching result T in step 5; specifically, obtain the graph C in the graph matching result T A and C B The correspondence between points and edges in the figure, where the nodes are supervoxel sets V A ,V B ESF Descriptor V EA and V EB , so V EA and V EB The corresponding relationship can get the corresponding point p between the cross-source point cloud set A', B' a , p b , and then calculate the transformation matrix R and translation vector U from the corresponding points of the two point clouds; where the corresponding point p a p b The corresponding calculation formula is as follows: p a =Rp b +U Two cross-source point clouds are registered based on the transformation matrix R and the translation vector U. First, the 3D RANSAC algorithm is used to remove outliers in the matching results of point cloud set A and radar scan point cloud set B. Then, the iterative closest point algorithm is used to refine the point cloud registration and reduce the reprojection error of point cloud sets A and B. Finally, a well-registered street view building point cloud set is obtained.

Images