Incremental multi-group point cloud registration method, device and equipment

Abstract

The invention relates to an incremental multi-group point cloud registration method, device and equipment, and the method comprises the steps: building a sparse scene graph through retrieval and geometric verification, and constructing a potential overlapping relation of point clouds; and performing point cloud registration processing through incremental registration steps such as initial point cloud pair selection and registration, iteration type next frame selection and registration, track creation and continuation, binding adjustment and the like. In consideration of a nonlinear coupling relationship between poses of a plurality of groups of point clouds, a plurality of error control algorithms and optimization strategies are introduced into a point cloud registration processing framework, so that global consistency of the whole point clouds is effectively considered, processing of abnormal values of local regions is also considered, the accuracy, robustness and integrity of registration can be effectively improved, and the accuracy, robustness and integrity of point cloud registration are improved. And the registration performance is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and relates to an incremental multi-group point cloud registration method, device and equipment. Background Art

[0002] Point cloud registration is an important problem in the field of computer vision. It plays a key role in applications such as 3D reconstruction, robot navigation, and medical image processing. Different registration methods have their own advantages and applicable scenarios. Choosing the right method can improve the accuracy and robustness of registration. At present, the research in the field of multi-group point cloud registration mainly focuses on global multi-group registration methods. This type of method regards multi-group registration as a graph optimization problem and uses the relative pose coupling relationship to minimize the relative pose error between all nodes in the scene graph. The workflow usually includes two steps: the first step is to use the feature matching algorithm to restore the relative pose of any two local point clouds; the second step is to optimize the matching error between any two point clouds to solve the absolute pose of all point clouds in the same coordinate system. This algorithm focuses on overall optimization and the pursuit of global consistency of the scene. It lacks an effective repair or elimination mechanism for local outliers (such as wrong matching pairs) and is sensitive to input data errors or outliers. Even if there are only a few wrong matching pairs, it may have a significant impact on the optimization process, affecting the robustness of the algorithm, resulting in technical problems of insufficient performance in registration. Summary of the invention

[0003] In response to the problems existing in the above-mentioned traditional methods, the present invention proposes an incremental multi-group point cloud registration device, an incremental multi-group point cloud registration method and a computer device, which can effectively improve the accuracy, robustness and completeness of the registration, thereby greatly improving the registration performance.

[0004] In order to achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0005] On the one hand, an incremental multi-group point cloud registration method is provided, comprising the steps of:

[0006] Perform correlation search on unordered point cloud collections to construct a sparse scene graph that carries the relative pose relationship of point clouds; correlation search includes global point cloud retrieval, pairwise registration of point clouds, and geometric verification;

[0007] Using the correspondence between feature points provided by the sparse scene graph, feature trajectory constraints are incrementally constructed from the feature point level; the feature trajectory is used to record the matching relationship between each feature point in different frames;

[0008] The feature points in the feature trajectory are fused through aggregation operation to obtain the landmark points in the global coordinate system;

[0009] The 3D-3D correspondence between the scanned frame and the landmark points is constructed according to the feature trajectory constraints, and the absolute position and posture of the scanned frame is obtained by using singular value decomposition-random consistency sampling.

[0010] On the other hand, an incremental multi-group point cloud registration device is also provided, comprising:

[0011] The scene construction module is used to perform correlation search on unordered point cloud sets and construct a sparse scene graph that carries the relative pose relationship of point clouds; the correlation search includes global retrieval of point clouds, pairwise registration of point clouds, and geometric verification;

[0012] The trajectory constraint module is used to use the correspondence between feature points provided by the sparse scene graph to incrementally construct feature trajectory constraints from the feature point level; the feature trajectory is used to record the matching relationship between each feature point in different frames;

[0013] The landmark module is used to fuse the feature points in the feature trajectory through aggregation operations to obtain the landmark points in the global coordinate system;

[0014] The absolute pose module is used to construct the 3D-3D correspondence between the scan frame and the landmark points according to the feature trajectory constraints, and obtain the absolute pose of the scan frame by using singular value decomposition-random consistency sampling.

[0015] On the other hand, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned incremental multi-group point cloud registration method when executing the computer program.

[0016] One of the above technical solutions has the following advantages and beneficial effects:

[0017] The above-mentioned incremental multi-group point cloud registration method, device and equipment, in order to solve the problem of point cloud disorder, establish a sparse scene graph through retrieval and geometric verification, and construct the potential overlapping relationship of point clouds; then perform point cloud registration processing through incremental registration steps such as initial point cloud selection and registration, iterative selection and registration of the next frame, trajectory creation and continuation, and bundle adjustment. Considering the nonlinear coupling relationship between the poses of multiple groups of point clouds, multiple error control algorithms and optimization strategies are introduced into the framework of the point cloud registration processing, which effectively takes into account the global consistency of the overall point cloud and the processing of outliers in the local area, which can effectively improve the accuracy, robustness and integrity of the registration, and greatly improve the registration performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of an incremental multi-group point cloud registration framework in one embodiment;

[0020] Figure 2 A schematic diagram of a process of an incremental multi-group point cloud registration method in one embodiment;

[0021] Figure 3 1 is a flowchart of correlation search in one embodiment; wherein (a) is an unordered point cloud, (b) is a point cloud fully connected graph, (c) is point cloud retrieval, (d) is pairwise registration, (e) is geometric verification, and (f) is a sparse connection graph;

[0022] Figure 4 Schematic diagram of constructing an initial scene graph in one embodiment; wherein (a) is a fully connected graph of an unweighted scene, (b) is a scene connection graph with weights, and (c) is a retrieval result;

[0023] Figure 5 Schematic diagram of geometric verification in one embodiment; wherein (a) is a fully connected graph before verification, (b) is a connection graph during verification, and (c) is a coefficient scene graph;

[0024] Figure 6 It is a flow chart of the incremental registration stage of IncreMVR in one embodiment;

[0025] Figure 7 A schematic diagram of a scene graph centrality score in one embodiment;

[0026] Figure 8 A schematic diagram of an aggregation operation during an initialization process in an embodiment;

[0027] Fig. 9 is a flow chart of pose estimation based on a clustering method in one embodiment;

[0028] Fig.10 is a feature point classification diagram in one embodiment;

[0029] Fig.11 A schematic diagram of BA optimization in one embodiment;

[0030] Fig.12 It is a schematic diagram of the fragmentation feature trajectory;

[0031] Fig.13A flowchart of feature point position optimization in one embodiment;

[0032] Fig.14 A correlation search scene graph in an embodiment; wherein (a) is a full scene graph, (b) is a coefficient scene graph based on global features, (c) is a coefficient scene graph based on global and local features, and (d) is a minimum spanning tree;

[0033] Fig.15 Schematic diagram of the Top-10 search results in one embodiment; wherein (a) is the true value overlap rate, and (b) is the pose estimation accuracy rate;

[0034] Fig.16 A visualization diagram of an observed point cloud and its matching point cloud results in one embodiment;

[0035] Fig.17 It is a line graph of incremental growth of landmark points in one embodiment; wherein (a) is the number of trajectories continued or created per frame of point cloud in the incremental process, and (b) is the change of landmark points;

[0036] Fig.18 A schematic diagram showing visualization of multiple groups of registration results of three types of data sets in one embodiment;

[0037] Fig.19 Detailed comparison diagram of two sets of scene reconstruction results in one embodiment; wherein (a) is the result of SGHR, (b) is the result of IncreMVR, and (c) is the true value;

[0038] Fig. 20 Schematic diagram of comparison of registration integrity effect in one embodiment; wherein (a) is the result of SGHR, (b) is the result of IncreMVR, and (c) is the true value;

[0039] Fig.21 Schematic diagram of comparison between the presence and absence of a clustering module in one embodiment; wherein (a) is a bar graph of the number of correspondences involved in the registration of the i-th frame, and (b) is a scatter plot of absolute pose differences of the scanned frames;

[0040] Fig. 22 Schematic diagram of projection error growth trend with and without BA optimization in one embodiment; wherein (a) is the projection error trend with and without BA optimization in the reconstruction process, and (b) is a single BA projection error convergence curve;

[0041] Fig.23 Schematic diagram of the module structure of an incremental multi-group point cloud registration device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as those commonly understood by those skilled in the art of the present application. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0043] It should be noted that the reference to "embodiment" in this application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The presentation of this phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It will be appreciated by those skilled in the art that the embodiments described in this application may be combined with other embodiments. The term "and / or" used in the specification of the present invention and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0044] To fully construct a large-scale scene, it is often necessary to scan the scene multiple times due to the limitation of the sensor's perspective, and the scanned point cloud must be aligned to the same coordinate system. When the point cloud to be registered is obtained by sequential scanning by the sensor, the point cloud will be stored in a structured form and registered strictly according to the sequence information corresponding to the order. This type of task is relatively simple. Another case is that the point cloud data comes from multiple sensor sources or multiple acquisition periods. The point cloud collection is in a disordered state and cannot be aligned using sequence information. At this time, the disorder in the point cloud collection will produce a "jigsaw" problem, that is, how to find adjacent point clouds with overlapping relationships to facilitate subsequent feature point matching and pose solving. However, there are obvious differences in perspective between unordered point clouds, or there are repeated geometric structures, which hinder the correct construction of overlapping relationships between point clouds. This paper aims to explore a robust multi-group point cloud registration method to solve the problem that unordered point cloud collections are prone to mismatching. The research goal of this paper is: given a set of N s An unordered point cloud collection of point cloud data (scan frames) By estimating the absolute pose of a set of scan frames All scanned frames are registered to the same reference coordinate system to achieve reconstruction of the complete scene. i represents the i-th scan frame, T i is the i-th pose, (R i , t i ) represents the correlation between scan frames.

[0045] The task of 3D reconstruction based on 2D image sequences is also a long-term research topic in stereo vision. Its process objectives are similar to those of multi-point cloud registration tasks. The mainstream technical route in this task is the incremental Structure-from-Motion (SfM) method, which first extracts and matches the features of the target image, and then geometrically verifies and generates a perspective scene graph. In the reconstruction stage, the best image pair is first selected for initialization, and then new perspectives are incrementally added, outliers are filtered, feature trajectories are created and continued, and the reconstruction results are gradually improved using Bundle Adjustment (BA).

[0046] This paper designs an incremental multi-view registration method IncreMVR (Incremental MultiviewRegistration, IncreMVR), whose framework is as follows Figure 1 As shown. Unlike previous studies that usually adopt global solutions, IncreMVR adopts an incremental registration solution to gradually align the scanned frames into a global coordinate system. To address the problem of point cloud disorder, IncreMVR establishes a sparse scene graph through retrieval and geometric verification to construct potential overlapping relationships of point clouds; then performs incremental registration through the steps of initial point cloud selection and registration, iterative selection and registration of the next frame, trajectory creation and continuation, and bundle adjustment. Considering the nonlinear coupling relationship between the poses of multiple groups of point clouds, multiple error control algorithms and optimization strategies are introduced in this framework to effectively consider both the global consistency of the overall point cloud and the processing of outliers in local areas. In addition, IncreMVR designs a trajectory optimization module to realize the coarse-to-fine registration process, which can not only directly use the detector-based feature matching method, but also solve the problem of trajectory fragmentation in the detector-free matching method, so that the overall framework can adapt to two feature matching methods at the same time.

[0047] The following will describe the implementation of the present invention in detail with reference to the accompanying drawings in the embodiment diagram of the present invention.

[0048] See also Figure 2 In one embodiment, an incremental multi-group point cloud registration method is provided, which may include the following processing steps S12 to S18:

[0049] S12, performing correlation search on the unordered point cloud set to construct a sparse scene graph carrying the relative pose relationship of the point clouds; the correlation search includes global retrieval of point clouds, pairwise registration of point clouds, and geometric verification;

[0050] S14, using the feature point correspondence provided by the sparse scene graph, incrementally constructing feature trajectory constraints from the feature point level; the feature trajectory is used to record the matching relationship between each feature point in different frames;

[0051] S16, fusing the feature points in the feature trajectory through an aggregation operation to obtain a landmark point in a global coordinate system;

[0052] S18, constructing a 3D-3D correspondence between the scan frame and the landmark points according to the feature trajectory constraint, and using singular value decomposition-random consistency sampling to solve and obtain the absolute pose of the scan frame.

[0053] The above-mentioned incremental multi-group point cloud registration method, in order to solve the problem of point cloud disorder, establishes a sparse scene graph through retrieval and geometric verification, and constructs the potential overlapping relationship of point clouds; then performs point cloud registration processing through incremental registration steps such as initial point cloud selection and registration, iterative selection and registration of the next frame, trajectory creation and continuation, and bundle adjustment. Considering the nonlinear coupling relationship between the poses of multiple groups of point clouds, multiple error control algorithms and optimization strategies are introduced into the framework of the point cloud registration processing, which effectively considers the global consistency of the overall point cloud and takes into account the processing of outliers in local areas, which can effectively improve the accuracy, robustness and integrity of the registration, and greatly improve the registration performance.

[0054] It can be understood that given an unordered point cloud set This embodiment constructs a sparse scene graph of the point cloud through correlation search. The scene graph provides two important constraints for the subsequent incremental registration: the potential overlap between scanned frames and the correspondence between feature points. The scene graph is defined as Each of these nodes Represents the scan frame P i Its feature points {p a}, edge (i, j)∈ε represents the scan frame P i With P j The correlation between ij , t ij ), is a node set, and ε is an edge set. Generally speaking, the scene consists of a large number of scanned frames. If an exhaustive strategy is used to estimate the relative pose between each point cloud and all other point clouds when the point clouds are registered pairwise in the first stage, it will lead to more erroneous matching relationships and higher computational complexity. Therefore, this embodiment will make full use of the global and local features of the point cloud, extract feature information such as the geometric structure, shape or locality of the point cloud, search for scanned frames with potential relationships based on the feature information, and then construct the matching relationship between their feature points and estimate the relative pose to generate a sparse scene graph.

[0055] The correlation search is divided into the following three steps: (1) point cloud retrieval, (2) point cloud pairwise registration, and (3) geometric verification. The point cloud connection status changes are as follows: Figure 3 As shown. (a) is an unordered point cloud. The set is first initialized as a directed fully connected scene, as shown in (b). In the point cloud retrieval stage, the global features of the point cloud are extracted and the feature similarity is calculated. The irrelevant or weakly related connection relationships are quickly eliminated to construct an initial scene graph with weights, as shown in (c). After obtaining the initial scene graph, the point cloud registration algorithm based on local features extracts and matches the feature points of the potential point cloud pairs in the scene graph, and constructs pairwise correspondences and relative pose estimates, as shown in (d). Finally, in order to compensate for the possible ambiguity of global features, geometric verification is further added, as shown in (e). The local feature information extracted in the pairwise registration stage is used to detect and eliminate erroneous matching pairs in the scene graph, and the final sparse scene graph with relative pose relationships is obtained, as shown in (f).

[0056] Global retrieval of point clouds: First, a retrieval technique is used to construct an initial scene graph for the unordered point cloud. The retrieval technique searches a large-scale point cloud database and compares the global feature similarities between point clouds to quickly find point cloud pairs with potential overlapping relationships. NetVLAD is an image retriever based on deep learning. This image retriever fully utilizes the advantages of neural network feature learning, shows high accuracy in image retrieval tasks and has been extended to point cloud retrieval tasks. This embodiment uses an image retriever pre-trained on point cloud data to perform end-to-end point cloud retrieval and construct a point cloud scene graph connection relationship. The retrieval process is divided into the following three steps: (1) local feature extraction, (2) feature clustering, and (3) similarity calculation.

[0057] (1) Local feature extraction: This stage is used to capture the geometric and structural information around each point in the point cloud. Input: Unordered point cloud set First, for each scan frame P i Using voxel grid sampling (VGS), the following neighborhood point set is constructed:

[0058] N p ={p′|||pp′||2<r,p′∈P i} (1)

[0059] in, P i The downsampled point set, as the sphere center of each voxel, is called a superpoint. i Each point p′ is divided into the nearest sphere with radius r.

[0060] NetVLAD will use the neighborhood point set Np The local descriptor corresponding to the superpoint p is constructed using the information. The local descriptor is used to characterize the local area features in the three-dimensional point cloud data. In this embodiment, the local descriptor of each superpoint is extracted and constructed by the YOHO (You Only LookOnce, object detection based on deep learning) algorithm with rotation equivariance:

[0061]

[0062] in, is the YOHO feature extractor, f p is the local feature corresponding to the sampling point p.

[0063] Get the local area feature set of all super points {f p}, NetVLAD performs p} Perform local feature aggregation and output the global features of each scan frame:

[0064] F=NetVLAD({f p}) (3)

[0065] in, Represents the global features of the point cloud, and K is the length of the global feature vector.

[0066] After obtaining the global features of each frame of point cloud, select any scan frame P i , the global feature similarity between this scan frame and other scan frames is calculated as follows:

[0067]

[0068] Among them, F i With F j P i With P j The global characteristics of s ij ∈[0, 1] represents P i With P j The overlap fraction between . Figure 4 As shown in (a), for the fully connected graph of the initialized unweighted scene, after NetVLAD traverses and searches each scan frame in the database, the output is as follows Figure 4 (b) shows the weighted scene connection graph. Each node in this graph Represents each scan frame P i , edge (i, j)∈ε includes the scan frame P i With P j The similarity estimate between them is the overlap score, where the depth of the color indicates the overlap score. The darker the color of the line, the higher the overlap score.

[0069] Point cloud P i The overlap fraction set s i Sort and take the point clouds corresponding to the first k (i.e. top-k retrieval) as the point clouds corresponding to P i Overlapping objects:

[0070]

[0071] After the above top-k search, each scan frame corresponds to k search objects, and the search results will be used as the initial connection relationship ε of the scene graph. coarse ,like Figure 4 (c) as shown.

[0072] Point cloud pairwise registration: This embodiment uses feature extraction and matching algorithms to extract each node v in the scene graph. i The feature point set {p a}, and the connection relationship ε of the scene graph coarse Each edge (i, j) in the And use SVD (based on singular value decomposition)-RANSAC (RANdom SAmple Consensus, random sampling consensus algorithm) to estimate the six-degree-of-freedom pose (R ij , t ij ) and output the number of correspondences and the inlier rate.

[0073] Deep learning methods have become increasingly mature in point cloud feature extraction and matching tasks. Currently, many related algorithms have emerged, and they have become the mainstream technical route in this field with their excellent performance. Existing deep learning matching algorithms can be mainly divided into two mainstream technical routes: detector-based and detector-free. The two methods have their own advantages in terms of registration effect and their own strengths in applicable scenarios. In scenes with rich features, the detector-based matching method can extract a large number of feature points for matching, thereby obtaining accurate registration results. For scenes with less or repeated textures, the detector-based method may not be able to extract enough feature points for matching, while the detector-free matching method is not subject to this limitation. Therefore, this embodiment will adapt these two matching methods. In actual applications, the appropriate method is selected according to the specific scenario and needs.

[0074] Detector-based matching methods such as the YOHO algorithm and the neural network SpinNet can generate correspondences between fixed feature points and participate in the subsequent feature trajectory construction. This type of method is fully adapted to the framework of this embodiment and can directly call and construct the 3D-3D correspondence between feature points in the scene graph. and estimate the six-degree-of-freedom pose (R ij , t ij ).

[0075] Another detector-free matching method adopts the form of dense matching, lacks fixed feature points, and has the problem of trajectory fragmentation during multi-group registration, and cannot be directly adapted to the framework of this embodiment. This embodiment improves the detector-free method so that this type of method can also be applied to the framework of this embodiment. Taking the GeoTransformer (geometric transformer) algorithm, a detector-free point cloud feature extraction method, as an example, the GeoTransformer algorithm uses a coarse-to-fine matching strategy to obtain 3D-3D correspondences. In the coarse matching stage, dense matching is performed between the downsampled features of the two point clouds to obtain superpoint correspondences, and in the fine matching stage, the correspondence positions are optimized on the fine-grained feature map. This framework uses the correspondence of the GeoTransformer algorithm in the coarse registration stage to participate in the construction of feature trajectories, generate rough multi-group registration results, and then additionally designs an optimization module for multiple perspectives to optimize the feature point positions at a fine-grained level to further improve the accuracy of the registration results. The optimization details are introduced later.

[0076] Geometric verification: In the above steps, the scene graph is pre-retrieved to obtain the initial connection relationship ε coarse , and use the matching algorithm for each edge e ij ∈ε coarse Given a corresponding relationship and relative pose (R ij , t ij ) as weights. Although the retrieval achieves effective sparsification of the scene graph, in order to improve computational efficiency, point cloud retrieval technology usually focuses more on extracting high-level semantic features of the point cloud, such as shape, etc. This feature association method may cause incorrect connections due to the lack of feature discrimination when facing scenes with many repeated structures. In order to overcome this limitation, this embodiment adds geometric verification to the construction of the scene graph to make full use of the spatial information of the point cloud and filter out errors and ambiguities in similarity that occur in the retrieval.

[0077] This embodiment selects the number of inliers and the inlier rate as verification indicators, calculates the Euclidean distance between point pairs through geometric transformation, and sets a distance threshold τ d , matches exceeding this distance threshold are considered valid inliers, while matches below this distance threshold are considered outliers. ij ∈ε coarse , use the following geometric transformation to count the corresponding relationship of each edge The number and rate of interior points of :

[0078]

[0079]

[0080] Among them, NM is The number of corresponding relations, N inlier for The Euclidean distance after geometric transformation is less than the distance threshold τ d The number of interior points, ζ inlier The inlier rate is the inlier rate, which reflects the degree of fit of the geometric transformation model to the corresponding relationship data.

[0081] When the inlier rate is high, it means that the fitted model can fit most of the data points well, that is, the overall correspondence is consistent, and the model's description of the correspondence is relatively accurate and robust; when the inlier rate is low, the model may be interfered by a large amount of noise or outliers, the fitting effect is poor, and the entire correspondence cannot be well described. Therefore, this embodiment introduces the inlier number threshold τ n and the inlier rate threshold τ p , set the following verification conditions to filter out abnormal edges:

[0082] ε={N (ij) inlier >τ n ,ξ (ij) inlier >τ p |e ij ∈ε coarse (8)

[0083] like Figure 5 (d) shows that the dark solid lines are edges that meet the verification conditions, and the dashed lines are abnormal edges. After geometric verification, only Figure 5 The dark solid line in (e) connects the lines, and the final output Figure 5 (f) Sparse scene graph.

[0084] Through the above three steps of point cloud global retrieval, point cloud pairwise registration and geometric verification, this embodiment constructs a sparse scene graph with potential connection relationships between unordered point cloud sets. Compared with the exhaustive method to obtain the correspondence, the correlation search method proposed in this embodiment has two major advantages in constructing a sparse scene graph: (1) Accelerating the search process. Point cloud retrieval only needs to quickly locate the possible matching point cloud subset through spatial partitioning or index structure, thereby greatly reducing the search space and improving the matching efficiency; (2) Reducing redundant calculations. Directly registering all point clouds pairwise may result in a large amount of redundant calculations, especially when processing large-scale data. Point cloud retrieval can reduce the redundant calculation burden by quickly eliminating point clouds that cannot be matched; and further use local features to construct correspondences and geometric checks on the sparse scene graph, providing more accurate and robust connection relationships for subsequent multi-group registrations.

[0085] The purpose of the incremental registration stage is to align all scanned frames in the same coordinate system using two correlation constraints in the scene graph after a given sparse scene graph, that is, to solve the absolute position of the scanned frame. Compared with the global registration method, the incremental process focuses more on local optimization. Through the step-by-step registration method, the scanned frame can be locally adjusted in each step, and then iterated step by step until the overall registration goal is achieved. Therefore, the incremental registration shows greater flexibility and robustness, and is suitable for different scenarios and application requirements.

[0086] This embodiment takes incremental registration as the basic idea, and uses the correspondence between feature points provided by the sparse scene graph to incrementally construct feature track constraints from the feature point level. The feature track is used to record the matching relationship between each feature point in different frames. By creating and continuing the feature track, the newly added data in the incremental process is associated with the previous data. Subsequently, the feature points in the track are fused through aggregation operations to obtain landmark points in the global coordinate system. With the help of track constraints, the scanned frame constructs a 3D-3D correspondence with the landmark points and uses SVD-RANSAC to solve the absolute pose. Figure 6 As shown in Figure 1, the incremental registration process consists of five stages: (1) initial point cloud pair selection and registration; (2) next frame point cloud registration; (3) outlier filtering; (4) feature trajectory creation and continuation; and (5) bundle adjustment.

[0087] Incremental registration initialization: Initialization mainly refers to the selection and registration of the initial point cloud pair. A pair of initial point clouds (P0, P1) are selected from the image, and the corresponding relationship between the two is used to initialize the feature trajectory and generate landmark points in the global coordinate system through aggregation operations. The selection and registration of the initial point cloud pair is the basis for the subsequent incremental registration. If the error in this step is large, the subsequent registration is likely to fail due to the error accumulation and transmission effect.

[0088] In order to obtain a good incremental registration initialization, this embodiment designs a heuristic initial pair selection strategy based on two principles of initial pair selection in the existing three-dimensional reconstruction field: the distance between each viewpoint and the initial pair should be kept at a low level and the initial reconstruction points should be sufficient. Specifically, this embodiment considers the two principles of centrality and the number of corresponding relationships as constraints in the heuristic search for the initial point cloud pair, where the centrality refers to the position of the initial point cloud pair on the scene graph, which can reflect the degree of correlation between the scanned frame and other viewpoints on the scene graph. The closer the scanned frame is to the center of the scene graph, the higher the centrality score. Figure 7As shown in the figure, node C is at the center of the scene graph and has the highest centrality. If a node closer to the center of the scene graph is selected as the initial point cloud pair, the distance between the remaining nodes and the initial point cloud pair can be kept at a low level overall, maximizing the compactness and reducing error accumulation.

[0089] Therefore, given a sparse scene graph In this embodiment, all nodes are sorted according to their centrality. Suppose a group of nodes The centrality is The centrality of each node is calculated as follows:

[0090]

[0091] Where c is the proportional constant, a ij x j Represents the centrality of the neighbors. The centrality of node i is obtained by summing the centralities of its neighbors. It is diffused iteratively until the centrality ratio of each node in the entire scene graph remains stable, and the final centrality of each node is obtained. Then all node centralities are sorted and the node with the highest centrality is set as the initial node, denoted as P0.

[0092] After selecting the initial node P0, this embodiment uses the number of corresponding relationship constraints to select the next node P1. The corresponding relationship between point cloud pairs is of great significance for the subsequent aggregation operation to generate 3D landmark points. The more corresponding relationships there are, the more initial landmark points will be generated, providing more trajectory constraints for subsequent registration and improving the stability and accuracy of the overall registration. Therefore, this embodiment counts the number of corresponding relationships m between the initial node P0 and its adjacent nodes, and sets a minimum corresponding relationship number threshold τ m The number of corresponding relations is the largest and satisfies m>τ m The node is recorded as the second node P1. When the number of correspondences of the initial nodes does not meet the requirements, considering that there are enough scan frames in the set, the scan frames located in the center of the scene graph have little difference in the overall registration effect. Therefore, the centrality constraint can be appropriately relaxed and set as a soft constraint. The number of correspondences remains as a hard constraint, and then the initial nodes are replaced in sequence according to the centrality sorting until the number of correspondences is met.

[0093] When an initial point cloud pair (P0, P1) is selected and the relative pose (R 01 , t 01), this embodiment creates a trajectory and 3D landmark points to generate an initial registration model. The trajectory is composed of a set of feature points representing the same spatial point in the global coordinate system on different scanning frames. To avoid ambiguity, the spatial point in the global coordinate system is named as a landmark point. Aggregation operation is used to fuse the feature points in the trajectory to generate 3D landmark points, as shown in the figure. Figure 8 As shown:

[0094] Figure 8 There are two groups of point cloud frames (P0, P1) on the left side of the middle, one light and one dark, which contain the same scanning area. a With p b They come from the light-colored point cloud frame and the dark-colored point cloud frame and form a corresponding relationship. Through the aggregation operation, the two feature points are converted to the global coordinate system and merged into landmark points. Specifically, let the coordinate system of the first node P0 be fixed as the global coordinate system, the absolute pose of node P0 is the unit matrix (R0, t0) = (I, 0), and the absolute pose of node P1 is equivalent to the relative pose (R1, t1) = (R 01 , t 01 ), and then based on the 3D-3D correspondence As well as the absolute poses (R0, t0) and (R1, t1), a set of linear equations is constructed, whose goal is to minimize the reprojection error of the feature points after geometric transformation to obtain the 3D landmark point q ab , as shown below:

[0095]

[0096] Through For each pair of corresponding relationships in the above equation, and solve the optimal solution, we can get a set of 3D landmark points {q ab The generated landmark points and the set of multi-view feature points involved in creating the points are uniformly represented as feature trajectories: The association information between each landmark point and multiple viewpoint feature points is provided, and the corresponding relationship will be used to continue the existing feature track or create a new feature track in the subsequent incremental registration process.

[0097] Next frame point cloud registration: After initialization, the new scan frame will continue to build a 3D-3D correspondence with the aggregated landmark points and register it to the current model by solving an iterative nearest neighbor (ICP) problem. In order to minimize the error accumulation of each frame registration and maintain the overall pose estimation quality and the integrity of the registration model, this embodiment selects the scan frame P that can observe the most landmark points. l , as one of the criteria for selecting the next frame of point cloud. Assuming that there are k scan frames involved in the registration, the feature point pa With p b Belong to any two registered scan frames, and jointly participate in the construction of the ,th trajectory to generate a landmark point q ab There exists another 3D point p in the unregistered scan frame c , this point is related to p a Have a corresponding relationship, so p c is considered to be able to observe the landmark point q ab In addition to the above-mentioned landmark point observation quantity evaluation index, this embodiment takes into account the principle of higher confidence in geometric modeling under multi-view constraints, and uses the number of associated point clouds as another evaluation index for the selection of the next frame of point clouds. The associated point clouds of the scanned frames are required to have been registered and have a corresponding relationship with them.

[0098] The next frame point cloud selection strategy is as follows: by traversing the unregistered point cloud set, select the one with the largest number of associated point clouds, the largest number of landmark points, and the number of points exceeding the set threshold τ l The scanned frame is used as the next frame of point cloud P to be registered l ; Otherwise, it is lower than the set threshold value τ l , the incremental registration process is interrupted and a new unregistered scene graph is reconstructed.

[0099] After selecting the next frame of point cloud to be registered, the 3D-3D correspondence between feature points and landmark points can be directly used to solve the absolute pose. Although the selection strategy proposed above can obtain as many correspondences as possible to participate in the absolute pose estimation, the absolute pose estimation cannot be guaranteed to be accurate by relying solely on quantity. The main reason is that the scene graph constructed by the first-stage correlation search contains residual erroneous connection relationships. These erroneous connections may cause two unrelated frames to be registered and link the subsequent frame processing, resulting in significant error accumulation.

[0100] In order to solve the above error accumulation problem, this embodiment designs a clustering-based pose estimation method. The overall process is as follows: Fig. 9 As shown, P1, P2, and P3 are registered point clouds, and P l The pose estimation of is divided into the following three steps: candidate pose estimation, candidate pose clustering, and final pose estimation. Specifically, the next frame point cloud P is selected l , with the help of P on the scene graph l and adjacent registered nodes (P1, P2, ..., P m ) to construct multiple sets of 3D feature point-landmark point correspondences, denoted as Then the RANSAC algorithm is used to estimate the candidate poses Then cluster this set of poses based on the differences in rotation and offset:

[0101]

[0102] RTE=||t1-t2||2 (12)

[0103] Among them, RRE and RTE represent the relative angle error and offset error between the two estimated absolute poses (R1, t1) and (R2, t2), respectively. There may be outliers generated by incorrect connection relationships in the set of estimated poses using RRE and RTE indicators.

[0104] right After clustering, the largest cluster is selected as the connection relationship set with the highest confidence, and after adding an outlier filtering mechanism, the valid feature point-landmark point correspondences in the cluster are gathered, and the RANSAC algorithm is used to solve the absolute pose T of the final next frame point cloud. l =(R l , t l ). By utilizing the candidate pose clustering process described above, geometrically consistent correspondences can be effectively clustered together, and outliers can be dispersed and filtered.

[0105] Feature trajectory creation and continuation: next frame point cloud P l After registration, P l The feature points are divided into three types according to whether they participate in trajectory update, such as Fig.10 As shown in the figure, the first category is that there is a corresponding relationship and landmarks can be observed, and they participate in the feature track continuation (Track continue); the second category is that there is a corresponding relationship but no landmarks, and they participate in track creation (Track create); the third category is that there is no matching pair and will not participate in the track update.

[0106] The specific operation of trajectory continuation is: given P l The first type of feature point p c , and its corresponding landmark point q ab and trajectory Create the following new linear equation for the trajectory and perform aggregation operations to update the landmarks:

[0107]

[0108] Among them, q abc Added p c After the above update, the trajectory will be continued as

[0109] The trajectory creation will further expand the registration model and the geometric structure recovery of the scene. a or p in P1 b , and P l p cIf there is a corresponding relationship and the trajectory has not been created, execute equation (10) to aggregate the new 3D landmark points and create a new trajectory.

[0110] To ensure the reliability of feature points involved in creating and continuing trajectories, the framework of this embodiment adds multiple outlier filtering mechanisms, including: (1) Distance constraint: Add Euclidean distance constraints to the corresponding relationships involved in absolute pose estimation, trajectory continuation or creation, and set a threshold τ d , only the inliers with a value smaller than the set threshold are allowed to participate in the subsequent registration steps. (2) Geometric consistency constraint: In the process of aggregation and absolute pose estimation, the RANSAC algorithm is added to fit the data by random sampling, retaining the data that passes the consistency test and removing the remaining outliers.

[0111] Bundle Adjustment: Error accumulation is a common problem in the incremental registration process. In the registration process, feature points are usually extracted from multiple point clouds, and the pose of the scanned frame and the position of the landmark points are calculated by matching the feature points. Due to factors such as sensor noise, feature matching errors, and inaccuracy in pose estimation, the registration results often have certain errors. BA (Bundle Adjustment) optimization is widely used to solve two main problems. First, error accumulation is one of the key issues that need to be solved in the incremental method. As new scanned frames are continuously received, new errors or uncertainties may be introduced, causing the reconstruction results to gradually deviate from the true results. Second, data association is another problem that needs to be solved. In incremental reconstruction, the newly added data needs to be associated with the previous data in order to correctly integrate it into the existing reconstruction results. BA optimization can make the association between the newly added data and the existing data more accurate and consistent by optimizing parameters such as pose and landmark point position, which helps to avoid data fragmentation or inconsistency and improve the robustness and stability of the entire scene.

[0112] In order to reduce error accumulation, this embodiment introduces the BA optimization mechanism after the next frame point cloud registration and trajectory update. Specifically, in order to ensure both optimization accuracy and efficiency, the optimization mechanism includes two optimization strategies: local BA optimization and global BA optimization, which are used in different registration stages respectively. The schematic diagram is shown in Figure 1 As shown in the figure. Local BA optimization acts in each round of incremental iterations, and only considers optimizing the trajectory of the current update state in the local range, so that the registration results of the newly added local areas are more accurate, and convergence to the local optimal solution in a shorter time is achieved. Global BA is responsible for staged adjustments and acts on all registered data, using global consistency constraints and minimizing observation errors in the overall range to obtain more comprehensive and accurate registration results.

[0113] Set the registration model to contain a set of feature tracks Each track Contains a landmark point q j And the corresponding feature points {p ij |i=1,...,M},set the absolute position of the scanning frame to (R i , t i ), the objective function of BA optimization is expressed as:

[0114]

[0115] Among them, the landmark point q of each feature trajectory j Using (R i , t i ) is projected onto the i-th scan frame to obtain the observation error between the projection position and the corresponding feature point. The purpose of BA optimization is to minimize the overall observation error by iteratively adjusting the parameters. Local BA will only adjust the scan frame pose (R i , t i ), the global BA will adjust the scan frame pose (R i , t i ) and landmarks {q i}.

[0116] The iterative optimization algorithm can use the LM (Levenberg-Marquardt) algorithm. In each iteration, the parameters are updated by calculating the gradient of the objective function and the Hessian matrix, so that the objective function value gradually converges to the minimum value. The update process is as follows:

[0117] Δθ=-(J T J+λI) -1 J T r (15)

[0118] Among them, Δθ is the increment of the parameter, J is the Jacobian matrix of the objective function with respect to the parameter, r is the residual vector, which represents the observation error, and λ is the damping coefficient, which is used to control the step size of the optimization process. By iteratively updating the parameters, the objective function value can be gradually minimized, thereby obtaining the optimal scanning frame pose and landmark point position.

[0119] In one embodiment, the above-mentioned incremental multi-group point cloud registration method may further include the following processing steps: extracting a fine-grained feature map of a block corresponding to each feature point from the feature trajectory;

[0120] The confidence matrix between the reference block and the fine-grained feature map of each query block is calculated using the optimal transfer algorithm;

[0121] The positions of the feature points in the feature trajectory are optimized and adjusted according to the confidence matrix to obtain an optimized feature trajectory;

[0122] The optimized feature trajectory is sent to the global BA algorithm to adjust the absolute pose of the scan frame and the landmark points to obtain the optimized registration result.

[0123] It can be understood that the trajectory optimization for the detector-free feature extraction method: the problem of adaptation of the detector-free feature extraction method: in the registration process, the accuracy of the spatial position of the feature points will directly affect the accuracy of the point cloud pose estimation and the position of the landmark points. The accurate feature point position provides more reliable data and improves the accuracy of the pose estimation. The scene graph relative pose estimation stage of this framework is compatible with two matchers based on correspondence and without correspondence. Among them, the matcher based on no correspondence, such as GeoTransformer, does not rely on specific feature detectors, and can obtain better registration results in scenes with less or repeated textures. However, since this type of method does not have fixed feature points, it is easy to have the following problems when participating in the construction of correspondence between multiple point clouds. Fig.12 The fragmented feature tracks shown in Figure 2 are not directly applicable to the multi-point cloud registration task.

[0124] Feature point optimization: Therefore, for the detector-free method to generate a registration model, this framework also designs a coarse-to-fine registration process. In the early incremental registration stage, a rough but complete registration model is generated by giving the superpoint correspondence constructed in the coarse registration stage of GeoTransformer. The set of feature trajectories that make up the model is defined as in This embodiment will fine-tune the GeoTransformer fine registration stage to adapt to the construction of multi-view correspondence relationships. Based on the principle of consistency of features in overlapping areas of multiple views, the optimal matching relationship is found in units of trajectories, and the matching point positions are updated. The optimization process is as follows: Fig.13 shown.

[0125] First, extract fine-grained feature maps: given a trajectory The feature points in In this embodiment, each feature point Construct k by nearest neighbor sampling The existing Kpconv feature extractor is used to construct a fine-grained feature map for each patch. Based on the trajectory constraints, each block represents the same area of ​​the scene, which can be used to find more precise point pair matches at a fine-grained level.

[0126] Second, generate the confidence matrix: Next, we need to calculate the similarity values ​​between all blocks to find the corresponding relationship with higher confidence. In order to reduce the computational complexity, we will randomly select a block. For reference perspective, the remaining blocks To query the perspective, the optimal transmission algorithm Sinkhorn in the original fine-grained registration stage of GeoTransformer is used to calculate the reference blocks With each query block The confidence matrix between the feature maps In the confidence matrix, each element represents the similarity between a pair of data points, and a high confidence means that the two data points are likely to have a corresponding relationship.

[0127] Third, adjust the position of feature points: By analyzing the confidence matrix, we can find data point pairs with high confidence. Since there may be multiple correspondences between blocks, in order to keep the number of trajectories unchanged, the position of the matching point in the reference perspective is fixed, and the point with the highest confidence in the query perspective is selected as the effective correspondence to obtain the optimized feature points. Finally, the optimized trajectory Once again sent to the global BA algorithm to adjust the absolute position of the scanned frame {(R i , t i )} and 3D landmark points {q j}, thus obtaining a more accurate registration result.

[0128] In this way, through the trajectory optimization scheme further designed above, the coarse-to-fine registration process is realized. Not only can the detector-based feature matching method be used directly, but also the problem of trajectory fragmentation in the detector-free matching method can be solved, so that the overall framework can adapt to the two feature matching methods at the same time, thereby further improving the registration performance and effect.

[0129] It should be understood that although the above process Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in the application, there is no strict order restriction for the execution of the steps, and the steps can be executed in other orders. Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequentially, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0130] In some embodiments, in order to more fully and intuitively illustrate the effect of the above-mentioned solution of the present application, an experimental example for the above-mentioned solution is given below:

[0131] The IncreMVR method proposed in this embodiment is evaluated on three existing indoor and outdoor scene datasets: 3D(Lo)Match, ScanNet and ETH. 3D(Lo)Match contains scan sequences from 62 indoor scenes. This example selects 8 test scenes for testing, including 1623 matching pairs with an overlap rate of >30% and 1781 matching pairs with a low overlap rate between 10% and 30%, respectively represented as 3DMatch and 3DLoMatch datasets. ScanNet contains RGBD sequences of 1513 indoor scenes. This example uses 32 test scenes, and selects 30 RGBD images for each test scene at an interval of 20 frames, converts them into point cloud sequences without retaining temporal information, totaling 960 frames of point cloud, including 13920 matching pairs. ETH contains 4 outdoor scenes, which has a larger field of view gap compared to 3DMatch and ScanNet datasets. Each scene scan covers a wide range of outdoor scenes, contains an average of 33 scan frames, and a total of 713 matching pairs are evaluated.

[0132] The following registration recall (RR) is used to evaluate the performance of IncreMVR on the 3D(Lo)Match and ETH datasets:

[0133]

[0134] Among them, RR means that the distance between the estimated pose and the true pose after transformation is less than the threshold τ r The ratio, N t is the number of points in the target point cloud, p i is the i-th point in the target point cloud, P s is the source point cloud, d(p i , P s ) is point p i To point cloud P s The closest distance, τ is a given threshold, is an indicator function that returns 1 when the condition in the brackets is true, otherwise it returns 0. Set the RR threshold τ of the 3D (Lo) Match dataset r = 0.2m, RR threshold τ of ETH dataset r =0.5m.

[0135] For the evaluation of the ScanNet dataset, this example uses the empirical cumulative distribution function (ECDF) of rotation error and translation error. ECDF is a function used to describe the error distribution, indicating how much of the sample data is less than or equal to the error threshold. The rotation error and translation error of each scan frame are calculated as follows:

[0136]

[0137] RTE=||t pre -t gt || 2 (18)

[0138] Among them, (R pre , t pre )、(R gt , t gt ) represent the estimated absolute pose and the true absolute pose, respectively. The recall rate of five rotation thresholds [3°, 5°, 10°, 30°, 45°], the mean and median of rotation error, and the recall rate of five translation thresholds [0.05m, 0.1m, 0.25m, 0.5m, 0.75m], the mean and median of translation error, are selected for the rotation error. A total of 14 ECDF indicators are used to evaluate the ScanNet dataset.

[0139] To verify the effectiveness of the incremental registration framework proposed in this paper, this example compares IncreMVR with several existing global multi-group registration methods: EIGSE3 registration method, L1-IRLS registration method, RotAvg registration method, LMVR registration method, LITS registration method, HARA registration method, SGHR registration method and MST registration method.

[0140] At the same time, in order to verify the accuracy of the point cloud scene graph generated by the IncreMVR method, an incremental registration algorithm based on the minimum spanning tree is added for comparison. By initializing the weighted fully connected graph and using the minimum spanning tree algorithm to determine the connection relationship between point clouds, this example uses the overlap rate as the weight coefficient of this method, which is consistent with all methods.

[0141] The experiment in this example uses two different groups of registration methods for relative pose estimation, including FCGF, SpinnNet, YOHO and GeoTransformer. At the same time, this example tests multiple groups of registration experiments with two initializations: the fully connected scene graph and the sparse scene graph in the retrieval step.

[0142] In the retrieval stage, the retrieval results of 3D(Lo)Match and ScanNet datasets are set to k=10, and k=6 for the ETH dataset to construct the scene graph. In the pairwise registration stage and the subsequent outlier filtering stage, the matching result is considered an outlier if the distance between two points meets the following conditions:

[0143] ||T ij (p a )-p b ||2<τ m (19)

[0144] Among them, (p a ,p b ) is the scan pair (P i , P j ), T ij Represents the estimated relative transformation. The framework of this paper is compatible with different point cloud matchers, where the detector-free method sets τ m = 20 cm, τ is set based on the detector method m =7cm.

[0145] In the geometric verification stage, 3D (Lo) Match and ETH internal point rates less than τ are removed respectively. p = 0.5, ScanNet inlier rate is less than τ p = 0.3 on the scene graph.

[0146] Scene graph construction: Fig.14 As shown in the figure, the results of the scene graphs of the fully connected, global method, this method and the minimum spanning tree corresponding to the sample scene are shown. The scene graph of the global method uses the point cloud retrieval method to take the Top-k results to build the scene graph; the incremental method proposed in this paper outputs a sparse scene graph according to the relevance search stage mentioned above; the scene graph of the minimum spanning tree builds the minimum spanning tree based on the overlap score of the point cloud retrieval stage.

[0147] Fig.15 The true value overlap rate and pose estimation accuracy of the Top-10 retrieval results of the 3DMatch dataset and the ScanNet dataset are reported. Fig.15 (a) shows the actual overlap rate of the Top-10 retrieval results with the retrieved scan frame after the pose truth transformation. From the broken line observation, it can be seen that the actual overlap rate and the overlap score show a good positive correlation trend, that is, the higher the score, the larger the overlapping area. Fig.15(b) shows the relative pose estimation accuracy of the Top-10 search results and the searched scan frames, with the pose accuracy limits RRE<10° and RTE<1. Among them, the light-colored curve only searches for the results before the Top-5, and the accuracy remains above 80%, while the Top-10 accuracy drops to about 40%. However, after adding geometric verification and filtering out false search pairs, the overall accuracy is greatly improved, and the Top-10 accuracy is increased to about 60%.

[0148] Fig.16 The distribution diagram of the results of the searched point cloud (marked by the dotted box) after the correlation search is shown. The Rank in the figure represents the search score ranking in the Top-10 search results, and the inlier rate comes from the corresponding relationship result of the pairwise registration. The scanned frames with higher search scores in the figure are closer to the searched point cloud, while the scanned frames with lower scores are farther away from the searched point cloud and have fewer overlapping areas, or even almost no overlapping areas. After the inlier rate constraint processing, the wrong search results Rank 8, 9, and 10 will be filtered out and do not constitute a connection relationship.

[0149] Feature trajectory construction and optimization: After a scene graph is given, a pair of initial point clouds are selected and trajectories are created in the initialization phase. As more scan frames are selected and registered, new trajectories will continue to be created and the current trajectory will continue. Fig.17 (a) The curve shows the number of trajectories continued or created per frame of the incremental process. Each trajectory will be aggregated into a landmark point, so the landmark points are as follows: Fig.17 As shown in (b), the number of registered point clouds gradually increases, thereby robustly building a three-dimensional point cloud model of the entire scene.

[0150] As shown in Table 1, the input and output of the two feature matchers, YOHO and GeoTransformer, are shown. The number of correspondences generated by the feature point-based (YOHO) match is greater than that generated by the block-based (GeoTransFormer) match, and the average feature trajectory length is relatively short.

[0151] Table 1

[0152]

[0153] In the incremental registration process, in order to maintain the accuracy of pose solution and landmark points, the BA optimization mechanism is added to maintain global consistency. In the experiment of this example, the BA optimization is written using the Pyceres library. Pyceres is the Python interface of the Ceres Solver library. Ceres is an open source C++ library commonly used for nonlinear minimization and large-scale optimization. Pyceres allows users to use Python language to model and solve optimization problems. Writing the BA optimization process in Pyceres is mainly divided into the following steps:

[0154] Define the optimization problem: including the objective function, optimization parameters and constraints. The objective function is the sum of squares of the projection error, the optimization parameters are the scan frame pose and the landmark point position, and the constraints are the feature point observation constraints, etc.

[0155] Build a Problem object: Use Pyceres to build a Problem object to represent the optimization problem. When building a Problem object, add the optimization parameter values ​​and constraint information at the same time.

[0156] Define the cost function: Define CostFunction for the observation data in scan frames or trajectories to calculate the projection error. This example uses the AutoDiffCostFunction class to define the Cost Function and add it to the Problem object.

[0157] Set the solver parameters. Set the solver parameters such as the number of iterations and gradient calculation method.

[0158] Solve the optimization problem: call the Solver function to optimize the Problem object. This example uses the Minimizer class to optimize and output the optimization results of the pose and landmark parameters.

[0159] Registration results of three test sets: The experiment in this example demonstrates the advantages of the incremental framework in terms of accuracy, robustness, and completeness. The test results of the method in this paper and the comparison method on the three datasets of 3D(Lo)Match, ScanNet and ETH are compared in detail. The test results are shown in Table 2, Table 3 and Table 4 respectively. Fig.18The reconstruction results of some scenes of the three datasets are visualized. (a) to (c) are the reconstruction results of 3DMatch-scan3 (55 frames), (66 frames) and (60 frames), (d) and (h) are the reconstruction results of ETH-wood_autumn (32 frames) and ETH-gazbo_winter (31 frames), (e) to (g) are the reconstruction results of ScanNet-0588_02 (30 frames), ScanNet0030_02 (30 frames) and ScanNet0575_02 (30 frames) (60 frames), respectively.

[0160] Accuracy: Table 2 shows the registration recall results of 3D(Lo)Match. The experiment used three point cloud registration algorithms, SpinNet, YOHO, and GeoTransformer, and tested the multi-group registration effects under the conditions of fully connected scene graph and sparse scene graph initialization. The results show that IncreMVR has better recall performance on both 3DMatch and 3DLoMatch datasets. Table 3 shows the rotation error R of the ScanNet dataset. e and translation error t e As a result, IncreMVR achieves the lowest average error and median error in rotation and offset. Table 4 shows the registration recall results of the ETH dataset, where the registration recall of IncreMVR is almost 99%. In addition, it is worth noting that IncreMVR shows good compatibility and can obtain the highest registration recall under both detector-based and detector-free point cloud registration methods. Fig.19 The image shows the alignment results of IncreMVR and the most advanced global alignment method SGHR for two sample scenes in the 3DMatch dataset, as well as the detailed comparison between the three and the true value. (a) is the result of SGHR, (b) is the result of IncreMVR, and (c) is the true value. Compared with SGHR, IncreMVR has higher geometric accuracy on the two samples, without obvious stratification or misalignment.

[0161] Table 2

[0162]

[0163]

[0164] Robustness: This example tests the registration effects of three datasets under a fully connected scene graph (with many wrong connections) (the "fully connected" part in Table 2, Table 3, and Table 4) and the registration effect of a low-overlap 3DLoMatch dataset (the "3DL" part in Table 2). IncreMVR greatly surpasses the comparison methods in terms of indicators, confirming the robustness of this method. The reason for this is mainly due to the local optimization mechanism unique to the incremental method, which can detect abnormal correspondences, so that the scene can be reconstructed using more accurate geometric constraints. In contrast, the global method is more sensitive to registration initialization, so it has limited adaptability to noisy or complex data, and produces suboptimal results.

[0165] Completeness: For scenes with poor scene graph initialization, this sample's incremental processing mechanism tends to divide the scene graph into multiple sub-connected graphs for independent reconstruction. Under such settings, this sample reports four cases with different initialization conditions, such as Fig. 20 As shown, (a) is the result of SGHR, (b) is the result of IncreMVR, and (c) is the true value. The first two rows of cases have higher accuracy in the retrieval stage (84%, 73%), while the last two rows have lower accuracy (44%, 55%). The global method (SGHR) based on the global consistency principle will lead to registration failure. In contrast, the method in this paper splits the scene graph and successfully constructs multiple sub-scenes.

[0166] Table 3

[0167]

[0168]

[0169] Table 4

[0170]

[0171] Ablation experiment analysis: Analyze the effects of cluster estimation, bundle adjustment and trajectory optimization modules. All analyses are performed on the 3D (L0) Match dataset and GeoTransformer is used as the two-group point cloud registration method. The ablation experiment results are shown in Table 5.

[0172] Table 4

[0173]

[0174] Cluster estimation: In this experiment, the clustering module is removed to evaluate the performance difference between the clustering algorithm and the point cloud registration process of the next frame. As shown in the first row of Table 5, removing the pose clustering will lead to a significant performance drop. Furthermore, this experiment takes a scene in the 3DMatch dataset as an example, and compares the number of corresponding relationships and the absolute pose difference of the scanned frame. The absolute pose difference (ΔR diff , t diff ):

[0175]

[0176] Among them, (R e + , t e + )、(R e - , t e - ) represent the absolute pose errors with and without clustering modules, R e ,t e Calculation is shown in equation (17) and equation (18). If ΔR diff >0 or Δt diff >0, which means that the clustering method makes the rotation or offset error smaller and the estimation more accurate, otherwise it becomes worse. The test results are as follows Fig.21 As shown, (a) is a bar graph of the number of correspondences involved in the registration of the i-th frame, and (b) is a scatter plot of the absolute pose differences of the scanned frames.

[0177] The results show that the number of correspondences obtained for each frame in the incremental registration process is statistically significant. The clustering method has more correspondences than the original method, and the pose differences of most scanned frames are distributed in the first quadrant, that is, ΔR diff >0 and Δt diff >0, the clustering method has higher pose accuracy and smaller drift than the original method. Therefore, adding the clustering method has two advantages in pose solution: first, decoupling the multi-view point cloud constraint relationship, constructing multiple feature point-landmark point correspondence constraints, so that multiple pose estimation results can be estimated and the scattered outliers can be eliminated using the consistency principle. Second, after eliminating the outliers, the remaining correspondences are aggregated and the final pose is solved, which is more advantageous than the original method in terms of the number of correspondences.

[0178] BA optimization: This experiment tests the method without adding BA optimization. The results are shown in the second row of Table 5, indicating that the absence of BA optimization will lead to performance degradation. In the incremental registration process of the scene, each aggregated landmark point will accumulate projection errors as the trajectory continues. BA effectively reduces the average projection error of the landmark points by iteratively adjusting the absolute pose and the landmark point position, such as Fig. 22(b) as shown. Fig. 22 (a) shows that the projection error increases more slowly after adding BA optimization than without BA optimization. Therefore, the experimental results show that the incremental registration framework with BA optimization has better stability and can better cope with data noise and incompleteness.

[0179] Trajectory optimization: Finally, this experiment verifies the effect of the feature trajectory optimization module. The experimental results are shown in the third row of Table 5. The trajectory optimization module improves the positioning accuracy of landmark points and reduces the posture error. The experiment shows that the spatial position accuracy of feature points has a significant impact on the pose estimation results. After optimization using the trajectory optimization module, the spatial position accuracy of feature points is improved. The model is significantly improved in accuracy through aggregation update and a global BA optimization.

[0180] The above three ablation experiments respectively evaluated the impact of different key modules in the multi-group point cloud registration task on the pose estimation and landmark point cloud accuracy, including the next frame point cloud clustering estimation method, BA optimization and feature trajectory optimization. Through the above experiments, it is verified that the three modules proposed in this paper play an important role in the multi-group point cloud registration task.

[0181] In one embodiment, Fig.23 As shown, there is also provided an incremental multi-group point cloud registration device 100, including a scene construction module 11, a trajectory constraint module 13, a landmark module 15 and an absolute pose module 27. Among them, the scene construction module 11 is used to perform correlation search on the unordered point cloud set to construct a sparse scene graph carrying the relative pose relationship of the point cloud; the correlation search includes global retrieval of point clouds, pairwise registration of point clouds and geometric verification. The trajectory constraint module 13 is used to use the feature point correspondence provided by the sparse scene graph to incrementally construct feature trajectory constraints from the feature point level; the feature trajectory is used to record the matching relationship between each feature point in different frames. The landmark module 15 is used to fuse the feature points in the feature trajectory through aggregation operations to obtain the landmark points in the global coordinate system. The absolute pose module 27 is used to construct a 3D-3D correspondence between the scan frame and the landmark points according to the feature trajectory constraints, and obtain the absolute pose of the scan frame by using singular value decomposition-random consistency sampling.

[0182] The above-mentioned incremental multi-group point cloud registration device 100, in order to solve the problem of point cloud disorder, establishes a sparse scene graph through retrieval and geometric verification, and constructs the potential overlapping relationship of point clouds; then performs point cloud registration processing through incremental registration steps such as initial point cloud selection and registration, iterative selection and registration of the next frame, trajectory creation and continuation, and bundle adjustment. Considering the nonlinear coupling relationship between the poses of multiple groups of point clouds, multiple error control algorithms and optimization strategies are introduced into the framework of the point cloud registration processing, which effectively takes into account the global consistency of the overall point cloud and the processing of outliers in the local area, which can effectively improve the accuracy, robustness and integrity of the registration, and greatly improve the registration performance.

[0183] In one embodiment, the above-mentioned incremental multi-group point cloud registration device may also include: a block feature module, which is used to extract a fine-grained feature map of the block corresponding to each feature point of the feature trajectory; a confidence calculation module, which is used to use the optimal transmission algorithm to calculate the confidence matrix between the reference block and the fine-grained feature map of each query block respectively; a trajectory optimization module, which is used to optimize and adjust the position of the feature points in the feature trajectory according to the confidence matrix to obtain an optimized feature trajectory; a registration adjustment module, which is used to send the optimized feature trajectory into the global BA algorithm to adjust the absolute pose of the scanning frame and the landmark points to obtain an optimized registration result.

[0184] It can be understood that the explanation of each feature in each embodiment of the above-mentioned incremental multi-group point cloud registration device 100 can be understood in the same way as the explanation of the corresponding features of each embodiment of the above-mentioned incremental multi-group point cloud registration method, and will not be repeated here. Each module in the above-mentioned incremental multi-group point cloud registration device 100 can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of a device with data processing functions in the form of hardware, or can be stored in the memory of the aforementioned device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. The aforementioned device can be but not limited to various types of image data calculation and processing devices already available in the art.

[0185] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps or sub-steps in any embodiment of the above-mentioned incremental multi-group point cloud registration method are implemented.

[0186] It can be understood that the above-mentioned computer device, in addition to the memory and processor mentioned above, may also include other software and hardware components not listed in this specification, which can be determined according to the model of the specific image data processing device in different application scenarios, and this specification will not list them one by one in detail.

[0187] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (RambusDRAM, referred to as RDRAM) and interface dynamic random access memory (DRDRAM).

[0188] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, all of which belong to the protection scope of the present application. Therefore, the protection scope of the patent of this application shall be based on the attached claims.

Claims

1. An incremental multi-group point cloud registration method, characterized in that: Includes steps: Performing correlation search on the unordered point cloud set to construct a sparse scene graph carrying the relative pose relationship of the point clouds; the correlation search includes global retrieval of point clouds, pairwise registration of point clouds, and geometric verification; Using the correspondence between feature points provided by the sparse scene graph, feature trajectory constraints are incrementally constructed from the feature point level; The feature track is used to record the matching relationship between each feature point in different frames; The feature points in the feature trajectory are fused by aggregation operation to obtain landmark points in the global coordinate system; A 3D-3D correspondence between the scan frame and the landmark points is constructed according to the characteristic trajectory constraint, and the absolute position and posture of the scan frame is obtained by using singular value decomposition-random consistency sampling.

2. The incremental multi-group point cloud registration method according to claim 1, characterized in that: Also includes the steps: Extracting a fine-grained feature map of a block corresponding to each feature point from the feature trajectory; The confidence matrix between the reference block and the fine-grained feature map of each query block is calculated using the optimal transfer algorithm; The positions of the feature points in the feature trajectory are optimized and adjusted according to the confidence matrix to obtain an optimized feature trajectory; The optimized feature trajectory is sent to the global BA algorithm to adjust the absolute pose of the scan frame and the landmark points to obtain the optimized registration result.

3. An incremental multi-group point cloud registration device, characterized in that: include: A scene construction module is used to perform correlation search on an unordered point cloud set to construct a sparse scene graph carrying the relative pose relationship of the point clouds; the correlation search includes global retrieval of point clouds, pairwise registration of point clouds, and geometric verification; The trajectory constraint module is used to incrementally construct feature trajectory constraints from the feature point level using the feature point correspondence provided by the sparse scene graph; The feature track is used to record the matching relationship between each feature point in different frames; A landmark module, used for fusing the feature points in the feature trajectory through aggregation operation to obtain landmark points in a global coordinate system; The absolute pose module is used to construct a 3D-3D correspondence between the scan frame and the landmark points according to the characteristic trajectory constraint, and obtain the absolute pose of the scan frame by using singular value decomposition-random consistency sampling.

4. The incremental multi-group point cloud registration device according to claim 3, characterized in that: Also includes: A block feature module, used to extract a fine-grained feature map of the block corresponding to each feature point from the feature trajectory; A confidence calculation module, used to calculate the confidence matrix between the reference block and the fine-grained feature map of each query block respectively using an optimal transmission algorithm; The trajectory optimization module is used to optimize and adjust the positions of feature points in the feature trajectory according to the confidence matrix to obtain an optimized feature trajectory; The registration adjustment module is used to send the optimized feature trajectory into the global BA algorithm to adjust the absolute pose of the scanning frame and the landmark points to obtain the optimized registration result.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the incremental multi-group point cloud registration method according to claim 1 or 2 are implemented.

Images