A 3D registration and reconstruction method based on multi-domain and multi-dimensional feature maps

Abstract

The present invention discloses a three-dimensional registration and reconstruction method based on multi-domain multi-dimensional feature maps. First, a specified number of key points are randomly selected from the scene point cloud and the model point cloud. Then, each key point is traversed, and the corresponding multi-domain multi-dimensional feature maps are calculated as local feature descriptors. Based on the multi-domain multi-dimensional feature maps, the correspondence between key points is obtained, and the key point pairs are sorted according to the matching degree. In each iteration, a compatible triple is selected in order to generate a hypothesis, and it is checked whether the hypothesis can correctly register the model into the scene. The present invention has a short calculation time and a low time cost.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to a three-dimensional registration and reconstruction method based on multi-domain and multi-dimensional feature maps. Background Art

[0002] Three-dimensional registration and reconstruction is a process of aligning multiple two-dimensional images or point cloud data and fusing them into a complete three-dimensional model or image through algorithms. In the field of three-dimensional registration and reconstruction, there are some challenges. One of the challenges is the need for object segmentation when encoding objects in a point cloud scene. However, when an object is occluded, the segmentation process will result in severe information loss. Currently, the feature extraction methods mainly include local feature-based and global feature-based methods. In this case, using global descriptors will result in a large amount of information loss, while local descriptors only analyze and process the local features of the point cloud, so they are more suitable for the case where the model is occluded by other objects.

[0003] The point cloud normal vector is the basis for calculating local descriptors. When calculating the point cloud normal vector, choosing an appropriate radius size has a significant impact on the result. If the radius is too large, the calculated normal will contain information from distant point clouds, resulting in poor stability of the normal. When the radius is too small, the calculated normal only contains information from extremely close points and is easily affected by noise and outliers, resulting in a decrease in the accuracy of the normal. Therefore, being able to obtain a stable and accurate normal representation in various complex environments will significantly improve the performance of subsequent tasks and enhance the stability and accuracy of the final registration and reconstruction.

[0004] Another challenge is that there may be many outliers in the initial correspondence set during pose estimation. These outliers will lead to incorrect assumptions or incorrect correspondence relationships during the registration process. The existence of outliers will disrupt accurate matching and geometric constraints, making the estimated pose inaccurate or unstable. To solve this problem, robust estimation methods need to be adopted to exclude the interference of outliers and ensure the accuracy and consistency of the correspondence relationship. This will improve the accuracy and stability of registration, make the estimated pose more reliable, and further improve the accuracy and robustness of registration and reconstruction. Summary of the Invention

[0005] To overcome the deficiencies of the prior art, the present invention provides a three-dimensional registration and reconstruction method based on a multi-domain multi-dimensional feature map. First, a specified number of key points are randomly selected from the scene point cloud and the model point cloud; then, each key point is traversed, and the corresponding multi-domain multi-dimensional feature map is calculated as a local feature descriptor; according to the multi-domain multi-dimensional feature map, the corresponding relationship between the key points is obtained, and the key point pairs are sorted according to the matching degree; in each iteration, a compatible triple is sequentially selected to generate a hypothesis, and it is checked whether this hypothesis can correctly register the model into the scene. The present invention has a short calculation time and a low time cost.

[0006] The technical solution adopted by the present invention to solve its technical problems includes the following steps:

[0007] Step 1: Randomly select a specified number of key points from the scene point cloud and the model point cloud;

[0008] Step 2: Traverse each key point, and calculate the corresponding multi-domain multi-dimensional feature map as a local feature descriptor;

[0009] Step 3: According to the multi-domain multi-dimensional feature map, obtain the corresponding relationship between the key points, and sort the key point pairs according to the matching degree; sequentially select the K key point pairs with the highest matching degree from the sorted list from high to low; use these K key point pairs to construct a second-order compatibility matrix of size K×K; select 3 mutually compatible point pairs to form a compatible triple, calculate the sum of the degrees of the compatible triple, and sort the compatible triples according to the sum of the degrees, and select the T compatible triples with the highest sum of the degrees; the degree is the number of other point pairs that a point pair is compatible with;

[0010] Step 4: In each iteration, sequentially select a compatible triple to generate a hypothesis, and check whether this hypothesis can correctly register the model into the scene; if the model cannot be correctly registered into the scene in all iterations, then use the ICP algorithm to optimize the hypothesis with the largest overlap rate that is retained. After the optimization is completed, it is again determined whether the model can be correctly recognized in the scene.

[0011] Further, in the step 2, the process of calculating the multi-domain multi-dimensional feature map for each key point includes the following steps:

[0012] Step 2-1: Obtain the local point cloud block P of the area where the key point is located according to the coordinates of the key point and the size of the support radius l ;

[0013] Step 2-2: Calculate the normal vectors of different scales using different search radii;

[0014] Step 2-3: Calculate the multi-domain multi-dimensional features between the neighborhood point q and the key point p for normal vectors of different scales, and combine these features to form a multi-domain multi-dimensional feature map:

[0015]

[0016] where n i (v j ) is the magnitude of the j-th voxel value of the i-th multi-dimensional feature map, and |P l | represents the number of inlier points in P l .

[0017] Furthermore, in the said Step 3, the steps of selecting the T compatible triples with the largest degree are as follows:

[0018] Step 3-1: Use the Nearest Neighbor Similarity Ratio (NNSR) method to generate feature matches, and the score of each match is denoted as score:

[0019]

[0020] where f represents the feature map of the key point, f'1 represents the feature map with the highest similarity to the key point feature map, and f'2 represents the feature map with the second highest similarity to the key point feature map;

[0021] Step 3-2: Sort according to the scores, and select the K matches with the highest scores as the final feature match set;

[0022] Step 3-3: Initialize the set C = {c1, c2,..., c K} containing K matches, construct the first-order spatial compatibility matrix S, where S(i, j) being 1 represents that the match c i and the match c j are compatible, and S(i, j) being 0 represents incompatibility;

[0023] Step 3-4: Calculate the second-order spatial compatibility matrix S 2 ∈ R K×K :

[0024]

[0025] Step 3-5: Calculate the mutually compatible spatial compatible triples among the three matches, where the degree of the triple is defined as the sum of the degrees among the three matches, and select the T spatial compatible triples with the largest degree.

[0026] The beneficial effects of the present invention are as follows:

[0027] 1. Improve the feature representation ability: By calculating features in the spatial domain with different normal vector scales, more comprehensive and rich feature information is obtained. Compared with traditional methods, this method performs excellently in feature description.

[0028] 2. Strong robustness: This method reduces the influence of various interferences on point cloud feature extraction by effectively integrating feature information at different scales. Even in the case of various interferences in the target point cloud, this method can still have a high registration accuracy.

[0029] 3. High time efficiency: Through the second-order spatial compatibility matrix, this method significantly reduces the probability of outliers when generating hypotheses, reduces the number of iterations, and thus quickly calculates a high-quality rough matching result. Therefore, this method has a short calculation time and a low time cost. Brief Description of the Drawings

[0030] Figure 1 It is a flowchart of an embodiment of the present invention.

[0031] Figure 2 It is a calculation process of a multi-domain multi-dimensional feature map in an embodiment of the present invention.

[0032] Figure 3 It is a flowchart of guided sampling consistency based on a second-order graph in an embodiment of the present invention.

[0033] Figure 4 It is a schematic comparison diagram of the PRC performance between the MDMF of an embodiment of the present invention and other latest methods.

[0034] Figure 5 It is an effect diagram of multi-view point cloud registration and reconstruction in an embodiment of the present invention, (a) rabbit, (b) dragon. Detailed Embodiment

[0035] The present invention will be further described below with reference to the drawings and embodiments.

[0036] To address the problems of the prior art, the present invention proposes a three-dimensional registration and reconstruction method based on Multi-Domain Multi-dimensional Feature maps (MDMF). This method combines the spatial domain with different normal vector scales and the multi-dimensional geometric attributes of point pairs to obtain the three-dimensional spatial information of key points and generate multi-domain multi-dimensional feature maps. This method can comprehensively consider the local shape and geometric structure of the point cloud from multiple perspectives, thereby obtaining more comprehensive and rich feature information and better describing the local features of the point cloud. At the same time, by effectively combining different features, the redundancy between features can be reduced, further improving the representation ability of the features. In the coarse matching stage, Guided SAmple Consensus in Second-Order Graphs (GSAC-SOG), a variant of RANSAC, is introduced. By calculating the second-order spatial compatibility matrix and considering the mutual compatibility relationship between matches, a second-order spatial compatible triple is constructed, sorted, and sampled, thereby greatly reducing the influence of outliers. Through the application of the above methods, problems such as outliers, information loss, and accuracy in point cloud registration and reconstruction can be effectively solved, and the accuracy and stability of the registration result can be improved.

[0037] To achieve the above object, the present invention provides a solution to solve the related technical problems: a three-dimensional registration and reconstruction method based on multi-domain multi-dimensional feature maps, the steps of which are as follows:

[0038] S1: Randomly select a certain number of key points from the scene point cloud and the model point cloud.

[0039] S2: Traverse each key point and calculate the corresponding multi-domain multi-dimensional feature map as a local feature descriptor.

[0040] S3: According to feature matching, obtain the correspondence between key points, and sort the key point pairs according to the matching degree. Select the K key point pairs with the highest matching degree from the sorted list. Use these K key point pairs to construct a second-order compatibility matrix of size K×K, sort it according to the degree, and select the T compatible triples with the largest degree.

[0041] S4: In each iteration, select a compatible triple in order to generate a hypothesis, and check whether this hypothesis can correctly register the model into the scene. If the model cannot be correctly registered into the scene in all iterations, the ICP algorithm is used to optimize the hypothesis with the largest overlap rate that is retained. After optimization, determine again whether the model can be correctly recognized in the scene.

[0042] In addition, in S2, the process of each key point calculating the multi-domain multi-dimensional feature map includes the following steps:

[0043] S21: Obtain the local point cloud block P of the area where the key point is located according to the coordinates of the key point and the size of the support radius l .

[0044] S22: Calculate normal vectors of different scales using different search radii.

[0045] S23: For normal vectors of different scales, calculate the multi-domain multi-dimensional features between the neighborhood point q and the key point p.

[0046] Combine these features to form a multi-domain multi-dimensional feature map.

[0047]

[0048] where n i (v j ) is the size of the j-th voxel value of the i-th multi-dimensional feature map, and |P l | represents the number of internal points in P l .

[0049] In S3, the steps to select T compatibility triples with the largest degrees are as follows:

[0050] S31: Use the Nearest Neighbor Similarity Ratio (NNSR) method to generate feature matches. The score of each match is denoted as score:

[0051]

[0052] S32: Sort according to the scores and select the K matches with the highest scores as the final feature match set.

[0053] S33: Initialize the set C containing K matches as C = {c1, c2,..., c K}. Construct the first-order spatial compatibility matrix S, where S(i, j) = 1 indicates that the match c i and the match c j are compatible, and 0 indicates incompatibility.

[0054] S34: Calculate the second-order spatial compatibility matrix S 2 ∈ R K×K .

[0055]

[0056] S35: Calculate the mutually compatible spatial compatibility triples among the three matches, where the degree of the triple is defined as the sum of the degrees among the three matches, and select the T spatial compatibility triples with the largest degrees. Specific Embodiment:

[0058] The implementation of the technical solution of the present invention provides a three-dimensional registration and reconstruction algorithm based on a multi-domain and multi-dimensional feature map. Its flowchart is as Figure 1 shown, including reading model point cloud data and scene point cloud data, and randomly selecting key points; constructing descriptors based on the multi-domain and multi-dimensional feature map; generating guided sampling consistency of compatibility triples based on the second-order graph; hypothesis generation and hypothesis verification. The following combines specific examples to illustrate a three-dimensional registration and reconstruction algorithm based on a multi-domain and multi-dimensional feature map provided by the present invention.

[0059] (1) Read the model point cloud data and the scene point cloud data, and randomly select a certain proportion of the point cloud data as key points from them.

[0060] (2) Generate descriptors based on the multi-domain and multi-dimensional feature map for each key point. The specific steps are as follows (see Figure 2 ):

[0061] (2.1) According to the support radius of the key point, extract the local point cloud patch P in the area where the key point is located.

[0062] (2.3) Use three different search radii to calculate normal vectors at different scales around the key point.

[0063] (2.4) In the normal vector representations at different scales, calculate the multi-domain and multi-dimensional features between the key point p and the neighboring point q;

[0064] (2.5) The multi-domain and multi-dimensional features calculated from the normal vectors at each scale form a multi-dimensional feature map.

[0065] (2.6) Accumulate and fuse multiple multi-dimensional feature maps according to the corresponding positions, and perform normalization to obtain a multi-domain and multi-dimensional feature map.

[0066]

[0067] where n i (v j ) is the size of the j-th voxel value of the i-th multi-dimensional feature map, and |P l | represents the number of inliers in P l .

[0068] Fusing the three feature maps can effectively combine different features, reduce the influence of various noises, and thus improve the robustness of the algorithm.

[0069] (3) Generate a set of key point pairs according to feature matching, generate second-order spatial compatibility triples, and perform sorting. The specific steps are as follows (see Figure 3 ):

[0070] (3.1) Use the Nearest Neighbor Similarity Ratio (NNSR) method to perform feature matching on the key points in the model point cloud and the key points in the scene point cloud, and calculate the score score for each match:

[0071]

[0072] (3.2) According to the score score of each match, sort the matches and select the top K matches with the highest scores as the final feature matching set. Here, K is an experimental parameter, and in the present invention, the value of K is 100.

[0073] (3.3) Initialize the set C = {c1, c2,..., c K} that contains K matches.

[0074] (3.4) Construct a matrix D ∈ R K×K that contains the compatibility score information between the matches:

[0075]

[0076] where, are the source point and the target point in c i respectively; are the source point and the target point in c j respectively; t is a distance parameter used to control the tightness of the distance constraints.

[0077] In 3D vision, the compatibility between point pairs refers to the geometric relationship and consistency degree between two points in space. When there are certain geometric constraints between two point pairs and their attributes match each other, they can be considered compatible. These geometric constraints can include the distance between points, the consistency of normal vectors, the consistency of plane fitting, etc. In this technology, the geometric constraint selected is the distance between point pairs.

[0078] (3.5) Construct a first-order spatial-level compatibility matrix S ∈ R K×K , where S(i, j) represents whether the match c i and the match c j are compatible:

[0079]

[0080] where, is the threshold for judging whether two matches are compatible, and in the experiment, th = 0.9 is selected.

[0081] (3.6) Based on the first-order spatial compatibility matrix S ∈ R K×K , calculate the second-order spatial compatibility matrix S 2 ∈ R K×K .

[0082]

[0083] (3.7) Calculate the degree of each match, that is, how many matches each match is compatible with.

[0084] (3.8) Calculate the spatially compatible triples and their corresponding degrees. If three matches are compatible with each other, they form a spatially compatible triple, and the degree of the triple is the sum of the degrees among the three matches.

[0085] (3.9) Select the T spatially compatible triples with the largest degrees as the final set of spatially compatible triples. Here, T is an experimental parameter, and the value of T in the present invention is 30.

[0086] (4) The process of hypothesis generation and evaluation for second-order spatial compatibility triples is iterated as follows:

[0087] (4.1) At each iteration, select a compatible triple in sequence. Using this compatible triple, calculate the rigid transformation between the model point cloud and the scene point cloud through the following formula:

[0088]

[0089] (4.2) Determine whether the generated hypothesis can successfully register the model point cloud to the scene point cloud.

[0090] (4.3) If the registration is successful, stop the iteration. Otherwise, continue the next iteration until the iteration ends.

[0091] (4.4) During each iteration, judge the overlap rate of the generated hypothesis. Retain the hypothesis with the largest overlap rate as the best hypothesis.

[0092] (4.5) If the registration is still not successful after the iteration ends, use the ICP algorithm to refine the saved best hypothesis. Then, judge again according to the determination condition whether the model is correctly registered to the scene. If the registration is still not successful after ICP refinement, then the model will be determined as registration failure.

[0093] The algorithm of the present invention is applied to actual point cloud registration and reconstruction. In Figure 4 , the MDMF is compared with other latest technologies in terms of PRC performance. The results show that even in the presence of noise and reduced mesh resolution, the MDMF still has good performance and robustness. In Figure 5 , the effect diagrams of point cloud registration and reconstruction under multiple perspectives are shown. In Table 1, the registration accuracies of GSAC-SOG and RANSAC under different numbers of iterations are compared. It can be seen that at low numbers of iterations, the accuracy of the GSAC-SOG algorithm is still higher than that of the RANSAC algorithm at high numbers of iterations.

[0094] Comparison of Registration Accuracy between GSAC-SOG and RANSAC at Different Iteration Times in Table 1

[0095]

[0096] Among them, PRC represents the Precision-Recall Curve, that is, the precision-recall curve. Recall refers to the proportion of positive examples correctly identified by the model among all true positive examples; Precision refers to the proportion of actual positive examples among all samples identified as positive examples by the model. By adjusting the threshold of the classification model, points with different Recall and Precision values can be obtained, and then the PRC curve can be drawn to evaluate the performance of the model at different recall rates and false positive rates.

Claims

1. A three-dimensional registration and reconstruction method based on multi-domain multi-dimensional feature maps, characterized in that The method includes the following steps: Step 1: Randomly select a specified number of key points from the scene point cloud and the model point cloud; Step 2: Traverse each key point and calculate the corresponding multi-domain multi-dimensional feature map as the local feature descriptor; Step 3: According to the multi-domain multi-dimensional feature map, obtain the correspondence between key points, and sort the key point pairs according to the matching degree; sequentially select the K key point pairs with the highest matching degree from the sorted list from high to low; use these K key point pairs to construct a second-order compatibility matrix of size K×K; select 3 mutually compatible point pairs to form a compatible triple, calculate the sum of the degrees of the compatible triples, and sort the compatible triples according to the sum of the degrees, and select the T compatible triples with the highest sum of the degrees; the degree is the number of other point pairs that a point pair is compatible with; Step 4: In each iteration, select a compatible triple in order to generate a hypothesis, and check whether the hypothesis can correctly register the model into the scene; If the model cannot be correctly registered into the scene in all iterations, use the ICP algorithm to optimize the hypothesis with the largest overlap rate that is retained. After the optimization is completed, determine again whether the model can be correctly recognized in the scene.

2. The three-dimensional registration and reconstruction method based on multi-domain multi-dimensional feature maps according to claim 1, wherein In the step 2, the process of calculating the multi-domain multi-dimensional feature map for each key point includes the following steps: Step 2-1: Obtain the local point cloud patch P of the region where the key point is located according to the coordinates of the key point and the size of the support radius l ; Step 2-2: Calculate normal vectors of different scales using different search radii; Step 2-3: For normal vectors of different scales, calculate the multi-domain multi-dimensional features between the neighborhood point q and the key point p, and combine these features to form a multi-domain multi-dimensional feature map: where n i (v j ) is the magnitude of the j-th voxel value of the i-th multi-dimensional feature map, |P l | represents the number of inliers in P l .

3. The three-dimensional registration and reconstruction method based on a multi-domain multi-dimensional feature map according to claim 2, wherein, In the step 3, the step of selecting the T compatible triples with the largest degree is as follows: Step 3-1: Use the Nearest Neighbor Similarity Ratio (NNSR) method to generate feature matches, and the score of each match is denoted as score: where f represents the feature map of the key point, f′1 represents the feature map with the highest similarity to the key point feature map, and f′2 represents the feature map with the second highest similarity to the key point feature map; Step 3-2: Sort according to the scores, and select the K matches with the highest scores as the final feature match set; Step 3-3: Initialize the set \(C = \{c_1, c_2, \ldots, c_K\}\) containing \(K\) matches, and construct the first-order spatial compatibility matrix \(S\). \(S(i, j)=1\) indicates that match \(c_i\) and match \(c_j\) are compatible, and \(S(i, j)=0\) indicates incompatibility. K}, construct the first-order spatial compatibility matrix \(S\), where \(S(i, j) = 1\) represents that match \(c_i\) i and match \(c_j\) j are compatible, and \(S(i, j) = 0\) represents incompatibility; Step 3-4: Calculate the second-order spatial compatibility matrix S through the first-order spatial compatibility matrix 2 ∈R K×K : Step 3-5: Calculate the mutually compatible spatial compatible triples among the three matches, where the degree of the triple is defined as the sum of the degrees among the three matches, and select the T spatial compatible triples with the largest degree.

Images