Virtual-real registration method based on triangulation graph sampling consistency point cloud matching

Abstract

The invention relates to a virtual-real registration method based on triangulation graph sampling consistency point cloud matching, belongs to the technical field of computer vision, and aims at solving the problems that in an existing virtual-real registration method, it is difficult to consider the assumed verification sampling subset size and iteration efficiency at the same time, and a large number of false matching peers are likely to be caused by equilateral length constraints. Comprising the steps of S1, performing triangulation on a triangular relationship formed by point clouds; s2, iteratively updating the affinity matrix of the given point cloud; s3, screening inline triangular constraints based on the updated affinity matrix to complete rough matching; and S4, ICP fine registration is carried out on the roughly matched point clouds, matching of the model point clouds and the assembling environment point clouds in the assembling process is completed, and efficient virtual-real matching can be realized through the determined position relation.

Description

TECHNICAL FIELD

[0001] The present application relates to a virtual-real registration method based on triangulation graph sampling consistent point cloud matching, and belongs to the technical field of computer vision. BACKGROUND

[0002] The augmented reality technology can realize information dimension reduction perception of the engine assembly regulation process through visualizing high-fidelity interaction, but it needs reliable virtual-real registration technology to realize precise interaction in the face of complex engine assembly environment.

[0003] To realize high-precision virtual-real registration, the mainstream method is to construct affine invariance constraint based on point graph, to realize virtual-real space model point cloud pose positioning by using space point cloud matching. At present, point cloud registration is mainly realized based on equal length constraint, mainly represented by RANSAC and its variants. RANSAC algorithm uses random sampling to find suitable key points to form a minimum registration subset to realize initial registration, but noise and outliers usually greatly reduce the probability of correct sampling. The variants of RANSAC are mainly committed to proposing better sampling strategies to eliminate outliers and improve the robustness of the algorithm. The application of equal length constraint effectively improves the adaptability of the algorithm, but the equal length constraint calculation amount will increase rapidly with the increase of the number of points. In order to improve the efficiency of rough matching and reduce the sampling times, new affine invariance constraint needs to be proposed. SUMMARY

[0004] The present application is to solve the problems that the existing virtual-real registration method cannot balance the size of the hypothesis verification sampling subset and the iteration efficiency, and the equal length constraint easily causes a large number of false matching pairs. Therefore, the present application proposes a virtual-real registration method based on triangulation graph sampling consistent point cloud matching.

[0005] The technical scheme adopted by the present application to solve the above problems is as follows:

[0006] Step 1: based on the Voronoi graph between point clouds, combined with the Delaunay triangulation method to carry out triangulation;

[0007] Step 2: set a threshold value λ to optimize the affinity matrix X of the point cloud A and the point cloud B after triangulation, and set the objective function of the binary assignment matrix Z based on the optimized affinity matrix X;

[0008] Step 3: iteratively update the affinity matrix X combined with the objective function of the binary assignment matrix Z;

[0009] Step 4: based on the iteratively updated affinity matrix X, screen the corresponding in-line points of the point cloud A and the point cloud B, and perform rigid transformation on the point cloud A, the point cloud B and the corresponding in-line points to complete rough matching;

[0010] Step 5: Perform ICP fine registration on the point cloud after rough matching to complete the matching of the model point cloud and the assembly environment point cloud during the assembly process, and complete the virtual-real matching.

[0011] Furthermore, step 2 specifically includes:

[0012] Set the threshold λ to optimize the affinity matrix X of point cloud A and point cloud B after triangulation, and filter out the triangular relationship {Skx,Sky}n 1 of the corresponding point cloud, and |Skx,Sky|≤λ, where the element in point cloud A is a i , the element in point cloud B is b i , with the maximum number of rows and columns n of the affinity matrix X AB =max(n A ,n B ) as the matrix size, the filled matrix elements are all 0, set the binary assignment matrix and the number of outliers m, combined with element a i and b j The affinity of is used to obtain the objective function;

[0013] The expression of the objective function is:

[0014]

[0015] In formula (1), X is the affinity matrix of point clouds A and B, m is the number of outliers, that is, the number of elements that need to be screened out, and c E For element a i and b j affinity.

[0016] Furthermore, the iterative objective equation in step 3 is:

[0017]

[0018] In each iterative target solving process, the binary assignment matrix Z needs to satisfy the requirement that the sum of the values ​​in the same row and column does not exceed 1, and the sum of all matrix elements is the number of outliers m. The number of iterations is obtained based on the number of outliers m. Each iterative solution obtains the position corresponding to the maximum value of the affinity matrix X, and resets the remaining positions in the row and column where the maximum value is located to zero to complete the update of the affinity matrix X.

[0019] Furthermore, step 4 specifically includes:

[0020] Based on the iteratively updated affinity matrix X, a high-similarity triangulated graph is screened out. The frequency statistics method is used to count the occurrence frequency of each point cloud in the high-similarity triangulated graph. The point cloud with a frequency higher than the preset value is selected as the inline point. The number of inline points m is greater than or equal to 3. The rigid transformation matrix [R * ,t* ], according to the rigid transformation matrix [R * ,t * ] Perform rigid transformation on point cloud A and point cloud B and the corresponding inline points to complete coarse matching.

[0021] The beneficial effects of the present invention are:

[0022] The present invention constructs a triangulated graph and imposes equal triangle constraints on the point cloud, so that the geometric information it contains is richer. During the coarse matching process, the present invention iteratively updates the affinity matrix of the point cloud. At the same time, the number of iterations is determined by the number of outliers. The outliers are generally set to be small, so that the computational complexity of each iteration is very low. The method of updating the affinity matrix has high computational efficiency, which overcomes the problem of the existing virtual-real alignment method that it is difficult to balance the size of the hypothesis verification sampling sub-set and the iteration efficiency. After the coarse matching is completed, traditional ICP fine alignment is performed, and efficient virtual-real alignment is achieved by using point cloud matching, which provides a technical basis for the virtual-real fusion intelligent assembly of aerospace engines. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of the virtual-real registration method based on triangulated graph sampling consistency point cloud matching provided by the present invention;

[0024] Figure 2 Schematic diagram of the Voronoi diagram and Delaunay triangulation results of the point cloud provided by the present invention;

[0025] Figure 3 This is a schematic diagram of the virtual-real registration results based on a real engine assembly scene provided by the present invention. DETAILED DESCRIPTION Specific implementation method 1

[0027] Combine Figure 1 and Figure 2 This embodiment is described as follows. Figure 1 As shown, the steps of the virtual-real registration method based on triangulated graph sampling consistency point cloud matching described in this embodiment include:

[0028] S1: triangulate the triangular relationship formed by the point cloud;

[0029] Given the correspondence between three pairs of point clouds (x i1 ,y j1 )、(x i2 ,y j2 ) and (x i3 ,y j3 ), if the corresponding relationships satisfy the non-collinear relationship, a triangular relationship can be constructed based on the three points, where x i1 ,y j1 ,xi2 ,y j2 ,x i3 ,y j3 Is a point containing three-dimensional coordinate information. The side length of the triangle is the distance between the points. x =||x i1 -x i2 ||2,b x =||x i2 -x i3 ||2,c x =||x i3 -x i1 ||2, semi-perimeter p x =(a x +b x +c x ) / 2; side length a y =||y i1 -y i2 ||2,b y =||y i2 -y i3 ||2,c y =||y i3 -y i1 ||2, semi-perimeter p y =(a y +b y +c y ) / 2. Then the corresponding relationship between the areas of a pair of triangles obtained by three pairs of points (S x ,S y )for:

[0030]

[0031] A set of inline triangle correspondences is related to three sets of inline point correspondences, and contains richer geometric information. Therefore, compared with the equal length constraint, the equal triangle area constraint is more robust.

[0032] However, since rigid transformations in three-dimensional space require at least three sets of non-collinear inline points to determine their identity, for a point cloud with n points, the equal-length constraint yields C2n=n(n-1) / 2 pairs of correspondences, while the equal-triangle-area constraint yields C3n=n(n-1)(n-2) / 6 pairs of correspondences. Coarse matching involves randomly selecting a certain number of points to obtain a minimum sample set. When the equal-triangle-area constraint is used, the number of triangular relationships formed between each point increases dramatically with the number of points. Although the equal-triangle-area constraint imposes a significant computational burden, many triangular relationships are unnecessary from a triangular topology perspective. Therefore, these relationships can be streamlined by studying the topological structure of the triangulated graph.

[0033] This embodiment uses the Crust method based on Voronoi diagram and Delaunay to realize triangulation. Figure 2 As shown, triangulation simplifies the number of triangle relationship matching pairs (for example, for 64 sampling points, the original intrinsic triangle correlation number C364 = 41664 can be reduced to about 100. Because the triangulation calculation contains a certain degree of randomness, the number of triangle relationship matching pairs obtained from each calculation is not fixed). Then, a threshold λ is set to optimize the corresponding affinity matrix and select the triangle relationship corresponding to {Skx, Sky}n 1, satisfying |Skx, Sky| ≤ λ.

[0034] S2: Iteratively update the affinity matrix of a given point cloud;

[0035] Although for the subset c * As long as a set of inline triangle correspondences can satisfy the rigid transformation of 3D space, but because outliers may also have equal areas, the equal triangle area constraint cannot completely eliminate noise points. Therefore, to ensure robustness, several sets of inline triangle constraints are usually screened, and the subsets that do not meet the constraint number will be "downgraded" and not used. For obtaining the affinity matrix, this embodiment selects the following method:

[0036] S201: For the screening problem, this embodiment adopts a scoring mechanism description. Given two point clouds A and B (a i and b j represents the elements in the point clouds A and B, which refers to the triangular area constructed by the three points. In order to facilitate the solution, the number of rows and columns of the affinity matrix X is unified, with the maximum number of rows and columns n AB =max(n A ,n B ) as the matrix size, and the filled matrix elements are all 0. Let the binary assignment matrix (z ij is the i-th row and j-th column element of Z. If a pair of blocks matches, then z ij =1, otherwise z ij =0), the objective function can be defined as:

[0037]

[0038] In formula (2), X is the affinity matrix of point clouds A and B, m is the number of outliers, that is, the number of elements that need to be screened out, and c E For element a i and b j affinity.

[0039] S202: The binary assignment matrix Z must satisfy the requirement that the sum of the values ​​in the same row and column does not exceed 1, and the sum of all elements in the matrix is ​​the number of outliers m. This problem can be simplified to:

[0040]

[0041] During each iteration of the target solution, the binary assignment matrix Z must ensure that the sum of the values ​​in the same row and column does not exceed 1, and the sum of all matrix elements equals the number of outliers m. The number of iterations is determined based on the number of outliers m. Each iteration finds the position corresponding to the maximum value in the affinity matrix X, and the remaining positions in the row and column where the maximum value is found are reset to zero, completing the update of the affinity matrix X. Because the number of outliers is generally small and the computational complexity of each iteration is low, updating the affinity matrix is ​​computationally efficient.

[0042] S3: Filter the inline triangle constraints based on the updated affinity matrix to complete the rough matching;

[0043] To obtain the final inline point, it is necessary to find suitable matching points from the selected high-similarity triangulated graph. From the node graph, it can be seen that the more triangles a point participates in, the more critical its position is, and the more likely it is to be an inline point. This embodiment uses a frequency statistics method to count the frequency of each point in the high-similarity triangulated graph. The higher the frequency of its appearance, the more likely it is to be an inline point. At the same time, the number of inline points m is guaranteed to be greater than or equal to 3. The rigid transformation matrix [R * ,t * ], to perform rigid transformation on point cloud A and point cloud B and the corresponding inline points to complete the coarse matching.

[0044] During the rough matching process, the present invention iteratively updates the affinity matrix of the point cloud. At the same time, the number of iterations is determined by the number of outliers. The outlier value is generally set to be small, so that the computational complexity of each iteration is very low. The method of updating the affinity matrix has high computational efficiency, which overcomes the problem of the existing virtual-real registration method that it is difficult to balance the size of the hypothesis verification sampling dataset and the iteration efficiency.

[0045] S4: Perform ICP fine registration on the point cloud after rough matching to complete the matching of the model point cloud and the assembly environment point cloud during the assembly process. By determining the positional relationship, efficient virtual-real matching can be achieved.

[0046] Finding a completely correct match between two pairs of point clouds through coarse matching requires multiple iterations. Fine matching can refine the point cloud match after the coarse matching has achieved a close match, thereby reducing the number of coarse matching iterations. This embodiment uses traditional ICP fine registration after coarse matching to complete the matching of the model point cloud and the assembly environment point cloud during the assembly process. By determining the positional relationship, efficient virtual-real matching can be achieved. Specific implementation method 2

[0048] In order to verify the technical effect of the virtual-reality registration method of the present invention, this embodiment verifies the virtual-reality registration effect based on a real aerospace engine assembly scene, and matches the virtual engine rotor model and the actual engine model in spatial posture, so as to achieve real-time tracking of the real aerospace engine assembly process. The experiment is based on an augmented reality device, and mainly uses the depth sensor, RGB sensor and IMU sensor of the augmented reality device to obtain the depth map data stream and RGB map data stream, and obtain their orientation information. The method for obtaining the environmental point cloud is to perform an environmental scan through the StreamRecorder program or to perform a real-time spatial scan using the SpatialAwareness function. After obtaining the environmental point cloud and the virtual model point cloud, our virtual-reality registration method can be used to achieve the matching of the rotor's environmental point cloud and the model point cloud.

[0049] The virtual-real registration result is as follows Figure 3 As shown. In actual matching, this embodiment uses 50 RGB images and depth images to generate a model stream, and generates a point cloud through the vertices of the model stream. These data streams contain images obtained from multiple angles, including front views, side views, top views, etc., to ensure that the obtained environmental image contains a complete engine image. Since the error of objects scanned by augmented reality devices is at the millimeter level in large sizes, the requirement for the environment during matching is to avoid objects similar in size to the rotor to increase the accuracy of matching convergence. Our virtual-to-real registration method overcomes the high noise of the environmental point cloud (an inevitable drawback of the point cloud acquisition system of augmented reality devices), completes the matching task efficiently and accurately, and effectively ensures the reliability of the engine rotor augmented reality virtual-to-real registration function.

[0050] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the present profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement of the above embodiments made according to the technical essence of the present invention, within the spirit and principles of the present invention, without departing from the content of the technical solution of the present invention, shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A virtual-real registration method based on triangulated graph sampling consistency point cloud matching, characterized by: include: Step 1: Triangulate the point cloud based on the Voronoi diagram and Delaunay triangulation method; Step 2: Set the threshold λ to optimize the affinity matrix X of point cloud A and point cloud B after triangulation, and set the objective function of the binary assignment matrix Z based on the optimized affinity matrix X; Step 3: Iteratively update the affinity matrix X in combination with the objective function of the binary assignment matrix Z; Step 4: Based on the iteratively updated affinity matrix X, the corresponding inline points of point cloud A and point cloud B are screened, and rigid transformation is performed on point cloud A and point cloud B and the corresponding inline points to complete the rough matching; Step 5: Perform ICP fine registration on the point cloud after rough matching to complete the matching of the model point cloud and the assembly environment point cloud during the assembly process, and complete the virtual-real matching.

2. The virtual-real registration method based on triangulated graph sampling consistency point cloud matching according to claim 1, characterized in that: Step 2 specifically includes: Set the threshold λ to optimize the affinity matrix X of point cloud A and point cloud B after triangulation, and filter out the triangular relationship {Skx, Sky}n 1 of the corresponding point cloud, and |Skx, Sky|≤λ, where the element in point cloud A is a i , the element in point cloud B is b i , with the maximum number of rows and columns n of the affinity matrix X AB =max(n A ,n B ) as the matrix size, the filled matrix elements are all 0, set the binary assignment matrix and the number of outliers m, combined with element a i and b j The affinity of is used to obtain the objective function; The expression of the objective function is: In formula (1), X is the affinity matrix of point clouds A and B, m is the number of outliers, that is, the number of elements that need to be screened out, and c E For element a i and b j affinity.

3. The virtual-real registration method based on triangulated graph sampling consistency point cloud matching according to claim 1, characterized in that: The objective equation for the iteration in step 3 is: In each iterative target solving process, the binary assignment matrix Z needs to satisfy the requirement that the sum of the values ​​in the same row and column does not exceed 1, and the sum of all matrix elements is the number of outliers m. The number of iterations is obtained based on the number of outliers m. Each iterative solution obtains the position corresponding to the maximum value of the affinity matrix X, and resets the remaining positions in the row and column where the maximum value is located to zero to complete the update of the affinity matrix X.

4. The virtual-real registration method based on triangulated graph sampling consistency point cloud matching according to claim 1, characterized in that: Step 4 specifically includes: Based on the iteratively updated affinity matrix X, a high-similarity triangulated graph is screened out. The frequency statistics method is used to count the occurrence frequency of each point cloud in the high-similarity triangulated graph. The point cloud with a frequency higher than the preset value is selected as the inline point. The number of inline points m is greater than or equal to 3. The rigid transformation matrix [R * ,t * ], according to the rigid transformation matrix [R * ,t * ] Perform rigid transformation on point cloud A and point cloud B and the corresponding inline points to complete coarse matching.

Images