A multi-view three-dimensional point cloud reconstruction method and device

Abstract

This invention discloses a multi-view 3D point cloud reconstruction method and apparatus. The method includes the following steps: S01. Acquiring point cloud data of multiple pre-arranged marker points from different viewpoints; S02. Matching data frames from two adjacent viewpoints to find corresponding points, wherein the distance between any two points in the data frames of the two viewpoints is first calculated, and preliminary matching is performed based on the similarity of the distance relationship to find all candidate corresponding points and neighboring points. If a marker point is matched with more than two marker points simultaneously, a secondary matching is performed based on the similarity of the angle relationship; S03. Obtaining the coordinate system transformation relationship based on the matching relationship of corresponding points between adjacent viewpoint data frames; S04. After converting all data to the same viewpoint coordinate system, performing 3D model reconstruction. This invention has the advantages of simple implementation, high reconstruction efficiency and accuracy, wide applicability, and strong flexibility.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and in particular to a method and apparatus for multi-view three-dimensional point cloud reconstruction. Background Technology

[0002] In the field of 3D vision, 3D point cloud reconstruction technology is an important technique for restoring surface information of targets, and it has a wide range of applications, such as quality inspection and reverse engineering. Before the advent of 3D reconstruction technology, obtaining the dimensions of a workpiece or mold, or acquiring a target model, required very complex manual measurement or model design. However, using 3D reconstruction technology, the 3D data of the target can be quickly obtained for reverse engineering or to rapidly acquire the target model, which can greatly reduce economic and time costs. Furthermore, 3D reconstruction technology enables multi-view, flexible, and high-precision inspection, solving the problem that traditional methods cannot achieve comprehensive and diverse inspections.

[0003] For the 3D reconstruction of a target, existing technologies typically employ a target feature point extraction and matching method. This involves scanning the target and extracting target feature points that reflect its characteristics, then matching these feature points to achieve 3D reconstruction. However, this 3D reconstruction method is complex and relies heavily on the extraction of the target's own feature points. If the target features are not obvious or lack easily matching features, the target feature points cannot be extracted, making high-precision point cloud stitching impossible, and thus preventing the completion of high-precision 3D reconstruction. Summary of the Invention

[0004] The technical problem to be solved by this invention is: in view of the technical problems existing in the prior art, this invention provides a multi-view three-dimensional point cloud reconstruction method and device that is simple to implement, has high reconstruction efficiency and accuracy, wide applicability and strong flexibility.

[0005] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0006] A multi-view 3D point cloud reconstruction method, comprising the following steps:

[0007] S01. Collect point cloud data of multiple pre-arranged marker points from different perspectives. The marker points are arranged on the target to be measured or within a specified range near the target to be measured. Data collected from adjacent perspectives contain points with the same name, and the points with the same name are the same marker points.

[0008] S02. Match the data frames of two adjacent viewpoints to find corresponding points. First, calculate the distance between any two points in the data frames of the two viewpoints. Perform preliminary matching based on the similarity of the spatial distance between each point in the data frames of the two viewpoints and the remaining points in the data frames. Find all candidate corresponding points and their neighboring points to form a candidate point set. If there is a marker point in the candidate point set that is paired with more than two marker points, perform secondary matching based on the similarity of the angular relationship between each point in the candidate set corresponding to the two viewpoints and the remaining points in the candidate set. Obtain the correct matching marker point pair and output the final matching result.

[0009] S03. Based on the matching relationship of corresponding points between data frames from adjacent viewpoints, obtain the coordinate system transformation relationship between data frames from adjacent viewpoints;

[0010] S04. After converting all the collected data to the same viewpoint coordinate system according to the coordinate system transformation relationship, perform three-dimensional model reconstruction.

[0011] Furthermore, the specific steps in step S02 for preliminary matching based on the similarity of the spatial distance between each point in the two viewpoint data frames and the remaining points within the same data frame include:

[0012] S201. Calculate the distance between any two points within the data frames of the two viewpoints, and obtain the first distance matrix A and the second distance matrix B respectively. The first distance matrix A is calculated from the data frames of one viewpoint, and the second distance matrix B is calculated from the data frames of the other viewpoint.

[0013] S202. Calculate the difference between each distance point in the first distance matrix A and the second distance matrix B, and determine the similarity based on the magnitude of the difference;

[0014] S203. Find all distance points in the first distance matrix A and the second distance matrix B whose difference between distance points is less than a preset threshold;

[0015] S204. Based on all the distance points found in the first distance matrix A, candidate corresponding points and their neighboring points are determined to form a first candidate set N1. Based on the distance points found in the second distance matrix B, candidate marker points and their neighboring points are determined to form a second candidate set M1.

[0016] Furthermore, the elements of each row in the first distance matrix A and the second distance matrix B correspond to the first... i Spatial distance between each point and the other points i =1, 2, ..., m ,m The number of points in the data frame; in step S204, if it is determined that the number of elements in the k-th row of the first distance matrix A and the p-th row of the second distance matrix B with a difference less than a preset threshold exceeds a preset threshold, then the points corresponding to the k-th row and the p-th row are determined to be candidate points of the same name. Among the distance points with a difference less than the preset threshold, the points whose distances are calculated with the candidate points of the same name are the neighborhood points of the candidate points of the same name. k and p correspond to any number of rows in distance matrices A and B, respectively.

[0017] Furthermore, the step S02, which involves secondary matching based on the similarity of the angular magnitude relationships between each point in the candidate set obtained from the two viewpoint data frames and the remaining points in the candidate set, includes:

[0018] S211. Calculate the angle between any two element points in the first candidate set N1 to form a first angle matrix, and calculate the angle between any two element points in the second candidate set M1 to form a second angle matrix;

[0019] S212. If the angle value of the first target candidate corresponding point and the angle value of its corresponding neighboring point in the first angle matrix have the greatest similarity to the angle value of the second target candidate corresponding point and the angle value of its corresponding neighboring point in the second angle matrix, then the first target candidate corresponding point and the second target candidate corresponding point are determined to be a successfully matched pair of marker points.

[0020] Furthermore, in step S212, the difference between each angle value in the first angle matrix and each angle value in the second angle matrix is ​​calculated. If the number of difference points between the angle value of the first target candidate corresponding point and the angle value of its corresponding neighboring point and the angle value of the second target candidate corresponding point and the angle value of its corresponding neighboring point is less than a preset threshold is the largest, then the angle value similarity is determined to be the largest.

[0021] Furthermore, in step S03, if the number of currently obtained homonyms is greater than a preset threshold, the SVD singular value decomposition method is used to solve the transformation matrix between adjacent viewpoint data frames, so as to use the solved transformation matrix to complete the splicing of adjacent two frames of data; if the solved transformation matrix cannot complete the splicing of adjacent two frames of data, the currently obtained homonyms are sorted according to the number of their neighboring points, and the homonym with the smallest neighboring point is removed, and the SVD singular value decomposition method is used again to solve the problem and splice the adjacent two frames of data until the splicing is finally completed.

[0022] Furthermore, after step S03 and before step S04, the converted data is further optimized globally to minimize the global error e. The global optimization adopts a pose graph-based optimization method and uses loop closure detection to establish strong constraints across multiple frames to optimize camera extrinsic parameters. The loop closure detection is performed by acquiring images in a loop-based manner; or by using a bundle adjustment optimization method and loop closure detection to optimize camera extrinsic parameters.

[0023] Furthermore, the global error e is calculated using the following formula:

[0024]

[0025] Among them, M m M n P represents the transformation matrices from the m-th frame and the n-th frame to the first frame, respectively. mi P ni These represent the i-th pair of corresponding points in the m-th and n-th frames, respectively.

[0026] A multi-view 3D point cloud reconstruction device, comprising:

[0027] The acquisition module is used to acquire point cloud data of multiple pre-arranged marker points from different perspectives. The marker points are arranged on the target to be measured or within a specified range near the target to be measured. Data acquired from adjacent perspectives contain points with the same name, which are the same marker points.

[0028] The matching and search module is used to match data frames from two adjacent viewpoints to find corresponding points. First, the distance between any two points in the data frames of the two viewpoints is calculated. Preliminary matching is performed based on the similarity of the spatial distance between each point in the data frames of the two viewpoints and the remaining points in the data frames. All candidate corresponding points and their neighboring points are found to form a candidate point set. If a marker point in the candidate point set is paired with two or more marker points, a secondary matching is performed based on the similarity of the angular relationship between each point in the candidate set corresponding to the two viewpoints and the remaining points in the candidate set. The correct matching marker point pairs are obtained, and the final matching result is output.

[0029] The transformation relationship determination module is used to obtain the coordinate system transformation relationship between adjacent view data frames based on the matching relationship of corresponding points between adjacent view data frames;

[0030] The 3D reconstruction module is used to convert all the collected data to the same viewpoint coordinate system according to the coordinate system transformation relationship, and then perform 3D model reconstruction.

[0031] A computer device includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the method described above.

[0032] Compared with the prior art, the advantages of the present invention are as follows:

[0033] 1. This invention pre-arranges easily identifiable and matching marker points on or near the target to be measured, collects each marker point from different perspectives, and indirectly obtains the coordinate transformation relationship between adjacent perspective data frames based on the matching relationship of common marker points between data frames of two adjacent perspectives. Finally, it obtains the coordinate transformation relationship between each pair of perspectives under all perspectives, and then uses this coordinate transformation relationship to register all data under a unified coordinate system, thereby achieving fast and high-precision point cloud stitching and completing efficient and high-precision 3D model reconstruction. It can be flexibly applied to various scenarios and is not limited by the shape, size, or whether the features of the target to be measured are easily visible.

[0034] 2. In the process of finding common landmark points, this invention first performs coarse matching based on distance relationships to find candidate homonyms in adjacent viewpoint data frames whose spatial distance relationship with other points in the same data frame meets preset requirements. This can quickly and fuzzily find all possible homonyms. If there are ambiguities such as one-to-two or one-to-many or mismatches, a second matching is performed based on the angular relationship between the candidate homonyms and other points. The correct matching relationship is finally determined based on the angular similarity. This invention can make full use of the neighborhood relationship of spatial points to quickly and accurately determine the final landmark point pair relationship, solve the ambiguity problem of one-to-two or one-to-many, effectively improve the matching accuracy of common landmark points and reduce the mismatch rate.

[0035] 3. Furthermore, this invention employs different optimization methods based on different applicable scenarios. For cameras that are not suitable for intrinsic parameter optimization, a pose graph-based optimization and loop closure detection method is used to improve the accuracy of the transformation matrix and optimize the global stitching effect. For cameras that are suitable for intrinsic parameter optimization, a light speed adjustment and loop closure detection method is used to improve the accuracy of marker recognition. This invention can be flexibly applied to different application scenarios to achieve global optimization. Attached Figure Description

[0036] Figure 1 This is a schematic diagram illustrating the implementation process of the multi-view 3D point cloud reconstruction method in this embodiment.

[0037] Figure 2 This is a schematic diagram illustrating a mismatch situation in a specific application embodiment.

[0038] Figure 3 This is a schematic diagram illustrating the result of eliminating mismatches using the method of the present invention in a specific application embodiment.

[0039] Figure 4 This is a schematic diagram comparing the effects before and after optimization using the method of the present invention in a specific application embodiment.

[0040] Figure 5 This is a detailed flowchart illustrating the process of multi-view 3D point cloud reconstruction in a specific application embodiment of the present invention.

[0041] Figure 6 This is a schematic diagram illustrating the implementation process of finding common landmarks in a specific application embodiment of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0043] As disclosed in this invention, unless the context clearly indicates otherwise, the terms "first," "second," and similar terms used in this invention disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, terms such as "comprising" or "including" mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, without excluding other elements or objects.

[0044] like Figure 1 As shown, the steps of the multi-view 3D point cloud reconstruction method in this embodiment include:

[0045] S01. Collect point cloud data of multiple pre-arranged marker points from different perspectives. The marker points are arranged on the target to be measured or within a specified range near the target to be measured. Data collected from adjacent perspectives contain points with the same name, and the points with the same name are the same marker points.

[0046] S02. Match the data frames of two adjacent viewpoints to find corresponding points. First, calculate the distance between any two points in the data frames of the two viewpoints. Perform preliminary matching based on the similarity of the spatial distance between each point in the data frames of the two viewpoints and the remaining points in the data frames. Find all candidate corresponding points and their neighboring points to form a candidate point set. If there is a marker point in the candidate point set that is paired with more than two marker points, perform secondary matching based on the similarity of the angular relationship between each point in the candidate set corresponding to the two viewpoints and the remaining points in the candidate set. Obtain the correct matching marker point pair and output the final matching result.

[0047] S03. Based on the matching relationship of corresponding points between data frames from adjacent viewpoints, obtain the coordinate system transformation relationship between data frames from adjacent viewpoints;

[0048] S04. After converting all the collected data to the same viewpoint coordinate system according to the coordinate system transformation relationship, the three-dimensional model is reconstructed.

[0049] This embodiment pre-arranges easily identifiable and matching marker points on or near the target to be measured, and collects each marker point from different perspectives. Since the data frames collected from two adjacent perspectives have common marker points, the coordinate transformation relationship between adjacent perspective data frames can be indirectly obtained based on the matching relationship of common marker points (corresponding points) between the data frames of two adjacent perspectives. Finally, the coordinate transformation relationship between each pair of perspectives under all perspectives can be obtained. Then, by using this coordinate transformation relationship, all data can be registered in a unified coordinate system, which can achieve fast and high-precision point cloud stitching and complete efficient and high-precision 3D model reconstruction.

[0050] Simultaneously, during the search for common landmarks, considering that if two landmarks are of the same name, the spatial distance and angular size between these two landmarks and the surrounding points usually have a high degree of similarity, this embodiment utilizes the aforementioned characteristics of corresponding landmarks. First, it performs a coarse matching based on distance relationships to find candidate corresponding landmarks in adjacent viewpoint data frames whose spatial distance similarity with the surrounding points in the data frame meets preset requirements. This allows for a rapid fuzzy search of all possible corresponding landmarks, achieving coarse matching of common landmarks. Considering that landmarks are prone to pairwise proximity or equality, spatial distance is used to pair all landmarks whose distances meet the requirements, resulting in a... For ambiguities and mismatches in two-to-two (one marker point simultaneously forms a common marker point pair with two other marker points) or one-to-many (one marker point simultaneously forms a common marker point pair with three or more other marker points), a secondary matching is performed based on the angular relationship between the candidate corresponding point and the remaining points. The correct matching relationship is finally determined based on the angular similarity. Extending from the spatial distance relationship between marker points to the angular relationship, the neighborhood relationship of spatial points can be fully utilized to quickly and accurately determine the final marker point pair relationship, solve the ambiguity problem of one-to-two or one-to-many, effectively reduce mismatches, and can be flexibly applied to various scenarios without being limited by the shape, size or the visibility of features of the target being tested.

[0051] The aforementioned markers can be non-coded circular markers, placed on or near the target. During placement, it is sufficient to ensure that adjacent viewpoints can simultaneously scan both the circular markers and the target. In a specific application embodiment, the circular markers are used as features to be extracted. A 3D camera acquires image data and point cloud data of the circular markers. If data frames acquired from two adjacent viewpoints share common markers, the coordinate transformation relationship between the two camera viewpoints can be indirectly obtained by finding the mathematical relationship between the common markers (corresponding points) between adjacent viewpoints. It is understood that the shape and type of the markers can be selected according to actual needs; for example, coded points can be used instead of non-coded points, or a combination of coded and non-coded points can be used.

[0052] In this embodiment, step S02, which involves preliminary matching based on the similarity of the spatial distance between each point in two viewpoint data frames and the remaining points within the same data frame, includes the following specific steps:

[0053] S201. Calculate the distance between any two points within the data frames of the two viewpoints, and obtain the first distance matrix A and the second distance matrix B respectively. The first distance matrix A is calculated from the data frames of one viewpoint, and the second distance matrix B is calculated from the data frames of the other viewpoint.

[0054] S202. Calculate the difference between each distance point in the first distance matrix A and the second distance matrix B, and determine the similarity based on the magnitude of the difference;

[0055] S203. Find all distance points in the first distance matrix A and the second distance matrix B whose difference between distance points is less than a preset threshold;

[0056] S204. Based on all the distance points found in the first distance matrix A, determine the candidate corresponding points and the neighboring points of the candidate corresponding points to form a first candidate set N1. Based on the distance points found in the second distance matrix B, determine the candidate marker points and the neighboring points of the candidate marker points to form a second candidate set M1.

[0057] In this embodiment, the distance value from one point in one data frame to other points in adjacent data frames is subtracted from the distance value from one point in another data frame to other points. The point pairs with differences less than a threshold are counted. The magnitude of this value can reflect the similarity. The larger the value, the higher the similarity, and vice versa. By finding all distance points with differences less than a preset threshold, candidate homonyms and neighboring points can be determined.

[0058] In a specific application embodiment, the elements of each row in the first distance matrix A and the second distance matrix B correspond to the first... i Spatial distance between each point and the other points i=1, 2, ..., m , m The number of points in the data frame; in step S204, if the number of elements in the k-th row of the first distance matrix A and the p-th row of the second distance matrix B whose differences are less than a preset threshold exceeds a preset threshold, then the points corresponding to the k-th and p-th rows are determined to be candidate corresponding points. Among the distance points whose differences are less than the preset threshold, the points whose distances are calculated with the candidate corresponding points are the neighborhood points of the candidate corresponding points. k and p correspond to any row number of distance matrices A and B, respectively. The distance matrices A and B are as follows:

[0059] (1)

[0060] (2)

[0061] In this matrix, matrix A is constructed based on the point set S obtained from the previous viewpoint, and matrix B is constructed based on the point set T obtained from the next viewpoint. S0,1 represents the distance between points with indices 0 and 1 in the current point set S. The i-th row in the matrix represents the i-th row of the point set. i One point.

[0062] This embodiment, based on the maximum matching principle, first finds candidate points of the same name based on the maximum matching of spatial relationships. For example, if a point p in point set S and a point q in point set T are points of the same name, then spatially, p and q will have points with the greatest distance or angle similarity. That is, the distances between p and other points in point set S and the distances between q and other points in point set T will have multiple equal distances. The more points with equal distances, the greater the similarity between the two points. Based on this characteristic, points of the same name can be found using the maximum matching principle.

[0063] Taking the matrices (1) and (2) constructed above as examples, the distance element comparison is performed between the i-th row of matrix A and each row of matrix B. If the element comparison satisfies equation (3), the elements are considered equal. If the difference between the elements in the i-th row and the j-th row of B is less than a specified number (e.g., 3) or more of the elements, the difference is considered less than the preset difference threshold t1. i With T j These are candidate points with the same name. For example, if S... 0,1 T 0,1 S 0,2 T 0,2 S 0,k T 0,x If equation (3) is satisfied, then S0 and T0 are a set of candidate points with the same name, and S1, S2, and S... k These are the neighborhood points of S0, T1, T2, and T... x For the neighborhood points of T0, set S0, S1, S2, and S... k With T0, T1, T2, Tx Store them in sets N1 and M1. The specific sets N1 and M1 are shown in equation (4). The first column of N1 and M1 is the candidate points with the same name that have been found, and the rest are the neighborhood points of the candidate points with the same name.

[0064] (3)

[0065] (4)

[0066] After the above distance-based coarse matching, all possible points with the same name (common landmark point pairs) can be found. Among these, there may be one-to-two or one-to-many cases (for example, in equation (4), S0 and T0 and S1 and T0 are matched as points with the same name at the same time, and T0 is successfully paired with S0 and S1 at the same time). It is necessary to further determine the correct matching relationship. Considering that if it is a correct common landmark point pair, the angular relationship between the two and other surrounding points should also be the most similar, this example further utilizes this angular relationship characteristic. After performing coarse matching based on distance, a second matching is performed on each landmark point pair according to the similarity relationship between the angular size between each point in one candidate set and the other points in the set and the angular size between each point in another candidate set and the other points in the set, so as to finally determine the correct common landmark point pair.

[0067] In this embodiment, step S02, the secondary matching step based on the similarity of the angular magnitude relationship between each point in the candidate set obtained from the two viewpoint data frames and the other points in the candidate set, includes:

[0068] S211. Calculate the angle between any two element points in the first candidate set N1 to form a first angle matrix, and calculate the angle between any two element points in the second candidate set M1 to form a second angle matrix;

[0069] S212. If the angle value of the first target candidate corresponding point and the angle value of its corresponding neighboring point in the first angle matrix have the greatest similarity to the angle value of the second target candidate corresponding point and the angle value of its corresponding neighboring point in the second angle matrix, then the first target candidate corresponding point and the second target candidate corresponding point are determined to be a successfully matched pair of points.

[0070] This embodiment uses the maximum matching principle of the angle matrix when there are one-to-two or one-to-many matching points. It judges the similarity of point pairs by the size of the point pairs whose angle difference is less than a preset threshold. The point pair with the largest similarity in the size relationship of the angle values ​​with the other points is used as the mark point pair for the final successful matching, so as to effectively solve the problem of mismatch in one-to-two or one-to-many matching.

[0071] like Figure 2As shown, when two points S1 and S2 in a point set S have similar distance information, mismatches are likely to occur. The true corresponding points of S1 and S2 should be T2 and T1, but because S1 and S2 have similar distance information, S1 is mismatched with T1, and S2 is mismatched with T2. Therefore, the angular relationship between S1 and S2 and other candidate corresponding points in the point set S is calculated, and the angular relationship between T1 and T2 and other candidate points in the point set T is calculated. The true corresponding point pairs are obtained by using the maximum matching principle.

[0072] Taking the above formula (4) as an example, S0, T0 and S1, T0 are matched as points of the same name. An angle matrix is ​​used for further judgment. The first row point set and the second row point set of N1 and M1 are used to establish an angle matrix with the angle values ​​of each pair as elements. The maximum matching method of the angle matrix is ​​used to eliminate ambiguous points of the same name. For example, if S0, S1, S2, S k The combined angles and T0, T1, T2, T x The maximum matching characteristic indicates that S0 and T0 are correct corresponding points and are removed. Furthermore, for corresponding points that are not one-to-many, we can directly determine whether they have more than 3 angular similarities (i.e., whether they have more than 3 points with similar angular sizes to surrounding points). If so, they are correct corresponding points; otherwise, they are not corresponding points. The point sets N1 and M1 after the above filtering are the final matching result point sets N2 and M2.

[0073] In step S212, the differences between each angle value in the first angle matrix and each angle value in the second angle matrix are calculated. If the number of difference points between the angle values ​​of the first target candidate corresponding point and their corresponding neighboring points and the angle values ​​of the second target candidate corresponding point and their corresponding neighboring points that are less than a preset threshold is the largest, then the angle value similarity is determined to be the greatest. That is, the similarity between angle values ​​is judged by the distance between them. If the distance between angle values ​​is less than a preset threshold, it indicates that the two angle values ​​are similar. Then, the correct corresponding point can be determined by finding the point in the two angle matrices with the largest number of difference points between the angle values ​​that are less than the preset threshold.

[0074] In step S03 of this embodiment, if the number of currently obtained homonyms is greater than a preset threshold, the SVD singular value decomposition method is used to solve the transformation matrix between adjacent view data frames, so as to use the solved transformation matrix to complete the splicing of adjacent two frames of data; if the solved transformation matrix cannot complete the splicing of adjacent two frames of data, the currently obtained homonyms are sorted according to the number of their neighboring points, and the homonym with the smallest neighboring points is removed, and the SVD singular value decomposition method is used again to solve the problem and splice the adjacent two frames of data until the splicing is finally completed. If the number of matching points in N2 and M2 reaches a certain number, the transformation matrix between the viewpoint of the previous frame and the viewpoint of the next frame can be solved directly by the Singular Value Decomposition (SVD) method, thus achieving the splicing of adjacent frames. If the splicing of the two frames cannot be achieved after solving, it indicates that there are incorrect matching points in N2 and M2. In this case, N2 and M2 are further sorted according to the number of neighboring points contained in the matching point. The matching point with the smallest neighboring point is removed, and the splicing is calculated again based on SVD. Otherwise, the iteration continues until the correct splicing is achieved. If the correct splicing is still not achieved, it can be determined that the current two frames do not meet the requirement of the preset number of matching points.

[0075] After all frame point clouds are transformed to the same coordinate system through common marker point matching, errors will exist due to coarse stitching between adjacent frames. Consequently, cumulative errors will occur after stitching multiple frames, which will seriously affect the stitching quality. To eliminate the aforementioned cumulative errors, this embodiment includes a global optimization step after step S03 and before step S04 to further optimize the transformed data globally, eliminate the cumulative errors caused by multi-frame stitching, and ensure the effect of 3D reconstruction.

[0076] After converting adjacent frame data, the global error can be calculated using the following formula:

[0077] (5)

[0078] Among them, M m M n P represents the transformation matrices from the m-th frame and the n-th frame to the first frame, respectively. mi P ni Let M and N represent the i-th pair of corresponding points in the m-th and n-th frames, respectively. Global optimization is essentially optimizing M. m M n、 P mi P ni This minimizes the global error e.

[0079] Considering that the acquisition camera may or may not be suitable for intrinsic parameter optimization, this embodiment adopts the principle of pose graph optimization to add loop closure detection for the case where the camera is not suitable for intrinsic parameter optimization. That is, loop closure is added to the shooting trajectory. For example, the first frame captures region A, and after multiple frames, the camera recaptures region A, thus forming a loop. Loop closure detection is used to establish a strong constraint relationship across multiple frames to achieve extrinsic parameter optimization of the camera. The aforementioned extrinsic parameter is the transformation matrix with the first frame, which is M in equation (5). m M n Since both marker center coordinate identification and extrinsic parameters contain errors, optimizing only the extrinsic parameters offers limited accuracy improvement. This embodiment utilizes loop closure detection combined with bundle adjustment to simultaneously optimize both extrinsic and extrinsic parameters, thereby further improving marker recognition accuracy and making the marker center coordinates closer to the true values. This, in turn, improves the accuracy of the transformation relationship between adjacent viewpoints, and further enhances the accuracy of M... m M n、 P mi P ni Optimization can reduce the global value of 'e', ​​effectively improving the overall stitching effect.

[0080] In this embodiment, for cases where intrinsic parameter optimization is applicable to the camera, a method based on the principle of bundle adjustment optimization with loop closure detection is adopted. This can improve the accuracy of marker recognition, improve the accuracy of the transformation matrix, and optimize the global stitching effect.

[0081] This embodiment employs different optimization methods based on different applicable scenarios. For cameras that are not suitable for intrinsic parameter optimization, a pose graph-based optimization and loop closure detection approach is used to improve the accuracy of the transformation matrix and optimize the global stitching effect. For cameras that are suitable for intrinsic parameter optimization, a light speed adjustment and loop closure detection approach is used to improve the accuracy of marker recognition. This approach can be flexibly applied to different application scenarios to achieve global optimization.

[0082] In specific application embodiments, such as Figure 4 As shown in the figure, the left side corresponds to the marker point data before optimization, and the right side corresponds to the marker point data after optimization. It can be seen from the figure that before optimization, the gray-filled marker points and the black-filled marker points did not overlap much due to the cumulative error, which caused the layering phenomenon. After global optimization using the method described in this embodiment, the marker points overlap much, which can make the layering phenomenon basically disappear and the cumulative error eliminated.

[0083] The following example, using a specific application embodiment to achieve multi-view 3D point cloud reconstruction, further illustrates the present invention. Figure 5 As shown, the detailed steps for multi-view 3D point cloud reconstruction in this embodiment are as follows:

[0084] S1. Circular markers are attached to the workpiece to be tested. Image data and 3D point cloud data of the markers are collected from different perspectives using a 3D camera. It is necessary to ensure that adjacent perspectives have common markers.

[0085] S2. By constructing distance and angle matrices, the common landmarks (corresponding points) between data frames from adjacent viewpoints are found using the relationship between the distance and angle matrices, and the transformation relationship between the coordinate systems of adjacent two frames is calculated.

[0086] like Figure 6 As shown, the detailed steps for finding common landmarks (corresponding points) are as follows:

[0087] S2.1. For the point clouds of the marker points in the previous frame and the next frame respectively, calculate the spatial distance between any two points and establish distance matrices A and B, as shown in equations (1) and (2).

[0088] S2.2. Based on the maximum matching principle of the Hungarian algorithm, find the corresponding points of matrix A and matrix B: compare the distance elements of the i-th row of matrix A and each row of matrix B. If the element comparison satisfies equation (3), the elements are considered equal. If the difference between the elements of the i-th row and the j-th row of B exceeds a preset number of elements and is less than T1, then S2.2. i With T j For points with the same name, S i With T j Store the new matrices N1 and M1 to re-establish the matrix of corresponding points. The row numbers of matrices N1 and M1 are a set of corresponding points, and the resulting matrices N1 and M1 are shown in equation (4).

[0089] S2.3. For possible erroneous matches in N1 and M1, construct an angle matrix with pairwise angle values ​​as elements for the corresponding points in N1 and M1, use the maximum matching method of angle matrix to eliminate ambiguous corresponding points, and store the correct corresponding points into the new matrix N2 and M2.

[0090] S2.4. Again, use the angle matrix to extract corresponding points to remove erroneous matches and store them in new matrices N3 and M3. If the number of corresponding points in N3 and M3 is greater than the preset number, solve the transformation matrix between the viewpoint of the previous frame and the viewpoint of the next frame by using the SVD singular value decomposition method to realize the splicing of data between adjacent frames.

[0091] S3. Use the transformation matrix between adjacent viewpoints obtained in step S2 to transform the data to any viewpoint coordinate system. Specifically, the viewpoint of the first frame can be selected as the base coordinate system.

[0092] S4. Perform global optimization on the transformed data to eliminate accumulated errors and reconstruct the 3D model.

[0093] The multi-view 3D point cloud reconstruction device in this embodiment includes:

[0094] The acquisition module is used to acquire point cloud data of multiple pre-arranged marker points from different perspectives. The marker points are arranged on the target to be measured or within a specified range near the target to be measured. Data acquired from adjacent perspectives contain points with the same name, which are the same marker points.

[0095] The matching and search module is used to match data frames from two adjacent viewpoints to find corresponding points. First, the distance between any two points in the data frames of the two viewpoints is calculated. Then, a preliminary matching is performed based on the similarity of the spatial distance between each point in the data frames of the two viewpoints and the other points in the data frames. All candidate corresponding points and their neighboring points are found to form a candidate point set. If a marker point in the candidate point set is paired with two or more marker points, a secondary matching is performed based on the similarity of the angular relationship between each point in the candidate set corresponding to the two viewpoints and the other points in the candidate set. The correct matching marker point pairs are obtained, and the final matching result is output.

[0096] The transformation relationship determination module is used to obtain the coordinate system transformation relationship between adjacent view data frames based on the matching relationship of corresponding points between adjacent view data frames;

[0097] The 3D reconstruction module is used to convert all the collected data to the same viewpoint coordinate system according to the coordinate system transformation relationship, and then reconstruct the 3D model.

[0098] In this embodiment, the matching and searching module includes a first matching unit for performing preliminary matching based on the similarity of the spatial distance relationship between each point in two viewpoint data frames and other points in the same viewpoint data frame. The first matching unit includes:

[0099] The distance calculation subunit is used to calculate the distance between any two points within the data frames of two viewpoints, and obtains a first distance matrix A and a second distance matrix B respectively. The first distance matrix A is calculated from the data frames of one viewpoint, and the second distance matrix B is calculated from the data frames of the other viewpoint.

[0100] The distance difference calculation subunit is used to calculate the difference between each distance point in the first distance matrix A and the second distance matrix B, so as to determine the similarity based on the magnitude of the difference;

[0101] The search sub-unit is used to find all distance points in the first distance matrix A and the second distance matrix B whose difference between distance points is less than a preset threshold.

[0102] The candidate set construction sub-unit is used to determine candidate corresponding points and their neighboring points based on all distance points found in the first distance matrix A to form a first candidate set N1, and to determine candidate marker points and their neighboring points based on the distance points found in the second distance matrix B to form a second candidate set M1.

[0103] In this embodiment, the matching and search module includes a second matching unit that performs secondary matching based on the similarity of the angular relationship between each point in the candidate set obtained from the two viewpoint data frames and the remaining points in the candidate set. The second matching unit includes:

[0104] Angle calculation subunit is used to calculate the angle between any two element points in the first candidate set N1 to form a first angle matrix, and to calculate the angle between any two element points in the second candidate set M1 to form a second angle matrix.

[0105] In the matching sub-unit, if the angle value of the first target candidate corresponding point and the angle value of its corresponding neighboring point in the first angle matrix have the greatest similarity to the angle value of the second target candidate corresponding point and the angle value of its corresponding neighboring point in the second angle matrix, then the first target candidate corresponding point and the second target candidate corresponding point are determined to be a successfully matched pair of marker points.

[0106] In this embodiment, a global optimization module is also included, which is connected between the transformation relationship determination module and the 3D reconstruction module. This module is used to perform a global optimization step on the transformed data. The global optimization step adopts a pose graph-based optimization method and uses loop closure detection to establish strong constraints across multiple frames to achieve camera extrinsic parameter optimization; or it adopts a bundle adjustment-based optimization method and loop closure detection to achieve optimization.

[0107] The multi-view 3D point cloud reconstruction device in this embodiment corresponds one-to-one with the above-described multi-view 3D point cloud reconstruction method, and will not be described in detail here.

[0108] This embodiment also provides a computer device, including a processor and a memory, the memory for storing a computer program, and the processor for executing the computer program to perform the method as described above.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A multi-view 3D point cloud reconstruction method, characterized by the following steps: include: S01. Collect point cloud data of multiple pre-arranged marker points from different perspectives. The marker points are arranged on the target to be measured or within a specified range near the target to be measured. Data collected from adjacent perspectives contain points with the same name, and the points with the same name are the same marker points. S02. Match the data frames of two adjacent viewpoints to find corresponding points. First, calculate the distance between any two points in the data frames of the two viewpoints. Perform preliminary matching based on the similarity of the spatial distance between each point in the data frames of the two viewpoints and the remaining points in the data frames. Find all candidate corresponding points and their neighboring points to form a candidate point set. If there is a marker point in the candidate point set that is paired with more than two marker points, perform secondary matching based on the similarity of the angular relationship between each point in the candidate set corresponding to the two viewpoints and the remaining points in the candidate set. Obtain the correct matching marker point pair and output the final matching result. S03. Based on the matching relationship of corresponding points between data frames from adjacent viewpoints, obtain the coordinate system transformation relationship between data frames from adjacent viewpoints; S04. After converting all the collected data to the same viewpoint coordinate system according to the coordinate system transformation relationship, perform three-dimensional model reconstruction; The secondary matching based on the similarity of the angular magnitude relationship between each point in the candidate set obtained from the two viewpoint data frames and the other points in the candidate set includes: S211. Determine the first angle matrix and the second angle matrix based on the candidate sets obtained from the two viewpoint data frames; S212. If the angle value of the first target candidate corresponding point and the angle value of its corresponding neighboring point in the first angle matrix have the greatest similarity to the angle value of the second target candidate corresponding point and the angle value of its corresponding neighboring point in the second angle matrix, then the first target candidate corresponding point and the second target candidate corresponding point are determined to be a successfully matched pair of marker points.

2. The multi-view 3D point cloud reconstruction method according to claim 1, characterized in that, The specific steps in step S02 for preliminary matching based on the similarity of spatial distances between each point in the two viewpoint data frames and the remaining points within the same data frame include: S201. Calculate the distance between any two points within the data frames of the two viewpoints, and obtain the first distance matrix A and the second distance matrix B respectively. The first distance matrix A is calculated from the data frames of one viewpoint, and the second distance matrix B is calculated from the data frames of the other viewpoint. S202. Calculate the difference between each distance point in the first distance matrix A and the second distance matrix B, and determine the similarity based on the magnitude of the difference; S203. Find all distance points in the first distance matrix A and the second distance matrix B whose difference between distance points is less than a preset threshold; S204. Based on all the distance points found in the first distance matrix A, candidate corresponding points and their neighboring points are determined to form a first candidate set N1. Based on the distance points found in the second distance matrix B, candidate marker points and their neighboring points are determined to form a second candidate set M1.

3. The multi-view 3D point cloud reconstruction method according to claim 2, characterized in that, The elements of each row in the first distance matrix A and the second distance matrix B correspond to the first... i Spatial distance between each point and the other points i =1, 2, ..., m , m The number of points in the data frame; in step S204, if it is determined that the number of distance points whose difference between the elements of the k-th row of the first distance matrix A and the p-th row of the second distance matrix B is less than a preset threshold exceeds a preset threshold, then the points corresponding to the k-th row and the p-th row are determined to be candidate corresponding points. Among the distance points whose difference is less than the preset threshold, the points whose distance is calculated with the candidate corresponding points are the neighborhood points of the candidate corresponding points. k and p correspond to any number of rows in distance matrices A and B, respectively.

4. The multi-view 3D point cloud reconstruction method according to claim 2, characterized in that, The step of determining the first angle matrix and the second angle matrix based on the candidate set obtained from the two viewpoint data frames includes: Calculate the angle between any two elements in the first candidate set N1 to form a first angle matrix, and calculate the angle between any two elements in the second candidate set M1 to form a second angle matrix.

5. The multi-view 3D point cloud reconstruction method according to claim 4, characterized in that, In step S212, the difference between each angle value in the first angle matrix and each angle value in the second angle matrix is ​​calculated. If the number of difference points between the angle value of the first target candidate corresponding point and the angle value of its corresponding neighboring point and the angle value of the second target candidate corresponding point and the angle value of its corresponding neighboring point is less than a preset threshold is the largest, then the angle value similarity is determined to be the largest.

6. The multi-view 3D point cloud reconstruction method according to any one of claims 1 to 5, characterized in that, In step S03, if the number of corresponding points obtained is greater than a preset threshold, the SVD singular value decomposition method is used to solve the transformation matrix between adjacent view data frames, and the solved transformation matrix is ​​used to complete the splicing of adjacent two frames of data; if the solved transformation matrix cannot complete the splicing of adjacent two frames of data, the corresponding points obtained are sorted according to the number of their neighboring points, and the corresponding point with the smallest neighboring points is removed, and the SVD singular value decomposition method is used again to solve the problem and splice the adjacent two frames of data until the splicing is finally completed.

7. The multi-view 3D point cloud reconstruction method according to any one of claims 1 to 5, characterized in that, After step S03 and before step S04, the process also includes global optimization of the transformed data to minimize the global error e. The global optimization adopts a pose graph-based optimization method and uses loop closure detection to establish strong constraints across multiple frames to achieve camera extrinsic parameter optimization, or uses a bundle adjustment optimization method and loop closure detection to achieve camera extrinsic parameter optimization. The loop closure detection is performed by acquiring images in a loop-based manner.

8. The multi-view 3D point cloud reconstruction method according to claim 7, characterized in that, The global error e is calculated using the following formula: Among them, M m M n P represents the transformation matrices from the m-th frame and the n-th frame to the first frame, respectively. mi P ni These represent the i-th pair of corresponding points in the m-th and n-th frames, respectively.

9. A multi-view 3D point cloud reconstruction device, characterized in that, include: The acquisition module is used to acquire point cloud data of multiple pre-arranged marker points from different perspectives. The marker points are arranged on the target to be measured or within a specified range near the target to be measured. Data acquired from adjacent perspectives contain points with the same name, which are the same marker points. The matching and search module is used to match data frames from two adjacent viewpoints to find corresponding points. First, the distance between any two points in the data frames of the two viewpoints is calculated. Then, a preliminary matching is performed based on the similarity of the spatial distance between each point in the data frames of the two viewpoints and the remaining points in the data frames. All candidate corresponding points and their neighboring points are found to form a candidate point set. If a marker point in the candidate point set is paired with two or more marker points, a secondary matching is performed based on the similarity of the angular relationship between each point in the candidate set corresponding to the two viewpoints and the remaining points in the candidate set. The correct matching marker point pairs are obtained, and the final matching result is output. The transformation relationship determination module is used to obtain the coordinate system transformation relationship between adjacent view data frames based on the matching relationship of corresponding points between adjacent view data frames; The 3D reconstruction module is used to convert all the collected data to the same viewpoint coordinate system according to the coordinate system transformation relationship, and then perform 3D model reconstruction. The matching and search module includes a second matching unit that performs secondary matching based on the similarity of the angular magnitude relationship between each point in the candidate set obtained from the two viewpoint data frames and the remaining points in the candidate set. The second matching unit includes: Angle determination subunit is used to determine the first angle matrix and the second angle matrix based on the candidate set obtained from the two viewpoint data frames; In the matching subunit, if the angle value of the first target candidate corresponding point and the angle value of its corresponding neighboring point in the first angle matrix have the greatest similarity to the angle value of the second target candidate corresponding point and the angle value of its corresponding neighboring point in the second angle matrix, then the first target candidate corresponding point and the second target candidate corresponding point are determined to be a successfully matched pair of marker points.

10. A computer device comprising a processor and a memory, the memory being used to store computer programs, characterized in that, The processor is used to execute the computer program to perform the method as described in any one of claims 1 to 8.

Images