Quasi dense matching method based on object space and image space combined constraint

Abstract

The invention provides a quasi dense matching method based on object space and image space combined constraint. The method comprises nine steps: step 1, inputting inner parameter matrixes of a left image, a right image and a camera; 2, eliminating wrong matching points; 3, calculating an initial three-dimensional point set; step 4, object space plane fitting; step 5, constructing a matching primitive; step 6, transmitting plane parameters; step 7, carrying out object space and image space joint matching constraint; 8, performing iterative encryption on the triangulation network; and 9, outputting a matching result. The quasi-dense matching method based on the object space and image space combined constraint is designed for solving the problem that a traditional quasi-dense matching method depends on image gray information and has matching ambiguity in a gray similar area, and the number of matching points of the algorithm in the gray similar area is increased, and the precision of the matching points of the algorithm in the gray similar area is improved. Therefore, the method can be applied to architectural design, urban planning and cultural enthusiasm protection, and has wide application prospect and value.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image-based three-dimensional reconstruction technology, and in particular, utilizes a quasi-dense matching method to obtain point clouds. Background Art

[0002] With the advent of the digital age, 3D reconstruction technology has become increasingly widely used, playing a vital role in fields such as architectural design, urban planning, and cultural heritage preservation. Existing 3D reconstruction methods include LiDAR (Light Detection and Ranging), structured light, and image-based 3D reconstruction. Image-based 3D reconstruction methods are widely used due to their flexible data sources, reliable results, low hardware requirements, and low cost. Stereo matching is a key step in image-based 3D reconstruction, and the quality of matching points determines the success or failure of the 3D reconstruction results. Quasi-dense matching, as one of the main methods of stereo matching, offers advantages such as high precision, strong robustness, good scalability, high parallelization capabilities, and adjustability, making it a commonly used technique in many stereo vision applications. To achieve robust, efficient, and dense matching of same-name points, two main approaches are currently used for quasi-dense matching: one based on image space constraints and the other based on object space constraints. Image-space-constrained algorithms are prone to false matches in weakly textured areas with similar grayscale values. The number of matching points is not dense enough, and matching results are affected by the characteristics of adjacent points, making it difficult to obtain valid matching results in overlapping areas. Compared to quasi-dense matching methods based on image space constraints, object-space-constrained methods use the image's object space information to constrain the image-space positions of matching points, effectively reducing the algorithm's reliance on image grayscale information. However, object-space-constrained methods are still limited by the prerequisite that the image must be parallel to the ground plane, making them unsuitable for 3D reconstruction of close-range image sequences. While existing object-space-constrained methods have made progress in comprehensively utilizing object and image space information, reducing reliance on image grayscale information, and improving matching reliability, these methods are only applicable to central projection images captured by specialized equipment. These limitations reduce the universality of object-space-constrained methods and hinder their application in stereo matching of close-range image sequences. Therefore, how to effectively utilize both image and object space information remains an important research direction in this field. Summary of the Invention

[0003] In view of the deficiencies of the prior art, the present invention provides a quasi-dense matching method based on joint constraints of object space and image space.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A quasi-dense matching method based on joint constraints of object space and image space includes the following steps:

[0006] Step 1: Input the left and right images I1, I2 and the camera's intrinsic parameter matrix K;

[0007] Step 2: Use SIFT algorithm to extract matching points from the left and right images and use RANSAC algorithm to remove incorrect matching points;

[0008] Step 3: Calculate the initial 3D point set using triangulated projection;

[0009] Step 4: Use the RANSAC (Random Sample Consensus) algorithm to fit the plane where the initial 3D point is located in the object space;

[0010] Step 5: Triangulate the initial matching points and select the midpoints of each side of the triangle as matching primitives;

[0011] Step 6: Pass the object plane parameters to the candidate matching points;

[0012] Step 7: Joint matching constraints between the object and image space. Calculate the distance between the matching point and the object plane, and use the Normalized Cross-Correlation Coefficient (NCC) to weight the candidate matching points and determine the candidate matching points.

[0013] Step 8: Iterative refinement of triangulated mesh.

[0014] Step 9: Output of quasi-dense matching results.

[0015] Furthermore, the step 1 specifically includes:

[0016] Input experimental images I1, I2 and the corresponding camera intrinsic parameter matrix K.

[0017] Furthermore, the step 2 specifically includes:

[0018] The traditional scale-invariant feature algorithm (SIFT) is used to extract the initial matching points of the experimental image, and the RANSAC algorithm is used to eliminate the wrong matching points.

[0019] Furthermore, the step 3 specifically includes:

[0020] Step 3.1: Calculate the essential matrix between the left and right images and decompose the essential matrix to obtain the camera's rotation matrix R and translation vector T, thereby obtaining the camera's extrinsic parameter matrix;

[0021] Step 3.2: Take the left camera as the reference, that is, the rotation matrix of the left camera is the unit matrix and the translation vector is 0 T, combined with the intrinsic parameter matrices of the left and right cameras, the camera projection matrices of the left and right cameras are obtained respectively, and the initial matching points are triangulated to obtain the initial three-dimensional point set.

[0022] Furthermore, the step 4 specifically includes:

[0023] Step 4.1: Set the minimum subset threshold K1, distance threshold T and number of iterations N, where K1 represents the minimum number of points that should be included in the plane. If the number of inliers included in the fitted plane is less than K1, refit the plane. T represents the maximum distance from the inliers to the fitted plane; if P i (x i ,y i , z i ) to the plane is less than T, then P i (x i ,y i , z i ) are marked as inliers. R represents the number of iterations of the RANSAC algorithm;

[0024] Step 4.2: Randomly select n non-collinear points P1, P2 and P3 from P ( n >3), as the candidate subset of the initial plane model, and calculate the normal vector of the plane based on points P1, P2 and P3, where the normal vector is shown in Formula 1:

[0025] N=|n1 n2 n3|=P1P2*P1P3 (1)

[0026] Where n1, n2, and n3 represent the normal vector of a plane in three-dimensional space;

[0027] Step 4.3: Calculate the distance d from the remaining points in P to the estimated plane;

[0028] Step 4.4: Mark the data points whose distance d is less than the given threshold T as inliers, and count the number of inliers contained in the estimated plane;

[0029] Step 4.5: Iterate R times, selecting different sample points each time to obtain different planes and the corresponding number of inliers;

[0030] Step 4.6: Select the plane model with the largest number of inliers. If the number of inliers in the plane is greater than the threshold K1, assign the plane parameters to the inliers on the plane and remove these inliers from P.

[0031] Step 4.7: Repeat steps 2.4 to 2.6 until P is empty, or a new plane cannot be fitted and the remaining points are removed from the point set, and the iteration stops.

[0032] Furthermore, the step 5 specifically includes:

[0033] Step 5.1: Use the Delaunay algorithm to triangulate the initial matching points and obtain a triangulated network of the same name. Utilizing the stability of triangles, local geometric constraints are applied to candidate matching points, thereby limiting the search range of candidate matching points and making them more accurate and reliable.

[0034] Step 5.2: Calculate the midpoints of each side of the triangle and use them as matching primitives;

[0035] Step 5.3: Taking the midpoints of the sides of the right image triangle as the center, select points within the 5×5 range of its area as candidate matching points.

[0036] Furthermore, the step 6 specifically includes:

[0037] Assume that the plane parameters of the three vertices of an arbitrary triangle are m1, m2, and m3 respectively, calculate the distances between P(P1, P′1) and the three planes respectively, select the plane closest to point P(P1, P′1) as the reference plane for the candidate matching, and pass its parameters to the candidate matching point.

[0038] Furthermore, the step 7 specifically includes:

[0039] Step 7.1: Calculate the distance between the candidate matching point's corresponding 3D point and the object plane and the Normalized Cross Correlation (NCC) value of each candidate matching point;

[0040] Step 7.2: Weight the results and take the maximum score as the final matching result. If the score is greater than the threshold T1, the newly obtained candidate matching point is inserted into the seed point, where the score is d 、score ncc , score is defined as follows:

[0041]

[0042]

[0043]

[0044] Among them, T d is the distance threshold from the point to the plane, T ncc represents the NCC threshold;

[0045] Step 7.3: When n is less than T ncc When the score ncc is 0; and when n is greater than or equal to T ncc When the score nccWill gradually increase with the increase of n, score d Score and score ncc The score ranges from 0 to 1, with 1 being the highest score and 0 being the lowest score.

[0046] Step 7.4: Select the candidate matching point with the largest score as the new matching point.

[0047] Furthermore, the step 8 specifically includes:

[0048] Step 8.1: Count the number of newly added matching points S, re-perform Delaunay triangulation on the matching point set, and count the number of all triangles in the triangulation network S1, initializing i = 1;

[0049] Step 8.2: If S=0, go to step 8.5, otherwise go to step 8.3;

[0050] Step 8.3: Select the i-th triangle. If the area of ​​the i-th triangle is less than the specified threshold S2, repeat steps 5 to 7. Otherwise, go to step 8.4.

[0051] Step 8.4: Let i = i + 1. If i ≤ S1, go to step 8.2; otherwise, go to step 8.1.

[0052] Step 8.5: The iteration stops and the final dense matching point set is obtained.

[0053] Furthermore, the step 9 specifically includes:

[0054] The final quasi-dense matching result is saved in txt text format, where the four elements in each line of the txt text correspond to the coordinate values ​​of the matching point on the left and right images respectively.

[0055] Advantages of the present invention

[0056] The beneficial effects of adopting the above technical method are:

[0057] The present invention provides a quasi-dense matching method based on joint constraints of object and image space, which has the following beneficial effects:

[0058] (1) The present invention selects the midpoints of each side of the triangle as matching primitives, utilizes the stability characteristics of the triangle, imposes local geometric constraints on candidate matching points, and limits the search range of candidate matching points, thereby improving the algorithm's operating efficiency;

[0059] (2) The present invention introduces object plane constraints to restrict the three-dimensional point positions corresponding to candidate matching points to the vicinity of the object plane, and combines the grayscale constraints of the image plane to enhance the constraint ability on candidate matching points, so that the algorithm can extract more high-quality matching points in grayscale similar areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The description of the contents of the present invention will become more apparent and easier to understand when taken in conjunction with the following drawings, in which:

[0061] Figure 1 Flowchart of a quasi-dense matching method based on image-space and object-space joint constraints in a specific embodiment of the present invention;

[0062] Figure 2 A schematic diagram of constructing a matching primitive in a specific embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram of object plane constraints in a specific embodiment of the present invention;

[0064] Figure 4 These are three sets of experimental images in a specific embodiment of the present invention;

[0065] Figure 5 This is the experimental result of quasi-dense matching in a specific embodiment of the present invention;

[0066] Figure 6 This is the experimental result of three-dimensional reconstruction in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following embodiments are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0068] A quasi-dense matching method based on joint constraints of object and image space, such as Figure 1 Shown, including:

[0069] Step 1: Select experimental images and the corresponding camera intrinsic parameter matrix from the public dataset provided by https: / / www.epfl.ch / labs / cvlab / data / data-strechamvs. The experimental images selected in the present invention are as follows: Figure 4 shown.

[0070] Step 2: Use the traditional scale-invariant feature algorithm (SIFT) to extract the initial matching points of the experimental image, and use the RANSAC algorithm to eliminate the wrong matching points to obtain high-precision initial matching points.

[0071] Step 3: Use triangulated projection to calculate the initial 3D point set. This includes the following specific steps:

[0072] Step 3.1: Calculate the intrinsic matrix between the left and right images and decompose the intrinsic matrix to obtain the camera's rotation parameters R and translation parameters T, thereby obtaining the camera's extrinsic parameter matrix;

[0073] Step 3.2: Take the left camera as the reference, that is, the rotation matrix of the left camera is the unit matrix and the translation parameter is 0 T , combined with the intrinsic parameter matrices of the left and right cameras, the camera projection matrices of the left and right cameras are obtained respectively, and the initial matching points are triangulated to obtain the initial three-dimensional point set.

[0074] Step 4: Use the RANSAC (Random Sample Consensus) algorithm to fit the plane where the initial 3D points are located in the object space. This includes the following specific steps:

[0075] Step 4.1: Set the minimum subset threshold K1, distance threshold T and number of iterations N, where K1 represents the minimum number of points that should be included in the plane. If the number of inliers included in the fitted plane is less than K1, refit the plane. T represents the maximum distance from the inliers to the fitted plane, such as P i (x i ,y i , z i ) to the plane is less than T, then P i (x i ,y i , z i ) is marked as an interior point, and R represents the number of iterations of the RANSAC algorithm;

[0076] Step 4.2: Randomly select n non-collinear points P1, P2 and P3 (n>3) from P as the candidate subset of the initial plane model, and calculate the normal vector of the plane based on points P1, P2 and P3, where the normal vector is shown in Formula 1:

[0077] N=|n1 n2 n3|=P1P2*P1P3 (1)

[0078] Where n1, n2, and n3 represent the normal vector of a plane in three-dimensional space;

[0079] Step 4.3: Calculate the distance d from the remaining points in P to the estimated plane;

[0080] Step 4.4: Mark the data points whose distance d is less than the given threshold T as inliers, and count the number of inliers contained in the estimated plane;

[0081] Step 4.5: Iterate R times, selecting different sample points each time to obtain different planes and the corresponding number of inliers;

[0082] Step 4.6: Select the plane model with the largest number of inliers. If the number of inliers in the plane is greater than the threshold K1, assign the plane parameters to the inliers on the plane and remove these inliers from P.

[0083] Step 4.7: Repeat steps 2.4 to 2.6 until P is empty, or a new plane cannot be fitted and the remaining points are removed from the point set, and the iteration stops.

[0084] Step 5: Triangulate the initial matching points and select the midpoints of each triangle as matching primitives. This includes the following specific steps:

[0085] Step 5.1: Use the Delaunay algorithm to triangulate the initial matching points to obtain a triangulated network with the same name. Utilize the stability of triangles to impose local geometric constraints on candidate matching points, thereby limiting the search range of candidate matching points and making them more accurate and reliable.

[0086] Step 5.2: Calculate the midpoints of each side of the triangle and use them as matching primitives;

[0087] Step 5.3: Taking the midpoint of each side of the triangle in the right image as the center, select the points within the 5×5 range of its area as candidate matching points. The construction of the matching primitive is as follows: Figure 2 shown.

[0088] Step 6: Pass the object plane parameters to the candidate matching points. This includes the following specific steps:

[0089] Assume that the plane parameters of the three vertices of an arbitrary triangle are m1, m2, and m3 respectively. Calculate the distances between P(P1, P′1) and the three planes respectively, select the plane closest to point P(P1, P′1) as the reference plane for the candidate match, and pass its parameters to the candidate match point. The object plane constraint is as follows: Figure 3 shown.

[0090] Step 7: Combine the object and image space matching constraints, calculate the distance between the matching point and the object plane, and use the Normalized Cross-Correlation Coefficient (NCC) to weight the candidate matching points and determine the candidate matching points. This includes the following specific steps:

[0091] Step 7.1: Calculate the distance between the candidate matching point's corresponding 3D point and the object plane and the Normalized Cross Correlation (NCC) value of each candidate matching point;

[0092] Step 7.2: Weight the results and take the maximum score as the final matching result. If the score is greater than the threshold T1, the newly obtained candidate matching point is inserted into the seed point, where the score is d 、score ncc , score is defined as follows:

[0093]

[0094]

[0095]

[0096] Among them, T d is the distance threshold from the point to the plane, T ncc represents the NCC threshold;

[0097] Step 7.3: When n is less than T ncc score ncc is 0; and when n is greater than or equal to T ncc When the score ncc Will gradually increase with the increase of n, score d Score and score ncc The score ranges from 0 to 1, with 1 being the highest score and 0 being the lowest score.

[0098] Step 7.4: Select the candidate matching point with the largest score as the new matching point.

[0099] Step 8: Iterative encryption of triangulated network. This includes the following specific steps:

[0100] Step 8.1: Count the number of newly added matching points S, re-perform Delaunay triangulation on the matching point set, and count the number of all triangles in the triangulation network S1, initializing i = 1;

[0101] Step 8.2: If S=0, go to step 8.5, otherwise go to step 8.2;

[0102] Step 8.3: Select the i-th triangle. If the area of ​​the i-th triangle is less than the specified threshold S2, repeat steps 5 to 7. Otherwise, go to step 8.4.

[0103] Step 8.4: Let i = i + 1. If i ≤ S1, go to step 8.2; otherwise, go to step 8.1.

[0104] Step 8.5: The iteration stops and the final dense matching point set is obtained.

[0105] Step 9: Output of quasi-dense matching results. The quasi-dense matching results are as follows: Figure 5 As shown, in addition, the dense matching results are triangulated and displayed in the form of three views, as shown in Figure 6 shown.

[0106] The above description is only the best embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A quasi-dense matching method based on joint constraints of object and image space, characterized in that: The steps include: Step 1: Input the left and right images I1, I2 and the camera's intrinsic parameter matrix K; Step 2: Use SIFT algorithm to extract matching points from the left and right images and use RANSAC algorithm to remove incorrect matching points; Step 3: Calculate the initial 3D point set using triangulated projection; Step 4: Use the RANSAC (Random Sample Consensus) algorithm to fit the plane where the initial 3D point set is located in the object space; Step 5: Triangulate the initial matching points and select the midpoints of each side of the triangle as matching primitives; Step 6: Pass the object plane parameters to the candidate matching points; Step 7: Joint matching constraints between the object and image space. Calculate the distance between the matching point and the object plane, and use the Normalized Cross-Correlation Coefficient (NCC) to weight the candidate matching points and determine the candidate matching points. Step 8: Iterative refinement of triangulated mesh. Step 9: Output of quasi-dense matching results.

2. The quasi-dense matching method based on joint constraints of object and image space according to claim 1, characterized in that: The step 4 comprises the following steps: Step 4.1: Set the minimum subset threshold K1, distance threshold T and number of iterations N. K1 represents the minimum number of points that should be included in the plane. If the number of inliers included in the fitted plane is less than K1, the plane is refitted. T represents the maximum distance from the inliers to the fitted plane, such as P i (x i ,y i , z i ) to the plane is less than T, then P i (x i ,y i , z i ) are marked as inliers. R represents the number of iterations of the RANSAC algorithm; Step 4.2: Randomly select n non-collinear points P1, P2 and P3 (n>3) from P as the candidate subset of the initial plane model, and calculate the normal vector of the plane based on points P1, P2 and P3, where the normal vector is shown in Formula 1: N=|n1 n2 n3|=P1P2*P1P3 (1) Where n1, n2, and n3 represent the normal vector of a plane in three-dimensional space; Step 4.3: Calculate the distance d from the remaining points in P to the estimated plane; Step 4.4 marks the data points whose distance d is less than the given threshold T as inliers, and counts the number of inliers contained in the estimated plane; Step 4.5: Iterate R times, selecting different sample points each time to obtain different planes and the corresponding number of inliers; Step 4.6: Select the plane model with the largest number of inliers. If the number of inliers in the plane is greater than the threshold K1, assign the plane parameters to the inliers on the plane and remove these inliers from P. Step 4.7: Repeat steps 2.4 to 2.6 until P is empty, or a new plane cannot be fitted and the remaining points are removed from the point set, and the iteration stops.

3. The quasi-dense matching method based on joint constraints of object and image space according to claim 1, characterized in that: The step 7 comprises the following steps: Step 7.1: Calculate the distance between the candidate matching point's corresponding 3D point and the object plane and the Normalized Cross Correlation (NCC) value of each candidate matching point; Step 7.2: Weight the results and take the maximum score as the final matching result. If the score is greater than the threshold T1, the newly obtained candidate matching point is inserted into the seed point, where the score is d 、score ncc , score is defined as follows: Among them, T d is the distance threshold from the point to the plane, T nc represents the NCC threshold; Step 7.3: When n is less than T ncc score ncc is 0; and when n is greater than or equal to T ncc When the score ncc Will gradually increase with the increase of n, score d Score and score ncc The score ranges from 0 to 1, with 1 being the highest score and 0 being the lowest score. Step 7.4: Select the candidate matching point with the largest score as the new matching point.

4. The quasi-dense matching method based on joint constraints of object and image space according to claim 1, characterized in that: The step 8 comprises the following steps: Step 8.1: Count the number of newly added matching points S, re-perform Delaunay triangulation on the matching point set, and count the number of all triangles in the triangulation network S1, initializing i = 1; Step 8.2: If S=0, go to step 8.5, otherwise go to step 8.2; Step 8.3: Select the i-th triangle. If the area of ​​the i-th triangle is less than the specified threshold S2, repeat steps 5 to 7. Otherwise, go to step 8.

4. Step 8.4: Let i = i + 1. If i ≤ S1, go to step 8.2; otherwise, go to step 8.

1. Step 8.5: The iteration stops and the final dense matching point set is obtained.

Images