An Error-Resistant View Selection Method for 3D Reconstruction

Abstract

The present invention discloses an error-resistant view selection method for 3D reconstruction, belonging to the field of computer vision technology. Aiming at the problem that the uncertainty of triangulation increases due to the small camera baseline selected during the view selection of SfM, the present invention proposes an error-resistant view selection method. This method obtains a calculation error-resistant model based on the triangle of binocular ranging, and then constructs an error-resistant matrix. The candidate view sets of each view are determined according to the sorting results of each row of the error-resistant matrix. All the candidate view sets of all views are traversed, and the missing views are complemented into the corresponding candidate view sets according to the error-resistant value matrix. This method proposes a view selection method with the triangulation error-resistant value of the camera baseline as an index. This method can solve the problem of increased uncertainty of triangulation caused by selecting a small camera baseline, and ensure that all views can participate in 3D reconstruction, improving the accuracy of 3D reconstruction.

Description

Technical Field

[0001] The present invention relates to an error-resistant view selection method for 3D reconstruction, belonging to the technical field of computer vision. Background Art

[0002] View selection refers to determining the correlation degree of image set views in the Structure-from-Motion (SfM) method, taking the correlated views as the next views to be reconstructed for matching, and determining the reconstruction order therefrom. It is a key technology that has a dual impact on accuracy and speed in multi-view 3D reconstruction and is also an indispensable key strategy in SfM. In 3D reconstruction, due to different camera baselines when selecting views, the range of triangulation errors is different, which in turn affects the accuracy of 3D point recovery in the triangulation stage. The law of the effect of the camera baseline on the triangulation error is as follows: the range of triangulation errors decreases sharply as the baseline increases, and after reaching the minimum point, the growth of the error range is slow. Figure 3 When the classical SfM method determines the next view to be reconstructed based on the content similarity or feature matching results of the views during view selection, it tends to select views with smaller camera baselines, resulting in an increase in the range of triangulation errors and thus reducing the 3D reconstruction accuracy. To avoid selecting views with smaller camera baselines during view selection, some SfM methods set a threshold for the camera baseline, and only views that meet the threshold can participate in the reconstruction. However, in the law of the effect of the camera baseline on the triangulation error, the minimum error point is different for different data sets, so a fixed threshold is difficult to meet various reconstruction environments. Therefore, setting a threshold cannot fundamentally solve the problem of the decrease in reconstruction accuracy caused by selecting a camera baseline with a smaller error. Currently, a better solution is the graph theory-based method, which evaluates the final reconstruction quality by optimizing the cost function regarding the camera baseline, and then determines the matching relationship and reconstruction order of each view; however, this method requires calculating the reconstruction accuracy for the remaining views every time when selecting the next view to be reconstructed, which increases the computational complexity of the view selection process and seriously reduces the operation efficiency. Therefore, based on the above analysis of the existing view selection methods, it is difficult for the existing view selection methods to simultaneously ensure the accuracy of 3D reconstruction and the computational efficiency of view selection.

[0003] When the classical SfM method determines the next view to be reconstructed based on the content similarity or feature matching results of the views during view selection, it tends to select views with smaller camera baselines, resulting in an increase in the range of triangulation errors and thus reducing the 3D reconstruction accuracy. To avoid selecting views with smaller camera baselines during view selection, some SfM methods set a threshold for the camera baseline, and only views that meet the threshold can participate in the reconstruction. However, in the law of the effect of the camera baseline on the triangulation error, the minimum error point is different for different data sets, so a fixed threshold is difficult to meet various reconstruction environments. Therefore, setting a threshold cannot fundamentally solve the problem of the decrease in reconstruction accuracy caused by selecting a camera baseline with a smaller error. Currently, a better solution is the graph theory-based method, which evaluates the final reconstruction quality by optimizing the cost function regarding the camera baseline, and then determines the matching relationship and reconstruction order of each view; however, this method requires calculating the reconstruction accuracy for the remaining views every time when selecting the next view to be reconstructed, which increases the computational complexity of the view selection process and seriously reduces the operation efficiency. Therefore, based on the above analysis of the existing view selection methods, it is difficult for the existing view selection methods to simultaneously ensure the accuracy of 3D reconstruction and the computational efficiency of view selection. Summary of the Invention

[0004] In order to improve the computational efficiency of view selection in SfM technology and simultaneously ensure the accuracy of 3D reconstruction, the present invention provides an error-resistant view selection method for 3D reconstruction, and the technical solution is as follows:

[0005] The first object of the present invention is to provide an error-resistant view selection method for 3D reconstruction, including:

[0006] Step 1: Calculate the triangulation error value caused by a 1-pixel error in feature matching under the baseline of different cameras. Take the triangulation error value as an indication of the error resistance ability of the baseline in the triangulation stage. Calculate the triangulation error values under the baselines formed by pairwise correspondence of all views and form an error resistance value matrix.

[0007] Step 2: Sort each row in the error resistance value matrix in ascending order, and select the original indices of the first five elements in each row to obtain a candidate set of the next reconstruction views for each view.

[0008] Step 3: Using the error resistance value as an indicator, recursively process the next reconstruction views corresponding to each view, mark the recursively processed views, and finally the unmarked views are the missing views.

[0009] Step 4: Taking the missing view as the starting point of the baseline, find the corresponding baseline end point such that the baseline vector formed by the starting point to the end point has the minimum error resistance value. Add the missing view to the candidate set of the next reconstruction views of the end point view. The candidate sets of the next reconstruction views of all views after completion are the results of view selection.

[0010] Optionally, the process of calculating the error resistance value of the baseline and constructing the error resistance value matrix in Step 1 includes:

[0011] Given cameras O1 and O2 in a three-dimensional space, I1 and I2 are the corresponding physical imaging planes of cameras O1 and O2, P is a three-dimensional point restored according to the actual camera pose, P1 and P2 are the projection points of the three-dimensional point P on the physical imaging planes I1 and I2 respectively, e1 and e2 are epipoles, and f is the focal length.

[0012] Calculate that when P2 has a 1-pixel error on its epipolar line P2e2 due to the error in feature matching and becomes P2 ′ , calculate the error PP ′ of the triangulated three-dimensional point P ′ , and regard this error as the triangulation error resistance value of the baseline .

[0013] From the above, the coordinates of O1, O2, P and the direction of are known. Since the unit of the focal length f is pixel, its actual size is related to the true scale, and the influence on the size of all line segments is proportional. Therefore, setting any value can be ignored for the overall influence. Let the actual size of O2F be 1.

[0014] From the known point coordinates above, the following angles can be obtained using the cosine theorem: ∠PO2F, ∠O1O2F and ∠O1PO2. The relevant line segment lengths can be obtained through subsequent calculations, as shown in Equation (1):

[0015]

[0016] Calculate the baseline according to O2P2, O2e2, and e2P2 obtained above according to Equation (2). The triangulation anti-error value PP of the ′ , where γ = ∠O1PO2, α = ∠e2P2O2, β = ∠PO2O1, β′ = ∠PO2P′:

[0017]

[0018] Calculate the corresponding anti-error values for the n(n - 1) baseline vectors formed by the existing n views, and integrate the anti-error values corresponding to the n(n - 1) baseline vectors into an anti-error value matrix E;

[0019] The matrix E is a square matrix with n rows and n columns, where the value of E[i][j] represents the anti-error value of the baseline formed by camera i to camera j.

[0020] Optionally, the construction process of the candidate set of the next reconstruction view in step 2 includes:

[0021] View S i The set of anti-error values of the baselines formed with all views is E i = {E i1 , E i2 , …, E in}, where represents the anti-error value of view S i and view S n ; sorting the elements in the set E i in ascending order gives Then the set of the next views of view i is {S a1 , S a2 , S a3 , S a4 , S a5}.

[0022] Optionally, the steps of view checking and filling in the blanks in step 3 include:

[0023] Select the baseline with the smallest anti-error value among all baselines, regard the view corresponding to the starting point of this baseline as the initial view, and start recursion as the current view, mark the current view, and select the view with the smallest anti-error value from the set of the next views of the current view as the next view.

[0024] Optionally, the selection of the candidate set of the next reconstruction view includes the following two cases:

[0025] (1) If the next view has been marked, select the sub-optimal one as the next view from the set of the next views of the current view;

[0026] (2) If all the views in the set of the next views of this view have been marked, return to the current view of the previous iteration, and select the unmarked view from the set of the next views at this time as the next view;

[0027] Take the next view as the current view, and iterate the above steps until the current view returns to the initial view and all the views in its set of the next views have been marked, then the recursion ends;

[0028] At this time, the unmarked views are the views that have not participated in the reconstruction. For the view S that has not participated in the reconstruction x , find the minimum error-resistant baseline with the view as the baseline end point Take the view S x and add it to the set of the next views of the view.

[0029] Optionally, the diagonal of the matrix E represents the baseline from the camera i to the camera itself, and this baseline is meaningless. Set its value to the maximum value of the variable type so as not to affect the subsequent numerical arrangement.

[0030] The second object of the present invention is to provide a three-dimensional reconstruction method. First, use the error-resistant view selection method described in any one of the above to select the next view participating in the reconstruction, and then perform three-dimensional reconstruction according to the selected view.

[0031] The third object of the present invention is to provide a three-dimensional reconstruction system, including:

[0032] An image input module for acquiring an image set to be three-dimensionally reconstructed;

[0033] A view selection module for determining the reconstruction order of the acquired images. The view selection process includes using the above-mentioned error-resistant view selection method for three-dimensional reconstruction to select the next view participating in the reconstruction;

[0034] A three-dimensional reconstruction module for performing three-dimensional reconstruction according to the image reconstruction order determined by the view selection;

[0035] An output display module for outputting the three-dimensional scene result corresponding to the image according to the result of the three-dimensional reconstruction.

[0036] The fourth object of the present invention is to provide a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0037] The beneficial effects of the present invention are:

[0038] The present invention proposes an error-resistant view selection method to solve the problem of increased triangulation uncertainty caused by a small camera baseline when selecting views for SfM. This method obtains a calculation error-resistant model based on the triangle of binocular ranging, and then constructs an error-resistant matrix. The candidate view sets of each view are determined according to the sorting results of each row of the error-resistant matrix. All candidate view sets of the views are traversed, and the missing views are complemented into the corresponding candidate view sets according to the error-resistant value matrix. The error-resistant view selection method proposed by the present invention reduces the average reprojection error and absolute trajectory error in the reconstruction result, improves the three-dimensional reconstruction accuracy, and ensures the computational efficiency of view selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0040] Figure 1 It is a technical route framework diagram of an error-resistant view selection method for three-dimensional reconstruction according to an embodiment of the present invention.

[0041] Figure 2 It is a schematic diagram of triangulation error resistance in an embodiment of the present invention, where (a) is a triangulation model diagram and (b) is a triangulation error plane diagram.

[0042] Figure 3 It is a comparison effect diagram of the method of the present invention and the exhaustive method under the TUM dataset in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will further describe the embodiments of the present invention in detail with reference to the drawings.

[0044] Embodiment 1:

[0045] This embodiment provides an error-resistant view selection method for three-dimensional reconstruction. Refer to Figure 1 , the method includes:

[0046] Step 1: Calculate the triangulation error value caused by a 1-pixel error in feature matching under different camera baselines. The triangulation error value is regarded as an embodiment of the error-resistant ability of the baseline in the triangulation stage. Calculate the triangulation error values under the baselines formed by pairwise correspondence of all views, and form an error-resistant value matrix.

[0047] Step 2: Sort each row in the anti-error value matrix in ascending order, and select the original indices of the first five elements in each row to obtain the candidate set of the next reconstructed view for each view.

[0048] Step 3: Using the anti-error value as an indicator, recursively process the next reconstructed view corresponding to each view, mark the recursively processed views, and finally the unmarked views are the missing views.

[0049] Step 4: Taking the missing view as the starting point of the baseline, find the corresponding baseline end point such that the baseline vector formed by this starting point and the end point has the minimum anti-error value, and add the missing view to the candidate set of the next reconstructed view of the end point view. The candidate sets of the next reconstructed views of all views after completion are the results of view selection.

[0050] Embodiment 2:

[0051] This embodiment provides an anti-error view selection method for 3D reconstruction. Refer to Figure 1 , including the following steps:

[0052] Step 1: In the triangulation anti-error calculation, mainly calculate the triangulation error caused by a 1-pixel error in feature matching under different camera baselines, and use this error value as an indication of the anti-error ability of the baseline in the triangulation stage. Calculate the error values of the baselines formed by pairwise correspondence of all views and form an anti-error value matrix.

[0053] Step 1-1: The schematic diagram of the derivation of the triangulation anti-error value is as shown in Figure 2 . Among them Figure 2 (a) shows that O1 and O2 are two cameras, I1 and I2 are the physical imaging planes corresponding to cameras O1 and O2, P is the 3D point restored according to the actual camera pose, P1 and P2 are the projection points of the 3D point P on the physical imaging planes I1 and I2 respectively, e1 and e2 are the epipoles, and f is the focal length. Calculate the error PP ′ of the 3D point P ′ obtained by triangulation when P2 has a 1-pixel error on its epipolar line P2e2 due to the error in feature matching, that is, P2 ′ , and regard this error as the triangulation anti-error value of the baseline .

[0054] As known above, the coordinates of O1, O2, P and In the direction, since the unit of the focal length f is pixels, the corresponding actual size is related to the true scale, and the influence on the size of all line segments is proportional. Therefore, setting any value can ignore the influence on the whole. Let the actual size of O2F be 1. From the known point coordinates above, the following angles can be obtained using the cosine theorem: ∠PO2F, ∠O1O2F, and ∠O1PO2. The lengths of relevant line segments can be calculated subsequently, as shown in Equation (1).

[0055]

[0056] Based on O2P2, O2e2, and e2P2 obtained above, the baseline can be calculated according to Equation (2). The triangulation error resistance value PP ′ , where γ = ∠O1PO2, α = ∠e2P2O2, β = ∠PO2O1, and β′ = ∠PO2P′.

[0057]

[0058] Step 1-2: Based on the derivation of the error resistance value in Step 1-1, calculate the corresponding error resistance values for the n(n - 1) baseline vectors formed by the existing n views. Integrate the error resistance values corresponding to the n(n - 1) baseline vectors into a matrix E of error resistance values for convenient use in subsequent view selection. Matrix E is a square matrix with n rows and n columns, where the value of E[i][j] represents the error resistance value of the baseline formed from camera i to camera j. The diagonal of matrix E represents the baseline from camera i to itself, which is meaningless. Setting its value to the maximum value of the variable type will not affect the subsequent numerical arrangement.

[0059] Step 2: In the view selection part, sort each row in the error resistance value matrix in ascending order, and select the original indices of the first five elements in each row. Then, the candidate set of the next reconstruction view for each view can be obtained.

[0060] Step 3: Using the error resistance value as an index, recursively mark the views corresponding to each view among the candidate views. The views that are finally not marked are the missing views.

[0061] According to the error resistance value matrix as S i The view determines the next view set, which contains 5 views with the smallest error resistance values starting from the baseline of view S i As the baseline starting point. Represent this process with mathematical symbols: The set of error resistance values of the baselines formed by view S i and all views is E i ={E i1 ,E i2 ,…,E in}, where represents view S i and view Sn anti-error value. For the set E i Sorting the elements in ascending order gives Then the next view set of view i is {S a1 , S a2 , S a3 , S a4 , S a5}.

[0062] Step 4: Add the missing views to the candidate view set corresponding to the baseline view with the smallest anti-error value associated with them, then the checking and filling of views can be completed. And the next view sets of all the completed views are the results of view selection;

[0063] Select the baseline with the smallest anti-error value among all baselines, regard the view corresponding to the starting point of this baseline as the initial view, and start recursion as the current view. Mark the current view, and select the view with the smallest anti-error value from the next view set of the current view as the next view. The following two special situations will occur in this step:

[0064] (1) If the next view has been marked, then select the sub-optimal one as the next view in the next view set of the current view.

[0065] (2) If all the views in the next view set of this view have been marked, then return to the current view of the previous iteration. Select the unmarked view in the next view set at this time as the next view.

[0066] Take the next view as the current view and iterate the above process. Recursion ends until the current view returns to the initial view and all the views in its next view set have been marked. At this time, the unmarked views are the views not participating in the reconstruction. For the view S x , find the baseline with the smallest anti-error value with the view as the end point of the baseline Add the view S x to the next view set of the view.

[0067] Based on the above specific implementation manners, the effects of the present invention are verified by the following specific experiments:

[0068] On the TUM dataset of indoor scenes, Figure 3 It shows that when the average reprojection error is used as the evaluation index, the result of the method of the present invention is significantly lower than that of the exhaustive method, indicating that the view selection method of the present invention has better precision in supporting the reconstruction of three-dimensional points by the SfM method.

[0069] This embodiment was completed under the PyCharm Professional 2020.1 software, with the main dependent libraries being the Opencv-python library version 4.5.3 and the Numpy library version 1.19.2. The hardware environment is a laptop with a 3.20GHz i7 processor and 16GB of running memory, and the experimental process is relatively stable.

[0070] Embodiment Three:

[0071] This embodiment provides a three-dimensional reconstruction system, including:

[0072] An image input module, used to obtain an image set to be three-dimensionally reconstructed;

[0073] A view selection module, used to determine the reconstruction order for the obtained images. The view selection process includes using the anti-error view selection method for three-dimensional reconstruction described in Embodiment One or Embodiment Two to select the next view to participate in the reconstruction;

[0074] A three-dimensional reconstruction module, used to perform three-dimensional reconstruction according to the image reconstruction order determined by the view selection;

[0075] An output display module, used to output the three-dimensional scene result corresponding to the image according to the result of the three-dimensional reconstruction.

[0076] Some steps in the embodiments of the present invention can be implemented using software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk, etc.

[0077] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. An error-resistant view selection method for 3D reconstruction, characterized in that, The method includes: Step 1: Calculate the triangulation error value caused by a 1-pixel error in feature matching under the baselines of different cameras. Use the triangulation error value to reflect the anti-error ability of the baseline in the triangulation stage. Calculate the triangulation error values under the baselines formed by pairwise correspondence of all views, and form an anti-error value matrix. Step 2: Sort each row in the anti-error value matrix in ascending order, and select the original indices of the first five elements in each row to obtain the candidate set of the next reconstruction view for each view. Step 3: Using the anti-error value as an index, recursively process the next reconstruction view corresponding to each view, mark the recursively processed views, and the finally unmarked views are the missing views. Step 4: Taking the missing view as the starting point of the baseline, find the corresponding baseline end point so that the baseline vector formed by the starting point to the end point has the minimum anti-error value. Add the missing view to the candidate set of the next reconstruction view of the end point view. The candidate set of the next reconstruction views of all views after completion is the result of view selection.

2. The anti-error view selection method for three-dimensional reconstruction according to claim 1, characterized in that The process of calculating the anti-error value of the baseline and constructing the anti-error value matrix in Step 1 includes: Given cameras O1 and O2 in three-dimensional space, I1 and I2 are the physical imaging planes corresponding to cameras O1 and O2, P is the three-dimensional point restored according to the actual camera pose, P1 and P2 are the projection points of the three-dimensional point P on the physical imaging planes I1 and I2 respectively, e1 and e2 are the epipoles, and f is the focal length. Calculate the error \(PP'\) of the 3D point \(P'\) obtained by triangulation when \(P_2\) has a 1-pixel error on its epipolar line \(P_2e_2\) due to the error of feature matching and becomes \(P'_2\), and regard this error as the triangulation anti-error value of the baseline. of the triangulation anti-error value; The coordinates of O1, O2, and P and the direction are known above. Since the unit of the focal length f is pixels, the corresponding actual size is related to the true scale, and the influence on the size of all line segments is proportional. Therefore, setting any value can ignore the impact on the whole. Then, let the actual size of O2F be 1; From the above known point coordinates, use the cosine theorem to find the following angles: ∠PO2F, ∠O1O2F, and ∠O1PO2. Subsequently, relevant line segment lengths can be calculated, as shown in Equation (1). The baseline is obtained according to Equation (2) from O2P2, O2e2, and e2P2 obtained above. The triangulation anti-error value PP′, where γ = ∠O1PO2, α = ∠e2P2O2, β = ∠PO2O1, β′ = ∠PO2P′: Calculate the corresponding anti-error values for the n(n - 1) baseline vectors formed by the existing n views, and integrate the anti-error values corresponding to the n(n - 1) baseline vectors into an anti-error value matrix E. Matrix E is a square matrix with n rows and n columns, where the value of E[i][j] represents the anti-error value of the baseline formed from camera i to camera j.

3. The method for selecting anti-error views for 3D reconstruction according to claim 2, wherein The process of constructing the candidate set of the next reconstruction view in Step 2 includes: View S i The set of anti-error values with respect to the baseline formed by all views is E i ={E i1 , E i2 , …, E in}, where represents the anti-error value between View S i and View S n ; sorting the elements in set E i in ascending order gives The next view set of view i is {S a1 , S a2 , S a3 , S a4 , S a5}.

4. The anti-error view selection method for three-dimensional reconstruction according to claim 1, wherein The steps of view checking and filling in Step 3 include: Select the baseline with the minimum anti-error value among all baselines. Regard the view corresponding to the starting point of this baseline as the initial view and start recursion as the current view. Mark the current view, and select the view with the minimum anti-error value from the next view set of the current view as the next view.

5. The method for anti-error view selection for 3D reconstruction according to claim 1, characterized in that, The selection of the candidate set of the next reconstruction view in Step 4 includes the following two cases: (1) If the next view has been marked, select the sub-optimal one as the next view in the next view set of the current view. (2) If all views in the next view set of this view have been marked, return to the current view of the previous iteration, and select the unmarked view in the next view set at this time as the next view. Take the next view as the current view, and iterate the above two cases until the current view returns to the initial view and all views in its next view set have been marked, then the recursion ends. At this time, the unmarked views are the views not participating in the reconstruction. For the view S that does not participate in the reconstruction x , find the minimum error-resistant baseline with the view as the baseline end point Take the view S x and add it to the next view set of views.

6. The method for selecting an anti-error view for three-dimensional reconstruction according to claim 2, wherein The diagonal of the matrix E represents the baseline from camera i to the camera itself, which is meaningless. Set its value to the maximum value of the variable type so as not to affect the subsequent numerical arrangement.

7. A three-dimensional reconstruction method, characterized in that, The three-dimensional reconstruction method first selects the next view to participate in the reconstruction by using the error-resistant view selection method according to any one of claims 1-6, and then performs three-dimensional reconstruction based on the selected view.

8. A three-dimensional reconstruction system, characterized in that, The system includes: An image input module for obtaining an image set to be three-dimensionally reconstructed; A view selection module for determining the reconstruction order of the obtained images. The view selection process includes selecting the next view to participate in the reconstruction by using the error-resistant view selection method according to any one of claims 1-6; A three-dimensional reconstruction module for performing three-dimensional reconstruction according to the image reconstruction order determined by the view selection; An output display module for outputting the three-dimensional scene result corresponding to the image according to the result of the three-dimensional reconstruction.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1-7 is implemented.

Images