Incremental three-dimensional sparse reconstruction method based on mutual voting matching

Abstract

The invention relates to an incremental three-dimensional sparse reconstruction method based on mutual voting matching, and belongs to the field of computer vision. Comprising the steps of extracting pixel-level features of all images in an image sequence; performing similarity calculation on the images by means of a public vocabulary tree to obtain a similarity descending order set of each image and all other images, and constructing a similarity graph by taking the images as nodes and the similarity as edge weights; performing mutual voting matching of edges and points on the similarity graph to obtain a matching graph with matching information; and performing incremental reconstruction according to the matching graph. According to the method, the matching stage and the overall sparse reconstruction efficiency can be obviously improved on a large-scale data set, and compared with an original vocabulary tree method, the average reprojection error and the image reconstruction point number are increased.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically relates to an incremental three-dimensional sparse reconstruction method based on mutual voting matching, and mainly focuses on the efficient sparse reconstruction of unordered optical image sequences of static scenes. Background Art

[0002] Structure from motion (SfM) is a process of reconstructing a 3D structure by estimating the camera pose based on the projection of a series of images taken from different viewpoints. Since the resulting 3D point cloud is relatively sparse, this process is also called sparse reconstruction. There are several SfM strategies, including incremental, global, hierarchical, progressive, and hybrid. However, due to the requirements for robustness and stability in practical applications, incremental SfM is currently the most popular strategy. Incremental SfM is a sequential processing pipeline for iterative reconstruction. It usually starts with feature extraction and then performs feature matching. The resulting view map (matching map) is the basis of the reconstruction phase, which uses a pair of images selected by the algorithm to reconstruct and generate an initial model. The new images are then registered one by one, and the 3D points are triangulated and outliers are filtered. At the same time, the reconstruction is improved using bundle adjustment, and a relatively complete 3D model is finally obtained.

[0003] Over the past few decades, structure from motion has been an important topic in the field of 3D vision. Due to the robustness of structure from motion and the ability to estimate relatively accurate camera poses and 3D models of scenes using only optical image sequences, 3D sparse reconstruction techniques have been widely used, such as reconstructing landmarks using Internet data.

[0004] However, with the continuous improvement of image resolution and the application of high-dimensional feature descriptors, sparse reconstruction algorithms face huge computational burden when processing large-scale datasets. Summary of the invention

[0005] The technical problems to be solved by the present invention are:

[0006] In order to avoid the shortcomings of the prior art, the present invention provides an incremental three-dimensional sparse reconstruction method based on mutual voting matching, which is used to solve the huge computational burden faced by the existing sparse reconstruction algorithms when processing large-scale data sets.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] An incremental three-dimensional sparse reconstruction method based on mutual voting matching, characterized by comprising the following steps:

[0009] S1: Obtain optical image sequences of static scenes at different viewing angles and extract pixel-level features of all images in the image sequence;

[0010] S2: Calculate the similarity of images with the help of the common vocabulary tree to obtain a descending set of similarities between each image and all other images; construct a similarity graph with images as nodes and similarities as edge weights;

[0011] S3: Perform mutual voting matching of edges and points based on the similarity graph to obtain a matching graph with matching information;

[0012] S4: Incremental reconstruction based on the matching graph with matching information.

[0013] A further technical solution of the present invention: Extracting pixel-level features of all images in the image sequence as described in S1 includes:

[0014] S11: Assuming that the number of images in the sequence is N, calculate the difference pyramid of the kth image;

[0015] S12: Normalize the difference pyramid to obtain the SIFT 128-dimensional features of the image;

[0016] S13: Repeat the above steps S11 and S12 until all images have completed feature extraction.

[0017] A further technical solution of the present invention: S2 calculates the similarity of images by using a common vocabulary tree to obtain a descending set of similarities between each image and all other images, including:

[0018] S21: Use the public vocabulary tree to search each image. For each feature of the k-th image, a bag-of-words descriptor can be obtained after retrieval. The bag-of-words descriptors of all features of the k-th image are aggregated to obtain the image descriptor D k ;

[0019] S22: Repeat S21 to calculate the descriptor set D of all images = {D k , k∈{1,2,3,...,N}};

[0020] S23: Calculate the similarity between images, which is the cosine similarity of image descriptors:

[0021]

[0022] Set the number of searches to N R , get the kth image and other N R The similarity descending set S of images k ={S ij ,i,j∈{1,2,3,...,N R}}, k∈{1,2,3,...,N}.

[0023] A further technical solution of the present invention: The mutual voting matching of edges and points based on the similarity graph in S3 to obtain a matching graph with matching information includes:

[0024] S31: Based on the similarity graph, edge-to-point voting is performed, and the weight W of point i is i :

[0025]

[0026] S32: After obtaining the point weight, vote for the edge, and the weight W of the edge ij ij :

[0027] W ij =S ij +∑W x ,i,j∈{1,2,3,...,N},

[0028] Where x is the third point that can form a three-membered ring with points i and j;

[0029] S33: Take the edges with weights higher than the threshold λ to form a view, check the connectivity of the graph, and retrieve some necessary edges to make the graph fully connected;

[0030] S34: Match the image pairs on the view at the feature point level to obtain a matching graph containing matching information.

[0031] A further technical solution of the present invention: the incremental reconstruction based on the matching graph with matching information described in S4 comprises:

[0032] S41: Select an image with the most associated images in the matching as the first image, select the image with the most matches in the associated images of this image as the second image, solve the relative posture of the camera with these two images, perform triangulation on the matching points to obtain the corresponding three-dimensional point cloud, and finally use the bundle adjustment to optimize the camera posture and the three-dimensional point cloud model. If the reconstruction is successful, it is the initial image pair and proceed to the next step, otherwise continue to search according to the number of associated images;

[0033] S42: adding a new picture for reconstruction, this process is called image registration; based on the feature points existing in both the existing and the newly added perspectives, the parameters of the newly added camera can be solved using the PnP algorithm;

[0034] S43: triangulate the new feature points to obtain three-dimensional points, and continuously improve the initial three-dimensional point cloud model;

[0035] S44: Perform local bundle adjustment and outlier filtering;

[0036] S45: Repeat S42-S44 until the last picture to obtain the final sparse point cloud model.

[0037] A computer system, characterized in that it comprises: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0038] A computer-readable storage medium, characterized in that it stores computer-executable instructions, which are used to implement the above method when executed.

[0039] A computer program product, characterized by comprising computer executable instructions, wherein the instructions are used to implement the above method when executed.

[0040] The beneficial effects of the present invention are:

[0041] The present invention provides an incremental three-dimensional sparse reconstruction method based on mutual voting matching. First, feature extraction is performed on all two-dimensional images. Then, similarity scores between images are calculated using vocabulary tree retrieval, a similarity graph is constructed, and edge-to-point and point-to-edge mutual voting is performed on the graph. Then, the matching graph is pruned based on this to speed up the matching process. Finally, incremental reconstruction is performed based on the pruned view graph.

[0042] 1. The method of the present invention is highly efficient, and can significantly improve the efficiency of the matching stage and overall sparse reconstruction, especially on large-scale data sets.

[0043] 2. The results of the method of the present invention are more accurate. Compared with the original vocabulary tree method, the method of the present invention improves the average reprojection error and the number of image reconstruction points. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0045] Figure 1 Flow chart of the method of the present invention.

[0046] Figure 2 A sequence of optical images from different perspectives of the same static scene to be reconstructed.

[0047] Figure 3 Feature points of optical image sequence.

[0048] Figure 4 Similarity graph example.

[0049] Figure 5 Schematic diagram of peer-to-peer voting.

[0050] Figure 6 Schematic diagram of point-to-edge voting.

[0051] Figure 7 View diagram example.

[0052] Figure 8 Feature point matching example.

[0053] Fig. 9 Example of the final result.

[0054] Fig.10 Compare the final results. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0056] An incremental three-dimensional sparse reconstruction method based on mutual voting matching is provided in an embodiment of the present invention, and its process is as follows: Figure 1 As shown, it includes: extracting pixel-level features of all images in the image sequence; calculating the similarity of images with the help of a common vocabulary tree to obtain a descending set of similarities between each image and all other images, constructing a similarity graph with images as nodes and similarities as edge weights; performing mutual voting matching of edges and points on the similarity graph to obtain a view graph with matching information; and performing incremental reconstruction based on the view graph. The details are as follows:

[0057] (1) First, obtain the optical image sequence of the static scene at different viewing angles, such as Figure 2 As shown, the scale invariant feature points (Scale Invariant Feature Transform, SIFT) of the image are then extracted, as Figure 3 .

[0058] (2) Use the public vocabulary tree to search each image. For each feature of the k-th image, a bag-of-words descriptor can be obtained after retrieval. The bag-of-words descriptors of all features of the k-th image are aggregated to obtain the image descriptor D k . Calculate the descriptor set D of all images = {D k , k∈{1,2,3,...,N}}. The cosine similarity of image descriptors is used as the similarity between images. Set the number of retrievals to N R , get the kth image and other N R The similarity descending set of images {Sij ,i,j∈{1,2,3,...,N R}}, k∈{1,2,3,...,N}. A similarity graph is constructed with images as nodes and similarity as edge weights, such as Figure 4 .

[0059] (3) Perform mutual voting matching based on the similarity graph. First, perform edge-to-point voting, that is, take the average of a certain number of similarities around it as the weight of the point, such as Figure 5 The voting diagram for point A is shown below. Then the point-to-edge voting is performed, that is, the weight of the third point that can form a ternary ring with the edge is added on the basis of the similarity, such as Figure 6 The figure below is a voting diagram for edge AB, which adds the weights of C and D to the similarity of AB. After pruning, the view graph is checked for connectivity, and some necessary edges are retrieved to make the graph fully connected (appropriately select discarded edges with higher weights for retrieval). The final view diagram is as follows: Figure 7 . And perform feature point level matching on this image, such as Figure 8 .

[0060] (4) Select a pair of images with a wide enough baseline and enough feature points on the view with matching information for initial reconstruction, solve the relative pose of the camera, triangulate the matching points to obtain the corresponding three-dimensional point cloud, and finally use the bundle adjustment to optimize the camera pose and three-dimensional point cloud model. Then add new pictures for reconstruction. This process is called image registration. Based on the feature points that exist in both the existing perspective (camera) and the newly added perspective (camera), the parameters of the newly added camera can be solved using the PnP algorithm. Triangulate the new feature points to obtain three-dimensional points, and continuously improve the initial three-dimensional point cloud model. Each new image will undergo local bundle adjustment and outlier filtering. After trying the last picture, the final sparse point cloud model is obtained, such as Fig. 9 , where the triangular cone represents the estimated camera pose and the vertex of the triangular cone is the optical center.

[0061] The method of the present invention is applied to actual 3D sparse reconstruction, and the effect is compared with the sparse reconstruction method based on traditional vocabulary tree matching. Fig.10 As shown, it can be seen that the three-dimensional structure of the method of the present invention is more accurate.

[0062] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. An incremental three-dimensional sparse reconstruction method based on mutual voting matching, characterized in that: The following steps are involved: S1: Obtain optical image sequences of static scenes at different viewing angles and extract pixel-level features of all images in the image sequence; S2: Calculate the similarity of images with the help of the common vocabulary tree to obtain a descending set of similarities between each image and all other images; construct a similarity graph with images as nodes and similarities as edge weights; S3: Perform mutual voting matching of edges and points based on the similarity graph to obtain a matching graph with matching information; S4: Incremental reconstruction based on the matching graph with matching information.

2. The incremental three-dimensional sparse reconstruction method based on mutual voting matching according to claim 1, characterized in that: S1 extracts pixel-level features of all images in the image sequence, including: S11: Assuming that the number of images in the sequence is N, calculate the difference pyramid of the kth image; S12: Normalize the difference pyramid to obtain the SIFT 128-dimensional features of the image; S13: Repeat the above steps S11 and S12 until all images have completed feature extraction.

3. The incremental three-dimensional sparse reconstruction method based on mutual voting matching according to claim 1, characterized in that: S2 calculates the similarity of the images with the help of the common vocabulary tree to obtain a descending set of similarities between each image and all other images, including: S21: Use the public vocabulary tree to search each image. For each feature of the k-th image, a bag-of-words descriptor can be obtained after retrieval. The bag-of-words descriptors of all features of the k-th image are aggregated to obtain the image descriptor D k ; S22: Repeat S21 to calculate the descriptor set D of all images = {D k , k∈{1,2,3,...,N}}; S23: Calculate the similarity between images, which is the cosine similarity of image descriptors: Set the number of searches to N R , get the kth image and other N R The similarity descending set S of images k ={S ij ,i,j∈{1,2,3,...,N R }}, k∈{1,2,3,...,N}.

4. The incremental three-dimensional sparse reconstruction method based on mutual voting matching according to claim 1, characterized in that: S3 performs mutual voting matching of edges and points based on the similarity graph to obtain a matching graph with matching information, including: S31: Based on the similarity graph, perform edge-to-point voting, and the weight W for point i i : S32: After obtaining the point weight, vote for the edge, and the weight W of the edge ij ij : W ij =S ij +∑W x ,i,j∈{1,2,3,...,N}, Where x is the third point that can form a three-membered ring with points i and j; S33: Take the edges with weights higher than the threshold λ to form a view, check the connectivity of the graph, and retrieve some necessary edges to make the graph fully connected; S34: Match the image pairs on the view at the feature point level to obtain a matching graph containing matching information.

5. The incremental three-dimensional sparse reconstruction method based on mutual voting matching according to claim 1, characterized in that: The incremental reconstruction based on the matching graph with matching information in S4 includes: S41: Select an image with the most associated images in the matching as the first image, select the image with the most matches in the associated images of this image as the second image, solve the relative posture of the camera with these two images, perform triangulation on the matching points to obtain the corresponding three-dimensional point cloud, and finally use the bundle adjustment to optimize the camera posture and the three-dimensional point cloud model. If the reconstruction is successful, it is the initial image pair and proceed to the next step, otherwise continue to search according to the number of associated images; S42: adding a new picture for reconstruction, this process is called image registration; based on the feature points existing in both the existing and the newly added perspectives, the parameters of the newly added camera can be solved using the PnP algorithm; S43: triangulate the new feature points to obtain three-dimensional points, and continuously improve the initial three-dimensional point cloud model; S44: Perform local bundle adjustment and outlier filtering; S45: Repeat S42-S44 until the last picture to obtain the final sparse point cloud model.

6. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

7. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

8. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images