A multi-view three-dimensional reconstruction method based on a hybrid structure
By employing a hybrid structure propagation and view selection strategy in the multi-view 3D reconstruction algorithm, the problem of balancing reconstruction speed and quality in existing technologies is solved, and a faster 3D reconstruction process is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2023-11-14
- Publication Date
- 2026-07-07
Smart Images
Figure CN117496064B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of 3D reconstruction technology, and particularly relates to a multi-view method based on a hybrid structure. Figure 3 Dimensional reconstruction method. Background Technology
[0002] Given a set of multi-view scene images with camera parameters, multi-view Figure 3 3D reconstruction technology can recover the positions of most pixels in three-dimensional space and reconstruct the scene in an image. This technology can effectively assist computers in recognizing the three-dimensional world through two-dimensional images, and therefore has been widely used in fields such as autonomous driving and virtual reality, and is one of the research hotspots in the field of computer vision in recent years.
[0003] Dense point clouds based on depth map fusion are a commonly used representation of 3D reconstruction results. Due to the occlusion of objects in 3D space, the ambiguity in matching weakly textured regions in scene images, and the speed requirements of scene reconstruction in practical applications, accurately and quickly calculating the depth of each point in an image has become a major challenge for 3D reconstruction algorithms. Patch-Match-based reconstruction algorithms are currently the mainstream algorithms for multi-view stereo reconstruction. These algorithms transfer depth calculation from discrete space to continuous space, estimating depth through propagation, thus improving algorithm efficiency and the quality of reconstruction results. Patch-Match-based 3D reconstruction algorithms generally consist of two stages: the first stage uses photometric consistency as the matching cost to calculate the initial depth map of the input image; the second stage uses a cost function that combines photometric consistency and geometric consistency constraints to optimize depth estimation. However, when calculating the depth map, the red-black checkerboard pattern propagation method has high parallel computation efficiency but the corresponding view selection strategy has poor accuracy. Furthermore, the Markov chain-based view selection strategy can obtain more accurate matching results but has lower algorithm parallelism. Since both stages use the same propagation and view selection strategies, a balance between reconstruction quality and reconstruction speed cannot be achieved. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention provides a multi-view based on a hybrid structure. Figure 3 This dimensionality reconstruction method employs different propagation and view selection strategies in two phases. In the first phase, the algorithm uses row-column parallel propagation and leverages Markov chains for simultaneous view selection and depth estimation, providing better initial values for the subsequent phase. In the second phase, a chessboard-style fast propagation method is used, along with a voting-based view selection strategy to accelerate the reconstruction process. This algorithm balances the quality of depth estimation with computational time.
[0005] The technical solution adopted in this invention includes the following steps:
[0006] Step 1: Select an image from the input image set as a reference image, and perform downsampling operations on it for a preset number of times and a preset scale to obtain an image pyramid;
[0007] Step 2: Starting with the smallest size image at the top of the tower, calculate the depth map and normal map of the reference image;
[0008] Step 3: Upsample the depth map and normal map to the size of the next layer's large image, and use it as the initial value for calculating the depth map and normal map of the next layer's image. Calculate the depth map and normal map of the next layer's image.
[0009] Step 4: Repeat step 3 until the reference image is restored to its original size;
[0010] Step 5: Optimize the depth map and normal map again based on the geometric consistency of the image, and use a checkerboard propagation strategy to accelerate the process;
[0011] Step 6: Fuse the depth map and normal map of the image to generate a point cloud model.
[0012] Preferably, step 1 is described as follows:
[0013] If the preset downsampling number is k and the preset downsampling scale is 0.5, then the image pyramid establishes an image pyramid with k+1 layers, and the width and height of the top image are (0.5) of the width and height of the original image. k The dimensions of the tower base image are the same as those of the original image.
[0014] Preferably, step 2 is described as follows:
[0015] Step 2-1: Randomly initialize the depth map and normal map of the reference image, that is, assign a random depth θ and normal n to each pixel in the reference image;
[0016] Step 2-2: Based on the camera parameters of the reference image and the source image, calculate the corresponding region in the source image for the image patch at pixel l in the reference image, and then calculate the bilateral weighted NCC (Normalized CrossCorrelation) value of the two image patches. Finally, the optimal depth of pixel l is calculated. With optimal normal The formula is as follows:
[0017]
[0018] Where S is based on P l (m) Selected subset of source images, P l (m) represents the probability that the m-th source image at pixel l is selected.
[0019] Steps 2-3: Generate random depth and random normal Combined with the propagation depth of the previous pixel and normal n l-1 The depth perturbation value of the current pixel and normal perturbation value Generate an array of new depth normal hypotheses, forming the final list of depth normal hypotheses:
[0020]
[0021] Step 2-4: Calculate the matching cost for each pair of depth normal assumptions in the hypothesis list of Step 2-3, and select the depth normal value pair with the minimum cost as the new value at pixel l.
[0022] Preferably, step 3 is described as follows:
[0023] Since the downsampling preset scale is 0.5, during upsampling, the width and height of the depth map and normal map calculated in this layer need to be enlarged to twice their original size and passed to the next layer. In the calculation of the next layer, the depth map and normal map passed from this layer are used to replace the random initialization in step 2-1, and steps 2-2 to 2-4 are repeated to calculate the depth map and normal map for the next layer.
[0024] Preferably, step 5 is described as follows:
[0025] Step 5-1: Take the depth map and normal map of the original image size calculated in Step 3 as the initial depth and initial normal;
[0026] Step 5-2: Divide the image into a red and black checkerboard grid. For each pixel p, consider the four adjacent long strip regions and four V-shaped regions around it, and select the eight pixels with the lowest matching cost in these regions as sampling points. Calculate the matching cost COST between the reference image and all source images at the eight sampling points: COST = {COST...} n,m |n=1,2,...,8,m=1,2,...,M}, where M is the total number of source images. A constant threshold τ is set. up And a threshold τ(t) based on the iteration number t. If the matching cost of the Mth source image and the reference image has n1 sampling points whose cost is less than τ. up If the cost of n² sampling points is less than τ(t), then the source image is a matching view of the reference image at pixel p. Calculate the confidence C(COST) of each sampling point in each matching view. n,mThe mean confidence score of each sample point is taken as the weight of the matching view. The weighted matching cost of all sample points for the matching view is calculated, and the sample point with the smallest weighted matching cost is selected as the final sample point.
[0027] Step 5-3: Generate random depth and random normal Combining the propagation depth of the pixel values at the sampling points selected in step 5-2 and normal n sample The depth perturbation value of the current pixel and normal perturbation value Generate an array of new depth normal hypotheses, forming the final list of depth normal hypotheses:
[0028]
[0029] Step 5-4: Calculate the matching cost for each pair of depth normal assumptions in the hypothesis list of Step 5-3, and select the depth normal value pair with the minimum cost as the new value at pixel l for the view weight.
[0030] The beneficial effects of this invention are as follows:
[0031] This invention employs different propagation methods and view selection strategies in the two stages of 3D reconstruction, improving computational speed while maintaining the quality of the reconstruction results. In the first stage, which requires accurate initial values, the chain-like propagation method, Markov chain-based view selection, and multi-level depth calculation achieve initial values with high accuracy and completeness. The second stage focuses on accelerating reconstruction speed. Geometric consistency is incorporated into the cost function to optimize estimation, and a chessboard-style propagation method and a voting-based view selection highly parallel computation strategy are adopted to accelerate the computation process in this stage and improve efficiency. Attached Figure Description
[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0033] Figure 1 This is a flowchart of the three-dimensional reconstruction method provided in the embodiments of the present invention;
[0034] Figure 2 This is a flowchart illustrating the specific implementation of the three-dimensional reconstruction method provided in this embodiment of the invention.
[0035] Figure 3 This is an example of a checkerboard sampling area provided in an embodiment of the present invention;
[0036] Figure 4 This is a scene reconstruction result image of the ETH3D dataset provided in an embodiment of the present invention. Detailed Implementation
[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.
[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.
[0039] This embodiment provides a multi-view based on a hybrid structure. Figure 3 3D reconstruction methods. For example... Figure 1 and Figure 2 As shown, the method mainly includes the following steps:
[0040] (1) Select an image from the input image set as a reference image, and perform downsampling operations on it for a preset number of times and a preset scale to obtain an image pyramid;
[0041] In practice, if the preset downsampling number is k and the preset downsampling scale is 0.5, then the image pyramid will have k+1 layers, and the width and height of the top image will be (0.5) of the width and height of the original image. k The dimensions of the tower base image are the same as those of the original image.
[0042] (2) Starting from the smallest size image at the top of the tower, calculate the depth map and normal map of the reference image;
[0043] The specific implementation involves the following steps:
[0044] (2.1) First, the depth map and normal map of the reference image are randomly initialized, that is, each pixel in the reference image is assigned a random depth θ and normal n;
[0045] (2.2) Based on the camera parameters of the reference image and the source image, calculate the corresponding region in the source image of the image patch at pixel l in the reference image, and then calculate the bilateral weighted NCC (Normalized Cross Correlation) value of the two image patches. As a matching cost function.
[0046] Define a visibility label for the corresponding region of the m-th source image at pixel l in the reference image. If the m-th source image can provide a matching region for pixel l without occlusion interference, then The value is 1 if it is 1, otherwise it is 0.
[0047] The process of finding the optimal depth θ, normal N, and label set Z for all images X can be modeled as maximizing a posterior probability:
[0048]
[0049] Where the depth set θ = {θ l |l=1,2,...,L}, normal set N={n l |l=1,2,...,L}, the set of labels M represents the number of source images, and L represents the number of pixels in the reference image.
[0050] Based on variational inference to solve for the posterior function, P(Z,θ,N|X) can be approximated as q(Z,θ,N)=q(Z)q(θ,N), and the optimal Z, θ, and N are inferred using the GEM (Generalized Expectation Maximization) algorithm. The GEM algorithm is an iterative optimization algorithm that alternates between E-step and M-step optimization. In the E-step, q(θ) l ,n l ) is fixed as a constant, calculation Then obtain the probability distribution P of the view being selected. l (m):
[0051]
[0052] During M steps, fix Propagate the depth value of the previous pixel to pixel l, and update q(θ). l ,n l ), calculate the optimal depth of pixel l With optimal normal
[0053]
[0054] (2.3) Generate random depth and random normal Combined with the propagation depth of the previous pixel and normal n l-1 The depth perturbation value of the current pixel and normal perturbation value Generate an array of new depth normal hypotheses, forming the final list of depth normal hypotheses:
[0055]
[0056] (2.4) Calculate the matching cost for each pair of depth normal assumptions in the assumption list of step (2.3), and select the depth normal value pair with the minimum cost as the new value at pixel l.
[0057] (3) Upsample the depth map and normal map to the size of the next layer large image, and use it as the initial value for calculating the depth map and normal map of the next layer image. Calculate the depth map and normal map of the next layer image.
[0058] In practice, since the downsampling preset scale is 0.5, the width and height of the depth map and normal map calculated in this layer need to be enlarged to twice the original size during upsampling and then passed to the next layer. In the calculation of the next layer, the depth map and normal map passed from this layer are used to replace the random initialization in step (2.1), and steps (2.2) to (2.4) are repeated to calculate the depth map and normal map of the next layer.
[0059] (4) Repeat step (3) until the reference image is restored to its original size;
[0060] (5) Optimize the depth map and normal map again based on the geometric consistency of the image, and use a checkerboard propagation strategy to accelerate the process;
[0061] like Figure 3 As shown, the specific implementation involves the following steps:
[0062] (5.1) Take the depth map and normal map of the original image size calculated in step (3) as the initial depth and initial normal;
[0063] (5.2) Divide the image into a red and black checkerboard grid. For each pixel p, consider the four adjacent long strip regions and four V-shaped regions around it, and take the eight pixels with the lowest matching cost in the above regions as sampling points.
[0064] Calculate the matching cost COST between the reference image and all source images at 8 sampling points. n,m |n=1,2,...,8,m=1,2,...,M}, where M is the total number of source images.
[0065] Set a constant threshold τ up And the change threshold τ(t) based on the iteration number t. Where τ up =1.2, the formula for calculating τ(t) is as follows:
[0066]
[0067] Where τ init =1.0, t is the number of iterations for the depth normal hypothesis propagation, and α is a constant coefficient, α = 90.
[0068] If the matching cost between the m-th source image and the reference image has n1 sampling points whose costs are less than τ. up If the cost of n2 sampling points is less than τ(t), then the source image is a matching view of the reference image at pixel p.
[0069] Where the thresholds n1 = 2 and n2 = 3. That is, if the cost of two sampling points in the source image is less than 1.2 and the cost of three sampling points is less than τ(t), then the source image is considered to be a matching view of the reference image at pixel p.
[0070] After obtaining all matching views, calculate the confidence C(COST) of each sampling point in each matching view. n,m ):
[0071]
[0072] Where β is a constant coefficient, β = 0.4.
[0073] The average confidence score of the 8 sampling points is used as the weight of the matched view. To minimize the impact of incorrectly matched views, only the top 5 views with the highest weights are included in subsequent calculations, and the weights of the remaining views are set to 0.
[0074] For the nth sampling point, the weighted matching cost of the sampling point in the matching view is:
[0075]
[0076] Where ψ(X) m ) represents the view weight of the w-th matching view.
[0077] The sampling point with the minimum weighted matching cost is selected as the final sampling point.
[0078] (5.3) Generate random depth and random normal Combining the propagation depth of the pixel values at the sampling points selected in step (5.2) and normal n sample The depth perturbation value of the current pixel and normal perturbation value Generate an array of new depth normal hypotheses, forming the final list of depth normal hypotheses:
[0079]
[0080] (5.4) Calculate the matching cost for each pair of depth normal hypotheses in the hypothesis list, and select the depth normal value pair with the minimum cost as the new value at pixel p for the view weight.
[0081] (6) The depth map and normal map of the image are fused to generate a point cloud model.
[0082] Table 1 presents the evaluation results of the absolute error between the final depth map calculated from the Strecha dataset and the ground truth. The table shows the percentage of pixels with error thresholds within 2cm and 10cm, with the best results highlighted in bold. It can be seen that the algorithm presented in this paper achieves the highest percentage of pixels with an error within 2cm in the HerzJesu-P8 scene; the remaining results are slightly lower than COLMAP but higher than ACMH.
[0083] Table 1
[0084]
[0085] like Figure 4 As shown in Table 2, the results of point cloud reconstruction for the ETH3D dataset are evaluated. The table displays the accuracy (Acc.), completeness (Comp.), and F1 score at evaluation thresholds of 5cm and 10cm. The best results are shown in bold. Considering all scenarios, the point cloud results obtained by the algorithm in this paper are slightly higher than ACMH in accuracy and slightly higher than COLMAP in completeness at a 5cm error. At a 10cm error, the point cloud obtained by the algorithm in this paper shows improved completeness compared to both COLMAP and ACMH, and the F1 score also shows a slight increase compared to COLMAP and ACMH, increasing by 0.6% and 0.9% respectively.
[0086] Table 2
[0087]
[0088] Table 3 shows the runtime evaluation of scene reconstruction on the ETH3D dataset. As can be seen from the table, the speedup of the algorithm in this paper compared to the COLMAP algorithm can be consistently maintained at around 2.5 times or even higher.
[0089] Table 3
[0090]
[0091] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.
[0092] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.
Claims
1. A multi-view 3D reconstruction method based on a hybrid structure, characterized in that, Includes the following steps: (1) Select an image from the input image set as a reference image, and perform downsampling operations on it for a preset number of times and a preset scale to obtain an image pyramid; (2) Starting from the smallest size image at the top of the tower, calculate the depth map and normal map of the reference image; (3) Upsample the depth map and normal map to the size of the next layer large image, and use them as the initial values for calculating the depth map and normal map of the next layer image. Calculate the depth map and normal map of the next layer image. (4) Repeat step (3) until the reference image is restored to its original size; (5) Optimize the depth map and normal map again based on the geometric consistency of the image, and use a checkerboard propagation strategy for acceleration; as follows: (5.1) Take the depth map and normal map of the original image size calculated in step (3) as the initial depth and initial normal; (5.2) Divide the image into a red and black checkerboard grid. For each pixel p, consider the four adjacent long strip regions and four V-shaped regions around it, and take the eight pixels with the lowest matching cost in the above regions as sampling points. Calculate the matching cost COST between the reference image and all source images at 8 sampling points. n,m | n = 1, 2,..., 8, m = 1, 2, ..., M}, where M is the total number of source images; a constant threshold τ is set. up and the change threshold τ based on the number of iterations t t If, in the matching cost between the Mth source image and the reference image, the cost of n1 sampling points is less than τ. up And the cost of having n2 sampling points is less than τ t If the source image is a matching view of the reference image at pixel p, then the source image is a matching view of the reference image. Calculate the confidence C(COST) of each sampling point in each matching view. n,m The mean of the confidence scores of the sample points is taken as the weight of the matched view. Calculate the weighted matching cost of all sampling points for the matching view, and select the sampling point with the smallest weighted matching cost as the final sampling point; (5.3) Generate random depth and random normal Combined with the propagation depth of the pixel value at the sampling point selected in step (5.2) and legal direction The depth perturbation value of the current pixel and normal perturbation value This generates an array of new depth normal hypotheses, forming the final list of depth normal hypotheses. ； (5.4) Calculate the matching cost for each pair of depth normal assumptions in the assumption list of step (5.3), and select the depth normal value pair with the minimum cost as the new value at pixel l for the view weight; (6) The depth map and normal map of the image are fused to generate a point cloud model.
2. The multi-view 3D reconstruction method based on a hybrid structure according to claim 1, characterized in that, Step (1) specifically involves: when the preset downsampling number is k and the preset downsampling scale is 0.5, an image pyramid with k+1 layers is established, and the width and height of the top image are (0.5) of the width and height of the original image. k The dimensions of the tower base image are the same as those of the original image.
3. The multi-view 3D reconstruction method based on a hybrid structure according to claim 1, characterized in that, Step (2) specifically includes the following sub-steps: (2.1) Randomly initialize the depth map and normal map of the reference image, that is, assign a random depth θ and normal n to each pixel in the reference image; (2.2) Based on the camera parameters of the reference image and the source image, calculate the corresponding region of the image patch at pixel l in the reference image in the source image, and then calculate the bilateral weighted NCC value of the two image patches. Finally, calculate the optimal depth of pixel l. With optimal normal The formula is as follows: ； Where S is based on P l (m) Selected subset of source images, P l (m) represents the probability that the m-th source image at pixel l is selected; (2.3) Generate random depth and random normal Combined with the propagation depth of the previous pixel and legal direction The depth perturbation value of the current pixel and normal perturbation value This generates an array of new depth normal hypotheses, forming the final list of depth normal hypotheses. ； (2.4) Calculate the matching cost for each pair of depth normal assumptions in the assumption list of step (2.3), and select the depth normal value pair with the minimum cost as the new value at pixel l.
4. The multi-view 3D reconstruction method based on a hybrid structure according to claim 3, characterized in that, Since the downsampling preset scale is 0.5, when upsampling, the width and height of the depth map and normal map calculated in this layer need to be enlarged to twice and passed to the next layer. In the calculation of the next layer, the depth map and normal map passed in this layer are used to replace the random initialization in step (2.1), and steps (2.2) to (2.4) are repeated to calculate the depth map and normal map of the next layer.