Multi-view dense three-dimensional reconstruction method based on normal information assistance

Abstract

The invention relates to a multi-view dense three-dimensional reconstruction method based on normal information assistance, and belongs to the technical field of computer vision. The method comprises the following steps: starting from an input image sequence, firstly obtaining an initial hypothesis by using a multi-view reconstruction method based on adaptive checkerboard sampling, then constructing a planar prior model, and constraining the planar prior model by using normal information of the planar model so as to improve the effectiveness and correctness of the planar prior model, and further optimizing a result by using geometric consistency. And finally, in order to weaken the interference of noise points, filtering the hypotheses by adopting a dynamic filtering strategy, and finally fusing the hypotheses into a point cloud. According to the method, the three-dimensional geometry of an object or a scene can still be accurately recovered under the condition of facing a non-texture region, and the method comprises high-completeness recovery of the non-texture region.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a multi-view dense reconstruction method based on the assistance of normal information. Background Art

[0002] With the development of artificial intelligence (AI), image-based 3D reconstruction has become a highly promising field, with increasingly important real-life applications. It is widely used in video entertainment, virtual reality, virtual gaming, 3D printing, cultural relic reconstruction and preservation, autonomous driving, architectural surveying, smart homes, and other fields, significantly impacting people's daily lives.

[0003] Three-dimensional reconstruction methods generally fall into two categories: active and passive. Active techniques rely on capturing signals emitted by the target object itself to extract depth data, while passive techniques reconstruct the three-dimensional structure of the target object by interpreting its visual features in an image or video sequence. Compared to active methods, which are more hardware-dependent and expensive, passive methods can restore the three-dimensional information of a scene based on two-dimensional images from multiple perspectives acquired by ordinary consumer-grade cameras. Among them, image-based Multi-View Stereo (MVS) technology can restore the three-dimensional information of a target using a set of two-dimensional images of the target object and its camera parameters. Due to its low cost, simple and flexible operation, it has become the mainstream method of three-dimensional reconstruction technology.

[0004] Multi-view stereo vision technology can be divided into deep learning-based multi-view stereo vision technology and traditional geometric methods-based multi-view stereo vision technology. Learning-based multi-view stereo vision technology mainly uses the extremely powerful learning and fitting capabilities of deep neural networks to perform three-dimensional reconstruction of RGB or RGBD images. However, most of these methods are supervised learning methods, highly dependent on the dataset, and have weak generalization capabilities. The core of multi-view stereo vision technology based on traditional geometric methods is to use multiple images from different perspectives to estimate the position and pose of the camera. Subsequently, by extracting key features from these images and performing precise comparison and stitching, this technology can successfully convert the two-dimensional image data into an accurate three-dimensional model, achieving the transition from plane to stereo. However, the real world often contains a large number of weakly textured / untextured areas. These areas further lead to matching ambiguity problems in patch matching-based methods, ultimately resulting in poor reconstruction of textureless areas. Summary of the Invention

[0005] The technical problems to be solved by the present invention are:

[0006] In order to overcome the problems in the block matching-based multi-view geometry method, the present invention provides a multi-view dense 3D reconstruction method assisted by normal information, which can still accurately estimate the depth information of textureless areas that are difficult to reconstruct, and thus restore the 3D structure of objects or scenes.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A multi-view dense 3D reconstruction method based on normal information assistance, characterized by the following steps:

[0009] Step 1: Randomly select an image in the input image sequence as the reference image and the remaining images as the source images; and randomly initialize the reference image;

[0010] Step 2: Divide the pixels in the reference image into a red and black checkerboard grid. Combined with the adaptive checkerboard sampling strategy, each pixel in the reference image is set as the center pixel in turn, and the pixels in each of the eight regions around the pixel are used as candidate hypothesis solutions. The matching cost of the candidate hypotheses is then calculated to select the best hypothesis in each region for subsequent updates.

[0011] Step 3: Use the multi-view matching cost to estimate the weights of the best hypothesis selected in step 2 in different views. The hypothesis with the minimum multi-view matching cost will be used to update the center pixel.

[0012] Step 4: Generate random hypotheses and perturbation hypotheses to refine the solution space, and select the hypothesis with the lowest multi-view matching cost as the current hypothesis;

[0013] Step 5: Repeat steps 2 to 4 to make the current hypothesis converge;

[0014] Step 6: Build a plane prior model based on the current hypothesis, obtain the vertex normals of the plane prior model, calculate the mutual angles of the normals in the plane prior model, and select the plane prior model based on the threshold;

[0015] Step 7: Update the hypothesis using multi-view matching cost estimation assisted by normal-guided plane priors.

[0016] Step 8: Use geometric consistency-guided multi-view matching cost evaluation to further eliminate errors;

[0017] Step 9: Select images in the image sequence as reference images and generate hypotheses;

[0018] Step 10: Filter all hypotheses using a dynamic filtering strategy;

[0019] Step 11: Fuse all the filtered hypotheses to generate the output point cloud.

[0020] A further technical solution of the present invention: the random initialization process in step 1 includes:

[0021] Step 1-1: Randomly generate a depth information d for each pixel in the reference image x With normal information n x Assumption θ x =[d x ,n x ], this assumption can establish a three-dimensional plane for each pixel in the reference image, where d x is the distance from the three-dimensional plane to the origin, n x is the normal vector;

[0022] Step 1-2: From each source image, pass the homography matrix to the hypothesis θ randomly generated in step 1-1 x Calculate the matching cost. Specifically, given the camera parameters of the reference image and the camera parameters of the j-th source image, the homography between them is calculated as:

[0023]

[0024] Steps 1-3: Average the top 4 smallest costs to get the initial multi-view matching cost.

[0025] A further technical solution of the present invention: Step 2 is specifically as follows:

[0026] Step 2-1: Eight sampling areas are selected around the central pixel, including four long strip areas and four V-shaped areas. The long strip areas contain 11 sampling points, and the V-shaped areas contain 7 sampling points.

[0027] Step 2-2: Extended strategy;

[0028] Step 2-2-1: Select the best hypothesis in each region based on the multi-view matching cost of random initialization or the previous iteration, and define the candidate hypothesis set Θ = {θ i |i=1…8}; the initial cost in the random initialization is used for the first iteration, and subsequent iterations use the cost calculated in the previous iteration;

[0029] Step 2-2-2: Further use the eight structured hypotheses obtained by adaptive checkerboard sampling to infer the weight of each adjacent view and embed it into the cost matrix M:

[0030]

[0031] Where N represents the number of input images, m i,j represents the matching cost of the i-th region sampling point calculated by the j-th view;

[0032] Step 2-2-3: Based on the matrix M, a voting scheme is implemented in each column to determine whether a view is suitable. In addition, the matching cost of good views will decrease as the method iterates; the cost bound is defined as:

[0033]

[0034] Among them, τ0 represents the cost threshold of the initial matching, t represents the number of iterations, and α is a constant;

[0035] When the number of matching costs for each view that meets the conditions exceeds a certain threshold, the view is included in the current view selection set.

[0036] A further technical solution of the present invention is as follows: In order to evaluate the multi-view matching cost in step 3, the credibility of the matching cost is introduced, which is defined as:

[0037]

[0038] Where β is a constant, the weight of each view can be defined as:

[0039]

[0040] Where St is the view selection set, II(true)=1, II(false)=0;

[0041] Finally, the multi-view aggregated photometric consistency cost of a pixel is defined as:

[0042]

[0043] The hypothesis with the minimum multi-view matching cost will be selected as the best estimate of the current sample center pixel.

[0044] A further technical solution of the present invention: Step 4 specifically includes:

[0045] Step 4-1: Generate two additional hypotheses, one of which is obtained by perturbing the current hypothesis Another hypothesis is that it is generated randomly

[0046] Step 4-2: Two new hypotheses are randomly arranged and combined with the current hypothesis to form a set of seven hypotheses:

[0047]

[0048] Step 4-3: The final estimate of the sampled center pixel is the hypothesis with the minimum cost among the set in step 4-2.

[0049] A further technical solution of the present invention: the construction of the plane prior model in step 6 specifically includes:

[0050] Step 6-1: Collect reliable hypotheses with a multi-view matching cost less than 0.1 and put them into a set to form sparse reliable corresponding points;

[0051] Step 6-2: Use Delaunay triangulation to find three neighboring reliable hypotheses for the pixel points in the reference image that are not in the set of step 6-1 to adaptively generate triangular primitives of different sizes;

[0052] Step 6-3: Based on the assumptions in step 5, obtain the normals of the pixels at each vertex of the triangle primitive and calculate the angles between them:

[0053]

[0054] Among them, A(a x ,a y ,a z ) and B(b x ,b y ,b z ) is the known normal, and the unit of θ is radians, which can be converted to an angle using the above formula;

[0055] Step 6-4: Set the angle threshold β = 20. When the angle between the normal lines of the triangle primitives satisfies α < β, project the three reliable hypotheses of the triangle primitives into the reference coordinate system and form the matrix A to obtain the plane prior parameter z opt :

[0056]

[0057] Among them, z * is a unit-length variable;

[0058] Step 6-5: The final prior model is further filtered by the depth range of the current reference image. When the depth of the prior model is less than the minimum depth of the reference image or greater than the maximum depth of the reference image, the prior model of the pixel is deemed unreliable.

[0059] A further technical solution of the present invention is as follows: the multi-view matching cost evaluation assisted by plane prior in step 7 is specifically as follows:

[0060] The cost of multi-view matching assisted by plane prior is defined as:

[0061]

[0062] Among them, λ d is the bandwidth of the depth difference, λ n is the bandwidth of the normal difference, d iis the current assumed depth value, d p is the depth value of the prior model, n i is the currently assumed normal vector, n p is the normal vector of the prior model;

[0063] The hypothesis with the minimum plane prior-assisted multi-view matching cost is selected as the best estimate of the current center pixel.

[0064] A further technical solution of the present invention is as follows: The multi-view matching cost evaluation of the geometric consistency optimization in step 8 is specifically as follows:

[0065] The multi-view matching cost for geometric consistency optimization is defined as:

[0066]

[0067] Where Δe j (θ i ) is the reprojection error of the current hypothesis;

[0068] The hypothesis with the minimum geometric consistency optimized multi-view matching cost is selected as the best estimate of the current center pixel.

[0069] A further technical solution of the present invention: Step 10 specifically includes:

[0070] Step 10-1: The hypothesis of each pixel is projected into the adjacent image to produce the corresponding match;

[0071] Step 10-2: A consistent match is defined as simultaneously satisfying the following constraints: the angle between normals is less than 10°, the reprojection error is less than 2 pixels, and the relative depth error is less than a dynamic threshold τ, which is defined as:

[0072] ‖D s -D r ‖<τ

[0073] Where τ is a constant depth determined by the depth range of each image of the scene, for the input scene images {I0,…,I N-1} and its corresponding depth range τ is defined as:

[0074]

[0075] Step 10-3: If there are at least 2 source images whose corresponding matches satisfy all constraints in step 10-2, these hypotheses are projected into world coordinates and averaged into a 3D point.

[0076] The beneficial effects of the present invention are:

[0077] This paper proposes a multi-view dense 3D reconstruction method based on normal information. Starting with an input image sequence, it first uses a multi-view reconstruction method based on adaptive checkerboard sampling to obtain initial hypotheses. It then constructs a plane prior model and constrains it using its normal information, aiming to improve its validity and correctness. Geometric consistency is then used to optimize the results. Finally, to reduce noise interference, a dynamic filtering strategy is used to filter the hypotheses, which are then fused into a point cloud.

[0078] 1. The method of the present invention does not require any training process and has strong generalization ability.

[0079] 2. The method of the present invention can still accurately restore the three-dimensional geometry of objects or scenes in the face of textureless areas, including high-completeness restoration of textureless areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0081] Figure 1 Flowchart for the implementation of the present invention.

[0082] Figure 2 An image set is input to the embodiment of the present invention.

[0083] Figure 3 This is the reconstruction effect of the embodiment of the present invention.

[0084] Figure 4 This is a visualization effect diagram of the ETH3D dataset reconstruction in an embodiment of the present invention.

[0085] Figure 5 This is a visualization diagram of the space debris simulation dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0086] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0087] The present invention provides a multi-view dense reconstruction method based on normal information assistance, comprising the following steps:

[0088] S1: Select an image from the input image set as the reference image, divide the source image set into it according to the result of sparse reconstruction; and randomly initialize the reference image.

[0089] S2: Divide the pixels in the reference image into a red and black checkerboard grid, combine it with an adaptive checkerboard sampling strategy, and use the optimal hypothesis of each of the eight regions around the sampling center pixel for subsequent updates.

[0090] S3: The weights of the integrated candidate hypotheses in different views are evaluated using the multi-view matching cost, and the hypothesis with the minimum multi-view matching cost is used to update the center pixel.

[0091] S4: Generate perturbation hypotheses and random hypotheses to refine the solution space. The hypothesis with the lowest multi-view matching cost will be selected as the current hypothesis.

[0092] S5: Repeat steps S2 to S4 to make the current hypothesis converge.

[0093] S6: Construct a planar prior model based on the current convergence assumption.

[0094] S7: Based on steps S2 to S5, update the hypothesis using the multi-view matching cost evaluation based on the auxiliary plane prior.

[0095] S8: Based on steps S2 to S5, use the multi-view matching cost evaluation optimized by geometric consistency to further eliminate small errors.

[0096] S9: Select images in the input set as reference images in sequence to generate hypotheses.

[0097] S10: Filter the estimated depth hypotheses according to the dynamic filtering strategy, and then fuse all the hypotheses to generate the output point cloud.

[0098] Furthermore, the specific process of random initialization in S1 includes:

[0099] S11: Randomly generate a hypothesis θ containing depth information and normal information for each pixel in the reference image x =[d x ,n x ].

[0100] S12: Calculate the matching cost for the hypothesis randomly generated in S11 from each source image using the homography matrix.

[0101] S13: The initial multi-view matching cost is obtained by averaging the top 4 best costs.

[0102] Furthermore, the checkerboard sampling in S2 specifically includes:

[0103] S21: 8 sampling areas are divided into 4 long strip areas and 4 V-shaped areas;

[0104] Specifically, the refinement in S4 includes:

[0105] S42: Generate two additional hypotheses, one of which is obtained by perturbing the current hypothesis The other is randomly generated

[0106] S43: Two new hypotheses are randomly arranged and combined with the current hypothesis to form a set of seven hypotheses:

[0107]

[0108] S44: The final estimate of the center pixel will be the hypothesis with the minimum cost in the set of S43.

[0109] Furthermore, the construction of the plane prior model in S6 specifically includes:

[0110] S61: Collect reliable hypotheses with multi-view matching costs less than 0.1 and put them into a set.

[0111] S62: Use Delaunay triangulation to find three adjacent reliable hypotheses for pixels in non-S43 sets.

[0112] S63: Based on the assumption in S62, obtain the normals of the pixels at each vertex of the triangle primitive and calculate the angles between them:

[0113]

[0114] Among them, A(a x ,a y ,a z ) and B(b x ,b y ,b z ) is the known normal, and the unit of θ is radians, which can be converted to an angle using the above formula.

[0115] S64: Set the angle threshold β = 20. When the angle between the normal lines of the triangle primitives satisfies α < β, project the three reliable hypotheses into the reference coordinate system and form a matrix A to obtain the plane prior parameter z opt :

[0116]

[0117] Among them, z * is a unit-length variable.

[0118] S65: The final prior model is further filtered by the depth range of the current reference image.

[0119] Furthermore, the plane prior-assisted multi-view matching cost evaluation in S7 specifically includes:

[0120] S71: Define the cost of plane prior-assisted multi-view matching as,

[0121]

[0122] Among them, λ d is the bandwidth of the depth difference, λ n is the bandwidth of the normal difference. The hypothesis with the minimum plane prior-assisted multi-view matching cost will be selected as the best estimate of the current center pixel.

[0123] Furthermore, the multi-view matching cost evaluation for geometric consistency optimization in S9 specifically includes:

[0124] S91: Define the multi-view matching cost of geometric consistency optimization as,

[0125]

[0126] Where Δe j (θ i ) is the reprojection error of the current hypothesis. The hypothesis with the minimum geometric consistency optimized multi-view matching cost will be selected as the best estimate of the current center pixel.

[0127] Furthermore, point cloud fusion in S10 specifically includes:

[0128] S101: The hypothesis of each pixel is projected into the adjacent images to produce corresponding matches.

[0129] S102: A consistent match is defined as satisfying the following constraints: relative depth error is less than 0.01, the angle between normals is less than 10°, the reprojection error is less than 2 pixels, and the relative depth error is less than a dynamic threshold τ, which is defined as:

[0130] ‖D s -D r ‖<τ,

[0131] Where τ is a constant depth determined by the depth range of each image of the scene. For the input scene images {I0,…,I N-1} and its corresponding depth range τ is defined as:

[0132]

[0133] S103: If there are at least two source images whose corresponding matches satisfy the above conditions, these hypotheses are projected into world coordinates and averaged into a 3D point.

[0134] Example 1:

[0135] The embodiment of the technical solution of the present invention provides a multi-view dense 3D reconstruction method based on normal information assistance, the process of which is as follows: Figure 1 As shown, the following is a detailed explanation with examples:

[0136] (1) Random initialization: Figure 2 As shown in the figure, an image is selected from the input image set as a reference image, and a corresponding source image is selected for it. A random hypothesis (including depth information and normal information) is generated for each pixel in the reference image. A matching cost is calculated for each randomly generated hypothesis using the homography matrix from each source image. The initial cost is the average of the first four matching costs, sorted from smallest to largest.

[0137] (2) Adaptive checkerboard sampling: The 8 sampling areas of the central pixel are divided into 4 long strip areas and 4 V-shaped areas; each zigzag area starts with 8 samples, while each V-strip area starts with the initial 5 samples, and the sampling points with a radius of 4 around the central pixel are directly removed. First, the best hypothesis is selected in each area based on the multi-view matching cost of random initialization or the previous iteration (the initial cost in random initialization is used for the first propagation, while other iterations use the cost calculated in the previous iteration), and the candidate hypothesis set Θ = {θ i |i=1…8}. Next, the matching costs of the hypotheses in the candidate set relative to the different source images are calculated and embedded into the cost matrix M. Finally, a voting scheme is implemented for each row of matrix M. For a region, there should be at least one cost less than the good match threshold and at most two costs greater than the bad match threshold. When the above conditions are met, the expansion of the current sampling region is stopped. Otherwise, the region is expanded, and each expansion doubles the number of sampling points in the region. The above operation is repeated to determine whether to continue expansion.

[0138] (3) Multi-view matching cost evaluation: The candidate hypotheses are evaluated according to the calculation method of the multi-view matching cost, and the hypothesis with the minimum multi-view matching cost will be selected as the best estimate of the current center pixel.

[0139] (4) Refinement: Generate two new hypotheses and randomly permutate and combine them with the current hypothesis. The final estimate will be the hypothesis with the minimum cost.

[0140] (5) Constructing a plane prior model: using Delaunay triangulation to generate a plane prior model;

[0141] (6) Normal information guides the selection of plane prior models: obtain the normals of each vertex pixel of the plane prior model, calculate the angle between them, and select a credible plane prior model based on the threshold;

[0142] (7) Adaptive checkerboard sampling: The 8 sampling areas of the central pixel are divided into 4 long strip areas and 4 V-shaped areas; each zigzag area starts with 8 samples, while each V-strip area starts with the initial 5 samples, and the sampling points with a radius of 4 around the central pixel are directly removed. First, the best hypothesis is selected in each area based on the multi-view matching cost of random initialization or the previous iteration (the initial cost in random initialization is used for the first propagation, while other iterations use the cost calculated in the previous iteration), and the candidate hypothesis set Θ = {θ i |i=1…8}. Next, the matching costs of the hypotheses in the candidate set relative to the different source images are calculated and embedded into the cost matrix M. Finally, a voting scheme is implemented for each row of matrix M. For a region, there should be at least one cost less than the good match threshold and at most two costs greater than the bad match threshold. When the above conditions are met, the expansion of the current sampling region is stopped. Otherwise, the region is expanded, and each expansion doubles the number of sampling points in the region. The above operation is repeated to determine whether to continue expansion.

[0143] (8) Normal information guided plane prior assisted multi-view matching cost evaluation: The candidate hypotheses are evaluated according to the calculation method of the normal information guided plane prior assisted multi-view matching cost. The hypothesis with the minimum plane prior assisted multi-view matching cost will be selected as the best estimate of the current center pixel.

[0144] (9) Refinement: Generate two new hypotheses and randomly permutate and combine them with the current hypothesis. The final estimate will be the hypothesis with the minimum cost.

[0145] (10) Adaptive checkerboard sampling: The 8 sampling areas of the central pixel are divided into 4 long strip areas and 4 V-shaped areas; each line area starts with 8 samples, while each V-strip area starts with the initial 5 samples, and the sampling points with a radius of 4 around the central pixel are directly removed. First, the best hypothesis is selected in each area based on the multi-view matching cost of random initialization or the previous iteration (the initial cost in random initialization is used for the first propagation, while the other iterations use the cost calculated in the previous iteration), and the candidate hypothesis set Θ = {θ i|i=1…8}. Next, the matching costs of the hypotheses in the candidate set relative to the different source images are calculated and embedded into the cost matrix M. Finally, a voting scheme is implemented for each row of matrix M. For a region, there should be at least one cost less than the good match threshold and at most two costs greater than the bad match threshold. When the above conditions are met, the expansion of the current sampling region is stopped. Otherwise, the region is expanded, and each expansion doubles the number of sampling points in the region. The above operation is repeated to determine whether to continue expansion.

[0146] (11) Geometric consistency optimized multi-view matching cost evaluation: The candidate hypotheses are evaluated according to the calculation method of the geometric consistency optimized multi-view matching cost. The hypothesis with the minimum geometric consistency optimized multi-view matching cost will be selected as the best estimate of the current center pixel.

[0147] (12) Refinement: Generate two new hypotheses and randomly permutate and combine them with the current hypothesis. The final estimate will be the hypothesis with the minimum cost.

[0148] (13) Hypothesis fusion: Hypotheses for each pixel are projected into adjacent images to produce corresponding matches. If there are at least two source images whose corresponding matches satisfy the corresponding constraints of the dynamic filtering, these hypotheses are projected into world coordinates and averaged into a 3D point.

[0149] The method of the present invention is applied to actual three-dimensional dense reconstruction, and the effect is as follows Figure 3 As shown in the figure. From the performance of this method on the three-dimensional data set ( Figure 4 , Table 1) It can be seen that the method of the present invention can completely restore the dense 3D model of the object and is superior to the existing 3D dense reconstruction method. In addition, the performance of this method on the simulated space debris dataset is also particularly outstanding ( Figure 5 ), effectively recovering the weakly textured areas of space debris (sails, cube faces of the main body, etc.).

[0150] Table 1 shows the evaluation results of the reconstructed point cloud on the ETH3D dataset. The table shows the accuracy, completeness, and F1 score at different evaluation thresholds (2 cm and 10 cm), with the best results highlighted in bold.

[0151] Table 1 shows the evaluation results of point cloud reconstruction on the ETH3D dataset.

[0152]

[0153] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present invention, and these modifications or replacements should all be included in the scope of protection of the present invention.

Claims

1. A multi-view dense 3D reconstruction method based on normal information, characterized in that: Here are the steps: Step 1: Randomly select an image in the input image sequence as the reference image, and the remaining images as source images; and randomly initialize the reference image; Step 2: Divide the pixels in the reference image into red and black checkerboards. Combined with the adaptive checkerboard sampling strategy, set each pixel on the reference image as the center pixel in turn, and use the pixels in each of the eight regions around the pixel as candidate hypothesis solutions. Then, by calculating the matching cost of the candidate hypotheses, select the best hypothesis in each region for subsequent updates. Step 3: Use the multi-view matching cost to estimate the weights of the best hypothesis selected in step 2 in different views. The hypothesis with the minimum multi-view matching cost will be used to update the center pixel. Step 4: Generate random hypotheses and perturbation hypotheses to refine the solution space, and select the hypothesis with the minimum multi-view matching cost as the current hypothesis; Step 5: Repeat steps 2 to 4 to make the current hypothesis converge; Step 6: construct a plane prior model based on the current assumption, obtain the vertex normals of the plane prior model, calculate the mutual angles of the normals in the plane prior model, and select the plane prior model according to the threshold; Step 7: Update the hypothesis using multi-view matching cost estimation assisted by normal-guided plane priors. Step 8: Use geometric consistency-guided multi-view matching cost evaluation to further eliminate errors; Step 9: Select images in the image sequence as reference images in turn to generate hypotheses; Step 10: Filter all hypotheses using a dynamic filtering strategy; Step 11: Fuse all the filtered hypotheses to generate the output point cloud.

2. According to claim 1, a multi-view dense 3D reconstruction method based on normal information assistance is characterized in that: The random initialization process in step 1 includes: Step 1-1: Randomly generate a pixel containing depth information d for each pixel in the reference image. x With normal information n x The assumption θ x =[d x ,n x ], this assumption can establish a three-dimensional plane for each pixel in the reference image, where d x is the distance from the three-dimensional plane to the origin, n x is the normal vector; Step 1-2: From each source image, pass the homography matrix to the hypothesis θ randomly generated in step 1-1 x The matching cost is calculated. Specifically, given the camera parameters of the reference image and the camera parameters of the j-th source image, the homography transformation between them is calculated as: Step 1-3: Average the top 4 smallest costs to get the initial multi-view matching cost.

3. According to claim 1, a multi-view dense 3D reconstruction method based on normal information assistance is characterized in that: The step 2 is specifically as follows: Step 2-1: 8 sampling areas are selected around the central pixel, including 4 long strip areas and 4 V-shaped areas. The long strip areas contain 11 sampling points, and the V-shaped areas contain 7 sampling points. Step 2-2: Extended strategy; Step 2-2-1: Select the best hypothesis in each region based on the multi-view matching cost of random initialization or the previous iteration, and define the candidate hypothesis set Θ = {θ i |i=1…8}; the initial cost in random initialization is used for the first iteration, and subsequent iterations use the cost calculated in the previous iteration; Step 2-2-2: Further use the eight structured hypotheses obtained by adaptive checkerboard sampling to infer the weight of each neighboring view and embed it into the cost matrix M: Where N represents the number of input images, m i,j represents the matching cost of the i-th region sampling point calculated by the j-th view; Step 2-2-3: Based on the matrix M, a voting scheme is implemented in each column to determine whether a view is suitable. In addition, the matching cost of good views will decrease as the method iterates; the cost boundary is defined as: Among them, τ0 represents the cost threshold of the initial matching, t represents the number of iterations, and α is a constant; When the number of matching costs that meet the conditions for each view exceeds a certain threshold, the view is included in the current view selection set.

4. According to claim 1, a multi-view dense 3D reconstruction method based on normal information assistance is characterized in that: In order to evaluate the multi-view matching cost in step 3, the credibility of the matching cost is introduced, which is defined as: Among them, β is a constant, and the weight of each view can be defined as: Where St is the view selection set, II(true)=1, II(false)=0; Finally, the multi-view aggregated photometric consistency cost of a pixel is defined as: The hypothesis with the minimum multi-view matching cost will be selected as the best estimate of the current sample center pixel.

5. The multi-view dense 3D reconstruction method based on normal information assistance according to claim 1, characterized in that: The step 4 specifically includes: Step 4-1: Generate two additional hypotheses, one of which is obtained by perturbing the current hypothesis Another hypothesis is that it is randomly generated. Step 4-2: Two new hypotheses are randomly ordered and combined with the current hypothesis to form a set of seven hypotheses: Step 4-3: The final estimate of the sampled center pixel is the hypothesis with the minimum cost among the set in step 4-2.

6. The multi-view dense 3D reconstruction method based on normal information assistance according to claim 1, characterized in that: The construction of the plane prior model in step 6 specifically includes: Step 6-1: Collect reliable hypotheses with multi-view matching costs less than 0.1 and put them into a set to form sparse reliable corresponding points; Step 6-2: Use Delaunay triangulation to find three neighboring reliable hypotheses for the pixel points in the reference image that are not in the set of step 6-1 to adaptively generate triangular primitives of different sizes; Step 6-3: Based on the assumptions in step 5, obtain the normals of the pixels at each vertex of the triangle primitive and calculate the angles between them: Among them, A(a x ,a y ,a z ) and B(b x ,b y ,b z ) is the known normal, and the unit of θ is radians, which can be converted into an angle using the above formula; Step 6-4: Set the angle threshold β = 20. When the angle between the normal lines of the triangle primitives satisfies α < β, project the three reliable assumptions of the triangle primitives into the reference coordinate system and form a matrix A to obtain the plane prior parameter z opt : Among them, z * is a unit length variable; Step 6-5: The final prior model is further filtered by the depth range of the current reference image. When the depth of the prior model is less than the minimum depth of the reference image or greater than the maximum depth of the reference image, the prior model of the pixel is deemed unreliable.

7. The multi-view dense 3D reconstruction method based on normal information assistance according to claim 1, characterized in that: The plane prior-assisted multi-view matching cost evaluation in step 7 is specifically: The cost of multi-view matching assisted by plane prior is defined as: Among them, λ d is the bandwidth of the depth difference, λ n is the bandwidth of the normal difference, d i is the currently assumed depth value, d p is the depth value of the prior model, n i is the currently assumed normal vector, n p is the normal vector of the prior model; The hypothesis with the minimum plane prior-assisted multi-view matching cost is selected as the best estimate of the current center pixel.

8. The multi-view dense 3D reconstruction method based on normal information assistance according to claim 1, characterized in that: The multi-view matching cost evaluation of geometric consistency optimization in step 8 is specifically as follows: The multi-view matching cost for geometric consistency optimization is defined as: Among them, Δe j (θ i ) is the reprojection error of the current hypothesis; The hypothesis with the minimum geometrically consistent optimized multi-view matching cost is selected as the best estimate of the current center pixel.

9. The multi-view dense 3D reconstruction method based on normal information assistance according to claim 1, characterized in that: The step 10 specifically includes: Step 10-1: The hypothesis of each pixel is projected into the adjacent images to produce the corresponding match; Step 10-2: A consistent match is defined as satisfying the following constraints simultaneously: the angle between normals is less than 10°, the reprojection error is less than 2 pixels, and the relative depth error is less than a dynamic threshold τ, which is defined as: ‖D s -D r ‖<τ where τ is a constant depth determined by the depth range of each image of the scene, and for the input scene images {I0,…,I N-1 } and its corresponding depth range τ is defined as: Step 10-3: If there are at least 2 source images whose corresponding matches satisfy all constraints in step 10-2, these hypotheses are projected into world coordinates and averaged into a 3D point.

Images