Multi-scale three-dimensional reconstruction system and method based on dynamic optimization

Abstract

The invention discloses a multi-scale three-dimensional reconstruction system and method based on dynamic optimization. The method is characterized by comprising the following steps: A, inputting and preprocessing; b, sampling the expandable self-adaptive chessboard; c, depth estimation of layered optimization; d, performing dynamic weight multi-view point cloud fusion; e, performing global optimization; and F, outputting. Compared with the prior art, the method has the advantages that after extensible self-adaptive chessboard sampling, a layered optimization depth estimation method and a dynamic weight multi-view point cloud fusion strategy are fused, the method has the following advantages that 1, the innovativeness of extensible self-adaptive chessboard sampling is achieved; 2, the accuracy of the depth estimation method based on hierarchical optimization is improved; and 3, adaptability of the dynamic weight multi-view point cloud fusion strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and specifically refers to a multi-scale three-dimensional reconstruction system and method based on dynamic optimization. Background Art

[0002] Three-dimensional reconstruction technology is widely used in fields such as virtual reality, cultural relic protection, and industrial measurement. Its core goal is to extract depth information from multi-view images and generate high-precision three-dimensional point clouds. However, existing technologies have obvious limitations in complex scenes, especially in low-texture, occluded, and repetitive texture areas, making it difficult to ensure the accuracy and robustness of depth estimation.

[0003] Traditional three-dimensional reconstruction methods usually perform depth estimation based on multi-view matching and photometric consistency constraints. The typical processes of these methods include the following steps: First, obtain an initial depth hypothesis through multi-view matching; second, optimize the depth estimation using a local propagation strategy (such as checkerboard sampling); finally, generate a global point cloud through depth fusion with fixed weights. However, these technologies have the following disadvantages in practical applications:

[0004] 1. Limitations of local propagation

[0005] Traditional checkerboard sampling methods adopt a fixed propagation mode, usually limited to the four directions of up, down, left, and right. For low-texture or complex geometric shape areas, the propagation path may be restricted, resulting in propagation failure or hole generation, and unable to fully cover the geometric features of complex scenes. In addition, the fixed sampling strategy is difficult to adapt to dynamic scenes, and the sampling efficiency and accuracy are limited.

[0006] 2. Limitations of existing depth estimation methods

[0007] Existing depth estimation methods show obvious limitations in low-texture, occluded, and repetitive texture areas. In low-texture areas, due to the lack of significant pixel features, photometric consistency matching is prone to failure, resulting in unstable depth estimation results or holes; the complexity of occluded areas makes the multi-view matching cost have significant differences locally, and traditional methods lack robust geometric constraints, leading to depth jumps or error propagation; while in repetitive texture areas, photometric consistency is prone to matching ambiguity, resulting in mis-matching problems. These defects not only affect the local depth estimation accuracy, but may also further reduce the global quality of three-dimensional reconstruction.

[0008] 3. Limitations of fixed-weight fusion

[0009] Traditional point cloud fusion methods mostly use fixed weights to perform weighted averaging on multi-view depth values, ignoring the differences in the reliability of depth estimation from different perspectives. This fixed weight mechanism is difficult to dynamically adapt to the characteristics of different scenes, resulting in the negative impact of low-quality estimations on the fusion results, especially being particularly significant in occluded areas or complex texture scenes.

[0010] Existing technologies have significant limitations in sampling, depth estimation, and point cloud fusion: The fixed propagation mode is difficult to cover the geometric features of complex scenes, photometric consistency matching is prone to failure in low-texture and repetitive texture areas, and the traditional fixed-weight fusion method is difficult to dynamically adapt to scene characteristics, resulting in insufficient accuracy and robustness of depth estimation and point cloud reconstruction results and being unable to meet the high-precision reconstruction requirements of complex scenes. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a dynamic optimization-based multi-scale three-dimensional reconstruction system and method that can overcome the above-mentioned drawbacks and improve the three-dimensional reconstruction performance in complex scenes by means of a flexible sampling strategy, dynamic weight adjustment, and the introduction of global geometric constraints, in response to the steps proposed in the above background technology.

[0012] To solve the above technical problem, the technical solution provided by the present invention is: A dynamic optimization-based multi-scale three-dimensional reconstruction system and method, which includes the following steps:

[0013] A. Input and preprocessing;

[0014] B. Expandable adaptive checkerboard sampling;

[0015] C. Hierarchical optimization of depth estimation;

[0016] D. Dynamic weight multi-view point cloud fusion;

[0017] E. Global optimization;

[0018] F. Output.

[0019] As an improvement, the input and preprocessing include the following steps:

[0020] 1). Input a multi-view image sequence and corresponding camera parameters;

[0021] 2). Perform pyramid decomposition on the input images to generate a multi-scale image sequence.

[0022] As an improvement, the expandable adaptive checkerboard sampling includes the following steps:

[0023] 1). Initial sampling: Divide the pixels of the reference image into a red-black checkerboard pattern and perform initial depth propagation according to the eight-direction propagation mode;

[0024] 2) Dynamic expansion: Two expansion strategies (V-shaped end broken line propagation and rectangular area extended sampling) are adopted for the propagation failure area, and the expansion range and the number of samples are controlled according to the cost matrix and the dynamic threshold, and finally a preliminary depth estimation with a wider coverage range is generated.

[0025] As an improvement, the hierarchical optimized depth estimation includes the following steps:

[0026] 1) Trusted point screening: Based on the photometric consistency cost calculation, screen the trusted point set and construct the trusted point set I cred ={x|c photo (θ x )<τ cred}};

[0027] 2) Plane prior generation:

[0028] a) Apply Delaunay triangulation to divide the trusted points into triangular regions and generate local geometric planes;

[0029] b) Use K-nearest neighbor search for the edge regions and calculate and construct new triangular regions in combination with Heron's formula;

[0030] c) Optimize the objective function based on the fitting plane parameters

[0031] 3) Hierarchical optimization: Optimize the depth estimation layer by layer:

[0032] a) Optimize the initial depth by combining the plane prior and the geometric consistency cost at the coarse resolution scale;

[0033] b) After upsampling to the high-resolution layer, supplement the depth information of the low-level layer through the detail restorer and correct the estimation error layer by layer.

[0034] As an improvement, the dynamic weight multi-view point cloud fusion includes the following steps:

[0035] 1) Weight construction, calculate the dynamic weight based on photometric consistency, geometric consistency and global prior:

[0036] w total (p)=w photo (p)·w geo (p)·w prior (p)

[0037] 2) Weighted fusion, perform weighted averaging on the multi-view depth values through the dynamic weight to calculate the fused depth value:

[0038]

[0039] 3) Confidence filtering: Calculate the confidence of the fusion points through dynamic weights, and eliminate low-quality estimated points to optimize the global point cloud quality.

[0040] As an improvement, the global optimization includes the following steps:

[0041] Geometric consistency optimization: Minimize the geometric consistency loss by constraining the multi-view depth consistency:

[0042]

[0043] Global prior optimization: Combine the global geometric prior to minimize the global prior loss:

[0044]

[0045] Joint optimization: Comprehensive loss function L total = L geo + λ prior ·L prior , and optimize the depth map through the gradient descent method until convergence.

[0046] As an improvement, the output includes the following steps:

[0047] 1) Output the optimized global point cloud depth map;

[0048] 2) Generate a high-precision 3D reconstruction model based on the optimized point cloud.

[0049] After adopting the above method, the present invention has the following advantages:

[0050] 1. The innovation of expandable adaptive checkerboard sampling

[0051] In this application, by introducing an eight-direction propagation mode (including four rectangular regions and four V-shaped regions) and a dynamic sampling expansion mechanism, the flexibility and adaptability of depth propagation are significantly enhanced. The V-shaped end broken line propagation and the extended sampling strategy of the rectangular region can effectively cover the low-texture, occluded, and repetitive texture regions in complex scenes, reducing the holes and error accumulation caused by traditional sampling methods. The dynamic threshold mechanism further improves the sampling efficiency and optimizes the allocation of computing resources;

[0052] 2. The accuracy of the hierarchical optimization depth estimation method

[0053] Through hierarchical optimization, for each layer, photometric consistency cost calculation is first performed. Combining the reliable point set with the planar prior information constructed by Delaunay triangulation, high-robustness depth hypotheses are generated. With the assistance of the planar prior, geometric consistency cost optimization adjusts the depth estimation layer by layer to reduce the depth deviation between multiple views. The detail restorer updates the mechanism by combining upsampling and downsampling with photometric consistency differences to make up for the information loss between scales, ensuring the global consistency and stability of depth estimation, and is applicable to high-precision scene reconstruction;

[0054] 3. Adaptability of the dynamic weight multi-view point cloud fusion strategy

[0055] In this application, dynamic weights are constructed through photometric consistency, geometric consistency, and global priors, solving the problem that traditional fixed-weight methods cannot handle complex scenes. The multi-view point cloud fusion based on dynamic weights has significantly improved performance in low-texture and occluded areas. At the same time, low-quality estimated points are removed through confidence filtering, enhancing the coherence of the global geometric structure. Global optimization realizes high-precision optimization and global consistency of the point cloud through geometric consistency and global prior constraints, and is applicable to 3D reconstruction tasks under complex scenes. Description of the Drawings

[0056] Figure 1 It is a schematic structural diagram of a multi-scale 3D reconstruction system and method based on dynamic optimization. Detailed Embodiment

[0057] The present invention will be further described in detail below with reference to the drawings.

[0058] Combined with the attached Figure 1 , a multi-scale 3D reconstruction system and method based on dynamic optimization, characterized in that it includes the following steps:

[0059] A. Input and preprocessing:

[0060] 1). Input a multi-view image sequence and corresponding camera parameters;

[0061] 2). Perform pyramid decomposition on the input images to generate a multi-scale image sequence.

[0062] B. Expandable adaptive checkerboard sampling:

[0063] Hypothesis propagation is a prerequisite for multi-view matching cost evaluation and refinement. Therefore, choosing a reasonable sampling pattern will lead to accurate results. Traditional adaptive checkerboard propagation divides the image pixel area by checkerboards and only relies on local propagation for depth estimation. In areas with local low texture, occlusion, or complex geometries, traditional methods may not find suitable depth estimation points due to limited propagation paths, resulting in inaccurate estimation results or even holes.

[0064] To overcome these problems, an extensible adaptive chessboard propagation method is proposed. By improving the propagation strategy, increasing the propagation directions, and dynamically adjusting the sampling strategy, the depth estimation accuracy and robustness in complex scenes are improved. The specific innovations include a new propagation mode and the introduction of a dynamic sampling extension mechanism.

[0065] 1), Initial sampling: Divide the pixels of the reference image into a red-black chessboard pattern, and perform initial depth propagation according to the eight-direction propagation mode;

[0066] 2), Dynamic extension: For the areas where propagation fails, two extension strategies (V-shaped end broken line propagation and rectangular area extended sampling) are adopted. According to the cost matrix and the dynamic threshold, control the extension range and the number of samples, and finally generate a preliminary depth estimation with a wider coverage range.

[0067] (1) New propagation mode

[0068] A diffusion-like propagation scheme can greatly improve efficiency while maintaining high-quality hypothesis propagation. We first divide all pixels in the reference image into a red-black chessboard pattern, which uses the hypotheses in eight regions as candidate updates. In the initial stage of propagation, we divide the eight regions into four rectangular regions and four V-shaped regions. The directions of the four rectangular regions are horizontal (left and right) and vertical (up and down) propagation, which are responsible for handling the depth propagation of local planar regions. The four V-shaped regions are symmetrically distributed in the up, down, left, and right directions. The rectangular regions initially have 11 samples, and the V-shaped regions have 7 samples. For the regions where initial propagation fails, two dynamic extension strategies are adopted: V-shaped end broken line propagation extension and rectangular area extended sampling.

[0069] (2) Dynamic sampling extension mechanism

[0070] Select the best hypothesis from the random initialization or the propagation result of the previous iteration to form a candidate hypothesis set. For the candidate hypotheses in each region, calculate the matching cost with different source images to generate a cost matrix. According to the cost matrix M, apply a voting mechanism to each candidate direction to determine whether further extension is needed.

[0071] Define a dynamic threshold τ good , which is used to evaluate whether the cost of the candidate point is good enough:

[0072]

[0073] t iter : The current iteration number

[0074] t ext : The current extension number

[0075] N ext : The maximum extension number.

[0076] α: attenuation constant

[0077] Define another threshold τ bad , which is used to limit the bad matching results. There are two extension conditions. The first is that there are at least n good matching costs less than τ good in the region. The second is that at most n bad matching costs greater than τ bad are allowed. Expand the regions that do not meet the stopping conditions to increase the number of sampling points. After each expansion, update the candidate hypothesis set and the cost matrix M, recalculate the matching costs, and determine whether the expansion conditions are met. The number of expansions each time is dynamically adjusted according to the number of expansion times and the regional complexity to optimize the computing resources. When all regions meet the stopping conditions, or the maximum number of expansion times N ext is reached, the expansion strategy ends. Output the final expansion results, including the updated candidate hypotheses and the regional coverage.

[0078] (3) Technical advantages

[0079] Compared with the traditional checkerboard sampling method, the expandable adaptive checkerboard sampling has significant advantages in many aspects:

[0080] Flexibility of multi-directional propagation: The traditional checkerboard sampling is limited to the propagation in four directions of up, down, left, and right, with a limited coverage range and difficulty in handling complex scenarios. While this method effectively expands the propagation range and enhances the ability to capture complex geometric features such as diagonal edges and corners by introducing an eight-direction propagation mode (including four V-shaped regions and four rectangular regions).

[0081] Dynamic expansion mechanism: In the case of insufficient initial propagation, this method introduces the broken-line propagation at the V-shaped ends and the extended sampling strategy in the rectangular regions, flexibly adjusts the sampling range and the number of points according to the local scene characteristics, and significantly improves the depth estimation accuracy in low-texture regions and occluded regions.

[0082] Dynamic threshold control: By setting the dynamic thresholds τ good and τ bad , gradually optimize the propagation results according to the number of iterations and the number of expansions, improve the utilization efficiency of computing resources, and avoid over-expansion or unnecessary calculations.

[0083] C. Hierarchical optimization of depth estimation:

[0084] 1). Credible point screening: Based on the photometric consistency cost calculation, screen the set of credible points to construct the credible point set I cred ={x∣c photo (θ x )<τ cred};

[0085] 2) Plane prior generation:

[0086] a) Apply Delaunay triangulation to divide the reliable points into triangular regions and generate local geometric planes;

[0087] b) Use K-nearest neighbor search for the edge region and calculate and construct new triangular regions in combination with Heron's formula;

[0088] c) Optimize the objective function based on the fitting plane parameters

[0089] 3) Hierarchical optimization: Optimize depth estimation layer by layer:

[0090] a) Optimize the initial depth by combining the plane prior and the geometric consistency cost at the coarse resolution scale;

[0091] b) After upsampling to the high-resolution layer, supplement the depth information of the low-level layer through the detail restorer and correct the estimation error layer by layer.

[0092] The present invention proposes a depth estimation method based on hierarchical optimization. At each layer, first calculate the photometric consistency cost to generate an initial depth hypothesis; construct a local geometric plane through the reliable point set and triangulation to generate plane prior information. With the assistance of the plane prior, combine the geometric consistency cost calculation to optimize the depth estimation and reduce the depth deviation between multiple views. Different scales are connected through the detail restorer. After the depth estimation result of the low resolution is optimized through upsampling and the detail restorer, it provides a high-quality initial value for the high-resolution scale, thereby refining and optimizing the depth result layer by layer. This method has the following advantages: By combining hierarchical optimization and plane prior, it effectively improves the robustness of depth estimation in low-texture, occluded, and repetitive texture regions; reduces information loss between scales through the detail restorer to ensure global consistency and depth accuracy; significantly improves the 3D reconstruction quality of complex scenes in a layer-by-layer optimization manner and is applicable to high-precision, multi-scene 3D reconstruction tasks.

[0093] (1) Introduce geometric consistency

[0094] Geometric consistency is a constraint condition that measures the geometric matching degree of reconstructed points through multi-view depth projection. Specifically, it judges whether these depth estimations are geometrically consistent by comparing the depth values of pixel points in a reference view with the depth values at their projected positions in other target views. Geometric consistency can reflect the depth consistency of the same scene point under multiple views and is especially suitable for depth optimization in occluded regions or scenes with photometric inconsistency.

[0095] Geometric consistency is defined by the forward and backward reprojection error. Assume the depth D of point p r(p), the corresponding 3D point of point p in the reference image is:

[0096]

[0097] where M r is the camera matrix.

[0098] Project X r (p) onto the target view image and calculate the projection error:

[0099] Δ ∈ i = min(‖P j × X r (p) - p j ‖, δ)

[0100] Pj: The camera projection matrix of the target view.

[0101] pj: The projection point of point p in the target view.

[0102] δ: The truncation threshold, used to improve the robustness to occluded regions.

[0103] In the multi-view case, the geometric consistency cost can be calculated by accumulating the multi-view reprojection error:

[0104]

[0105] wj: The view weight.

[0106] λ: The weight balancing the geometric consistency term and the photometric consistency term.

[0107] In the pyramid multi-scale process, the geometric consistency shows significant superiority through layer-by-layer optimization of depth estimation. At the coarsest scale, the geometric consistency combines with the photometric consistency to optimize the initial depth map, especially reducing errors in low-texture regions; subsequently, when upsampling to a finer scale, the geometric consistency further adjusts the depth estimation, correcting errors through the geometric projection relationship of neighboring views to ensure the accuracy of layer-by-layer propagation. Compared with the photometric consistency, the geometric consistency is independent of pixel photometric information and is more robust to scenes with illumination changes, low-texture regions, and large depth changes. Its robustness is also reflected in the optimization of occluded regions in complex scenes. Using the multi-view reprojection constraint significantly improves the global consistency of depth estimation while reducing depth jumps and error propagation caused by view changes. In addition, the geometric consistency effectively compensates for the deficiencies of the photometric consistency during each layer of optimization, providing stable and reliable support for high-precision 3D reconstruction.

[0108] (2) Introduce a detail restorer

[0109] In the pyramid multi-scale process, the coarser resolution layers mainly optimize the depth through global consistency constraints. However, the low-resolution layers may lead to blurred edges and loss of texture details. Especially in the sampling mode of a fixed window, it is difficult to adapt to local complex structures. The detail restorer is introduced to supplement the lost details of the low-level estimation for the high-resolution layer through the joint optimization of photometric consistency, geometric consistency, and context, while eliminating the accumulation of estimation errors. The specific implementation process is as follows.

[0110] Upsample the depth estimation result of the previous scale to the current scale l to obtain the initial hypothesized depth map And use it to recalculate the photometric consistency cost at the current scale At the current scale, use the basic multi-view stereo (MVS) model to calculate a new hypothesized depth map And the corresponding photometric consistency cost Calculate the difference in photometric consistency cost through the following formula:

[0111]

[0112] If Then it is determined that there is an error in the initial estimate of the current pixel point p, where ξ is a constant for tolerating errors, used to improve the robustness of the model. According to the photometric consistency difference, use the following formula to update the corrected hypothesized depth map

[0113]

[0114] Ⅱ is the indicator function, when , Otherwise it is 0. For misestimated points, replace the initial depth with the re-optimized depth value.

[0115] Take the corrected hypothesized depth map Image and camera parameters as inputs, and further perform geometric consistency optimization in combination with plane prior assistance

[0116] (3) Plane prior construction

[0117] 1. Screen the reliable point set

[0118] Screen reliable points through the photometric consistency index c photo (θ x ), set the reliable threshold τ cred , and construct the reliable point set:

[0119] I cred ={x|c photo (θ x )<τ cred}

[0120] 2. Delaunay Triangulation

[0121] Apply Delaunay triangulation to the trusted point set I cred to divide the points into a series of triangular regions, generating an initial partition of the local geometric structure.

[0122] 3. Edge Region Processing

[0123] For the edge regions not covered by triangulation, use K-search to find the K nearest non-local points of the edge region points. Calculate the area of the triangle formed by three points through Heron's formula to avoid collinearity problems. The selected three points are projected onto the reference camera coordinate system to generate matrix A

[0124] 4. Plane Parameter Estimation

[0125] Based on the constructed point set, minimize the objective function to fit the plane parameters:

[0126]

[0127] Obtain the parameters z of each local plane opt , including the normal vector and the offset.

[0128] 5. Global Prior Filtering and Prior Plane Model Generation

[0129] Filter the prior plane information of each image by combining the depth map range to ensure that the retained planes conform to the global geometric constraints and avoid the interference of incorrect priors. Subsequently, fuse the filtered local plane fitting results with the global geometric prior to generate a complete plane prior model. This model is used for multi-view cost calculation in low-texture regions, effectively improving the stability and accuracy of depth estimation.

[0130] D. Dynamic Weight Multi-View Point Cloud Fusion:

[0131] Multi-View Stereo (MVS) technology is widely used in the generation and fusion of 3D point clouds. Its core lies in how to effectively fuse the depth estimates generated from different viewpoints to form a high-quality global point cloud. However, existing point cloud fusion methods mainly rely on weighted averaging with fixed weights or simple confidence filtering, and these methods have significant limitations in complex scenes. For example, fixed weights cannot dynamically adjust the reliability of depth estimates from different viewpoints, easily leading to local error propagation; in low-texture, occluded, or repetitive texture regions, solely relying on photometric consistency is difficult to ensure the accuracy of depth estimates; without global prior constraints, the point cloud fusion results may be incoherent in the global structure, affecting the reconstruction accuracy and robustness.

[0132] To this end, the present invention proposes a multi-viewpoint cloud fusion method based on dynamic weights, which dynamically adjusts the weights of each view by comprehensively considering photometric consistency, geometric consistency, and global prior information to achieve high-precision point cloud fusion in complex scenes. The method includes five steps: construction and extraction of global priors, construction of dynamic weights, weighted depth fusion, confidence filtering, and global optimization. It can significantly improve the depth estimation accuracy in complex regions, enhance the consistency of the global structure, and at the same time reduce the negative impact of low-quality estimations on the overall point cloud. Compared with the prior art, the present invention has the advantages of strong dynamics, high adaptability, and good global coherence, and is applicable to high-precision three-dimensional reconstruction in complex scenes.

[0133] The following are the specific implementation steps.

[0134] (1) Construction and extraction of global priors

[0135] Based on the sparse point cloud, the point cloud is divided into a series of triangular regions by applying Delaunay triangulation. The vertices of each triangle are fitted to generate local geometric planes, and the plane is parameterized by its normal vector and offset. Combining the adjacency relationship of the triangulation, multiple local planes are fused into an overall geometric model. By filtering and merging adjacent planes, the continuity and consistency of the model are ensured. The finally output global geometric prior consists of multiple local planes and is represented in the form of a set of parameterized planes within the projection range of the point cloud, providing global constraints and guidance for depth estimation, and significantly improving the three-dimensional reconstruction accuracy and stability in complex scenes.

[0136] (2) Construction of dynamic weights

[0137] During the multi-view depth fusion process, dynamic weights are assigned to the depth estimations of each view, aiming to solve the problem of the decline in reconstruction quality caused by error propagation in traditional fixed-weight methods. Since there are differences in illumination, occlusion, and scene characteristics among different views, the reliability of their depth estimations is different. The present invention dynamically adjusts the contribution degree of depth estimation to the fusion result through a dynamic weight assignment mechanism, combining photometric consistency, geometric consistency, and global prior information, reducing the negative impact of low-quality estimations, especially in low-texture and occluded areas, and significantly improving the robustness and global consistency of the fusion result. At the same time, the dynamic weight mechanism can optimize the utilization of computing resources, give priority to using high-quality depth estimations, and improve the overall point cloud reconstruction accuracy.

[0138] During the multi-view depth fusion process, dynamic weights are assigned to the depth estimations of each view. The dynamic weights consist of the following three parts:

[0139] Photometric consistency weight:

[0140] Calculated by measuring the reliability of photometric matching between the reference view and the target view:

[0141]

[0142] C photo (p): photometric consistency cost, reflecting the color consistency of pixel p.

[0143] σ photo : adjustment parameter, used to control the sensitivity of the weight distribution.

[0144] Geometric consistency weight:

[0145] Calculated by measuring the geometric consistency between multiple views:

[0146]

[0147] C geo (p) = ∥D r (p) - D t (p')∥: reprojection error between multiple views.

[0148] σ geo : adjustment parameter, used to control the influence of geometric consistency weight.

[0149] Global prior weight:

[0150] Calculated by the matching degree between the prior depth and the current depth estimate:

[0151]

[0152] D prior (p): depth value provided by the global prior.

[0153] D pred (p): current depth estimate value.

[0154] κ: balance factor, used to adjust the influence of the global prior in the weight.

[0155] Dynamic weight synthesis:

[0156] Integrate photometric consistency, geometric consistency and global prior to generate the final weight:

[0157] w total (p) = w photo (p) · w geo (p) · w prior (p)

[0158] (3) Weighted depth fusion

[0159] In the multi-view depth fusion stage, weighted averaging of multi-view depth values based on dynamic weights can effectively solve the error propagation problem caused by equal-weight processing in traditional methods. Through a dynamic weight allocation mechanism, the present invention dynamically adjusts the weights of depth values of each view according to photometric consistency, geometric consistency, and global prior information, ensuring that views with high reliability have a greater weight in the fusion, while the impact of unreliable estimates on the final result is significantly weakened. Through the weighted averaging method, not only the local accuracy of the point cloud is improved, but also the consistency of the global geometric structure is enhanced, especially in areas with low texture, occlusion, and repetitive texture. At the same time, this method optimizes the allocation of computing resources, improves the fusion efficiency, and is applicable to high-precision 3D reconstruction of complex scenes.

[0160] In the fusion stage, weighted averaging of multi-view depth values is performed based on dynamic weights:

[0161]

[0162] D i (p): The depth value of pixel p in view i.

[0163] w total,i (p): The dynamic weight of view i.

[0164] Depth maps of all views are fused through weighting to generate a preliminary global point cloud depth map.

[0165] (4) Confidence filtering

[0166] Calculate the confidence of the fused points according to the dynamic weights, aiming to comprehensively evaluate the reliability of multi-view depth estimation, so as to effectively filter out low-quality estimates and improve the quality of point cloud fusion. The present invention combines photometric consistency, geometric consistency, and global prior information, assigns weights to each view through a dynamic weight mechanism, and calculates the overall confidence of the fused points based on this. Points with high confidence indicate strong reliability of depth estimation, while points with low confidence may have large errors or be affected by interference. By removing these low-confidence points, the negative impact of error propagation on the fusion result is avoided. This method performs particularly well in areas with low texture, occlusion, and repetitive texture, significantly improving the local accuracy and global geometric consistency of the point cloud. At the same time, it optimizes the allocation of computing resources to meet the requirements of high-quality 3D reconstruction in complex scenes.

[0167] Calculate the confidence of the fused points according to the dynamic weights:

[0168]

[0169] If C conf (p) < T conf , remove point p to reduce the impact of low-quality points on the global point cloud. Output the point cloud after confidence filtering, removing low-quality estimated points.

[0170] E. Global optimization:

[0171] Geometric consistency optimization: The core of geometric consistency optimization is to ensure the consistency of multi-view depth projection. The goal is to make the depth estimates of the same scene point consistent across multiple views:

[0172]

[0173] D i (p): Depth value of view i.

[0174] Dj(p′): Projected depth value of the same scene point in view j.

[0175] Global prior optimization: The purpose of global prior optimization is to adjust the point cloud through the extracted scene prior information to make it conform to the overall geometric structure of the scene. The global prior loss is defined as:

[0176]

[0177] D prior (p): Global prior depth

[0178] D final (p): Depth value of the fused point cloud.

[0179] Total optimization (joint optimization) process:

[0180] The total optimization optimizes the point cloud by combining the weights of the geometric consistency loss and the global prior loss:

[0181] L total = L geo + λ prior ·L prior

[0182] λ prior : Adjust the balance between the two parts of the loss to ensure the trade-off between geometric consistency and global prior.

[0183] Use the gradient descent method to iteratively optimize the depth map and gradually reduce the value of L total . Recalculate D final (p) after each iteration until the loss function converges.

[0184] Introduce global optimization in point cloud fusion. By constraining multi-view geometric consistency and global geometric structure consistency, the global coherence and robustness of the point cloud are significantly improved. This method combines global prior information, effectively reducing the degradation of reconstruction quality caused by local error propagation, especially in low-texture, occluded, and repetitive texture regions. Global optimization can filter out abnormal estimations, reduce the influence of noise, and at the same time optimize the allocation of computing resources, improve the fusion efficiency and the adaptability to complex scenes, and finally achieve high-precision and globally consistent point cloud reconstruction.

[0185] F. Output:

[0186] 1). Output the optimized global point cloud depth map;

[0187] 2). Generate a high-precision 3D reconstruction model based on the optimized point cloud.

[0188] After adopting the above method, the following advantages are obtained:

[0189] 1. Expandable adaptive checkerboard sampling strategy: Break through the limitations of traditional fixed-direction propagation, propose an eight-direction propagation mode (including rectangular and V-shaped regions) and a dynamic sampling expansion mechanism, which can flexibly adapt to the geometric features of complex scenes, especially in low-texture, occluded, and complex geometric regions. Further optimize the propagation path and sampling efficiency through dynamic threshold control, significantly improving the coverage and accuracy of depth estimation;

[0190] 2. Hierarchical optimization depth estimation method: Innovatively propose a hierarchical optimization structure, divide the depth estimation process into a multi-scale pyramid structure, and refine and optimize from low resolution to high resolution layer by layer. At each scale, use photometric consistency to construct an initial depth hypothesis, construct a plane prior through a reliable point set and a local geometric plane, and combine geometric consistency to optimize depth estimation, reducing the depth deviation between multi-views. The hierarchical optimization structure realizes global consistency and multi-scene adaptability through layer-by-layer transmission and optimization, especially improving the robustness in low-texture and occluded regions;

[0191] 3. Multi-view point cloud fusion method based on dynamic weights: By comprehensively considering photometric consistency, geometric consistency, and global geometric prior information, dynamically adjust the weights of depth estimations of each view, so as to preferentially use high-reliability estimations during the fusion process, significantly enhancing the global consistency and local accuracy of the point cloud in complex scenes. Confidence filtering and global optimization strategies are introduced during the fusion process to further improve the accuracy of point cloud fusion and the coherence of the global geometric structure.

[0192] The above describes the present invention and its implementation manners. Such description is not restrictive, and the actual structure is not limited thereto. In summary, if those of ordinary skill in the art are inspired by it and, without departing from the gist of the present invention, design similar structural manners and embodiments to this technical solution without creative efforts, they shall fall within the protection scope of the present invention.

Claims

1. A multi-scale 3D reconstruction system and method based on dynamic optimization, characterized in that: It includes the following steps: A. Input and preprocessing; B. Scalable adaptive chessboard sampling; C. Layer-wise optimized depth estimation; D. Dynamic weight multi-view point cloud fusion; E. Global optimization; F. Output.

2. The multi-scale 3D reconstruction system and method based on dynamic optimization according to claim 1, characterized in that: The input and preprocessing comprises the following steps: 1) Input multi-view image sequence and corresponding camera parameters; 2) Perform pyramid decomposition on the input image to generate a multi-scale image sequence.

3. The multi-scale 3D reconstruction system and method based on dynamic optimization according to claim 1, characterized in that: The scalable adaptive chessboard sampling includes the following steps: 1) Initial sampling: Divide the reference image pixels into a red and black checkerboard pattern, and perform initial depth propagation according to the eight-directional propagation mode; 2) Dynamic expansion: Two expansion strategies (V-shaped end line propagation and rectangular area extended sampling) are adopted for the propagation failure area. The expansion range and number of samples are controlled according to the cost matrix and dynamic threshold, and finally a preliminary depth estimate with a wider coverage is generated.

4. The multi-scale 3D reconstruction system and method based on dynamic optimization according to claim 1, characterized in that: The hierarchical optimized depth estimation comprises the following steps: 1) Trusted point screening: Based on the photometric consistency cost calculation, the trusted point set is screened and the trusted point set I is constructed. cred ={x|c photo (θ x )<τ cred }; 2) Plane prior generation: a) Apply Delaunay triangulation to divide the credible points into triangular regions and generate local geometric planes; b) Use K nearest neighbor search on the edge area and combine Heron formula to calculate and construct a new triangular area; c) Optimize the objective function based on the fitted plane parameters 3) Layered optimization: Optimize depth estimation layer by layer: a) Optimize the initial depth by combining plane prior and geometric consistency cost at coarse resolution scale; b) After upsampling to the high-resolution layer, the low-level depth information is supplemented by the detail restorer to correct the estimation error layer by layer.

5. The multi-scale 3D reconstruction system and method based on dynamic optimization according to claim 1, characterized in that: The dynamic weight multi-view point cloud fusion comprises the following steps: 1) Weight construction, dynamic weights are calculated based on photometric consistency, geometric consistency and global priors: w total (p)=w photo (p)·w geo (p)·w prior (p) 2) Weighted fusion: weighted average of multi-view depth values ​​through dynamic weights to calculate the fused depth value: 3) Confidence filtering: Calculate the confidence of the fusion point through dynamic weights, and remove low-quality estimated points to optimize the global point cloud quality.

6. The multi-scale 3D reconstruction system and method based on dynamic optimization according to claim 1, characterized in that: The global optimization comprises the following steps: Geometric consistency optimization: Minimize geometric consistency loss by constraining multi-view depth consistency: Global prior optimization: Combine the global geometric prior and minimize the global prior loss: Joint Optimization: Comprehensive Loss Function L total =L geo +λ prior ·L prior ,The depth map is optimized by gradient descent until convergence.

7. The multi-scale 3D reconstruction system and method based on dynamic optimization according to claim 1, characterized in that: The output includes the following steps: 1) Output the optimized global point cloud depth map; 2) Generate a high-precision 3D reconstruction model based on the optimized point cloud.

Images