Double-end complementary online depth estimation and grid reconstruction coupling method and system

Abstract

The invention relates to the technical field of three-dimensional vision, in particular to a double-end complementary online depth estimation and grid reconstruction coupling method and system, and the method comprises the steps: selecting a reference frame from a multi-view image; performing down-sampling depth estimation on the reference frame, generating plane integrity priori according to effective priori generated by TSDF field ray projection fed back by an incremental grid reconstruction process of historical frame depth fusion, calculating multi-view matching cost under constraint of the plane integrity priori, and obtaining a depth map through local optimization and cost aggregation; a depth map is projected to a voxel coordinate system, a TSDF field is updated through TSDF voxel fusion, effective priori used for optimizing depth estimation in ray casting is obtained, and a triangular mesh is generated by extracting a zero contour surface from the TSDF field. Unification of high efficiency and accuracy of online grid reconstruction can be achieved, and the speed and accuracy of online reconstruction can be achieved at the mobile terminal.

Description

Technical Field

[0001] This invention relates to the field of 3D vision technology, and in particular to a dual-complementary online depth estimation and mesh reconstruction coupling method and system. Background Technology

[0002] Vision-based multi-view online 3D reconstruction, with its significant advantage of low cost, has been widely used in fields such as robot navigation, 3D modeling, autonomous driving, and mixed reality. It typically includes three stages: visual SLAM pose estimation, online depth estimation, and mesh reconstruction. Since visual SLAM technology is relatively mature, the key to achieving high-quality online 3D reconstruction lies in how to quickly calculate the accurate depth and mesh surface of the image.

[0003] Traditional depth estimation methods based on dense matching typically utilize fixed-window descriptors to calculate matching costs. These methods effectively reduce the computation time of matching, thus enabling rapid computation of image depth maps, but they are difficult to implement online reconstruction on mobile devices. Subsequent research, after platform optimization, has enabled online reconstruction on mobile devices; however, these methods do not consider the impact of large disparity variations on the cost during depth computation, resulting in limited depth accuracy and completeness. Many existing methods utilize surface normals of the image to constrain the matching cost calculation. However, these surface normals are usually obtained through the dominant tilt plane assumption, virtual normals, and depth maps. The normal calculation strategy is complex and computationally time-consuming, making it difficult to meet the requirements of online depth computation. Furthermore, existing methods often treat depth estimation and mesh reconstruction as two independent processes, neglecting their complementary relationship. Even some incremental mesh reconstruction methods only accept the depth map as input, failing to utilize the planar prior information of the mesh to optimize the depth estimation process. This independent calculation of depth and mesh is costly and time-consuming, and can lead to the accumulation of depth errors, making it difficult to meet the requirements of high-quality online reconstruction. Summary of the Invention

[0004] The purpose of this invention is to overcome the problems of low computational efficiency and poor reconstruction quality of traditional multi-view depth estimation and mesh reconstruction, and to provide a dual-complementary online depth estimation and mesh reconstruction coupling method and system. Through the dual-complementary process of online depth estimation and mesh reconstruction, the efficiency and accuracy of online mesh reconstruction are unified, enabling it to achieve the speed and accuracy of online reconstruction on mobile devices.

[0005] According to the design scheme provided by this invention, on the one hand, a dual-complementary online depth estimation and mesh reconstruction coupling method is provided, which performs dual-complementary online depth estimation and mesh reconstruction based on TSDF, wherein each complementary process includes:

[0006] Select a reference frame from multi-view images;

[0007] The effective prior generated by the TSDF field ray projection of the reference frame through downsampling depth estimation and incremental mesh reconstruction process based on the depth fusion of historical frames is used to generate a plane-complete prior. The multi-view matching cost under the plane-complete prior constraint is calculated, and the depth map is obtained through local optimization and cost aggregation.

[0008] The depth map is projected onto a voxel coordinate system, the TSDF field is updated by TSDF voxel fusion, and the effective priors for optimizing depth estimation are obtained from ray projection. Triangular meshes are generated by extracting zero isosurfaces from the TSDF field.

[0009] As a complementary online depth estimation and mesh reconstruction coupling method of the present invention, a reference frame is further selected from the multi-view image, including the following:

[0010] A scoring function is used to evaluate the impact of different camera configurations on 3D reconstruction accuracy based on the multi-view baseline length and camera tilt angle settings;

[0011] Images with a view score greater than a threshold are selected as reference frames based on the scoring function.

[0012] As a complementary online depth estimation and mesh reconstruction coupling method of the present invention, further, the effective prior generated by the TSDF field ray projection process, which performs downsampling depth estimation on the reference frame and generates a complete plane prior based on the incremental mesh reconstruction process of depth fusion of historical frames, includes:

[0013] The effective priors generated by the TSDF field ray projection are injected into the downsampling depth estimation process to obtain complete image prior information;

[0014] The plane prior of the current frame is estimated using the plane condition equation.

[0015] As a complementary online depth estimation and mesh reconstruction coupling method of the present invention, the multi-view matching cost under planar complete prior constraints is further calculated, including:

[0016] The discrete sampling range of image points is calculated using a non-uniformly spaced discrete sampling strategy, and the discrete sampling range is constrained using a plane prior.

[0017] The local homography transformation matrix between multiple view frames is calculated using planar priors, and the consistency of the matching cost calculation window between the current frame and the adjacent reference frames is ensured by local image deformation of adjacent reference frames.

[0018] The matching cost is calculated using the centrally symmetric Census matching cost.

[0019] As a complementary online depth estimation and mesh reconstruction coupling method of the present invention, the process of calculating the local homography transformation matrix between multiple view frames in the plane prior is further expressed as:

[0020] Where p′ and p are points in the two corresponding view frames, f i For the previous frame, f j For adjacent reference frames, This is the local homography transformation matrix between adjacent frames.

[0021] As a complementary online depth estimation and mesh reconstruction coupling method of the present invention, the depth map is further obtained through local optimization and cost aggregation, including:

[0022] The local cost aggregation and cost penalty are constrained by the color similarity function. In the local cost aggregation, the aggregation optimization order of horizontal first and then vertical is followed. The local cost optimization of the image point is carried out by weighted fusion of aggregation. The cost aggregation process is constrained by the planar prior. The range of cost aggregation is limited by the depth variation within the tolerance range. A uniform penalty is applied to depth positions that exceed the tolerance range.

[0023] The winner-takes-all algorithm is used to select the optimal solution from multiple depth values, and a complete depth map is obtained based on the optimal solution.

[0024] As a complementary online depth estimation and mesh reconstruction coupling method of the present invention, further, a penalty is uniformly applied to depth locations that exceed the tolerance range, including:

[0025] By using the discrete depth jump step size of the image point, the original depth position of the image point is offset to a depth position consistent with the surface direction, so as to perform zero-cost penalty transition of the image point.

[0026] Furthermore, this invention also provides a dual-complementary online depth estimation and mesh reconstruction coupled system, comprising: a depth estimation module and a mesh reconstruction module, wherein,

[0027] The depth estimation module performs downsampling depth estimation on the reference frame and generates a plane-complete prior based on the effective prior generated by the TSDF field ray projection feedback from the incremental mesh reconstruction process of historical frame depth fusion. It calculates the multi-view matching cost under the plane-complete prior constraint and obtains the depth map through local optimization and cost aggregation.

[0028] The mesh reconstruction module projects the depth map onto a voxel coordinate system, updates the TSDF field through TSDF voxel fusion and obtains effective priors from ray projection for optimizing depth estimation, and generates a triangular mesh by extracting zero isosurfaces from the TSDF field.

[0029] The beneficial effects of this invention are:

[0030] This invention utilizes depth and normal priors obtained through incremental TSDF field raycasting to solve for planar priors. A planar prior-guided strategy improves matching cost accuracy in scenes with large disparity variations. By incorporating prior depth and image segmentation results into local and semi-global optimization processes, it improves depth accuracy in weak textures and areas with large disparity variations, enhancing depth estimation accuracy in these regions and the robustness of indoor and outdoor scene reconstruction algorithms. By using image downsampling depth estimation and incremental TSDF field raycasting to calculate depth and normal priors, it constrains the depth search range of matching cost calculation and the semi-global matching algorithm, reducing the computation time for depth estimation and mesh reconstruction. The complementary online depth estimation and mesh reconstruction coupling effectively reuses prior information from the online mesh to optimize depth estimation, while the depth estimation results can in turn refine the mesh quality, simultaneously improving the speed and accuracy of online mesh reconstruction, achieving the speed and accuracy required for online reconstruction on mobile platforms. Attached image description:

[0031] Figure 1 This is a schematic diagram of the coupled process of online depth estimation and mesh reconstruction with dual-end complementarity in the embodiment;

[0032] Figure 2 This is a schematic diagram of the image plane prior calculation process in the embodiment;

[0033] Figure 3 This is a schematic diagram of the matching cost calculation process guided by planar priors in the embodiment;

[0034] Figure 4 This illustrates the local and semi-global cost optimization processes in the example. Detailed implementation method:

[0035] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the invention will be further described in detail below with reference to the accompanying drawings and technical solutions.

[0036] To address the problems of low computational efficiency and poor reconstruction quality in traditional multi-view depth estimation and mesh reconstruction, this invention provides a dual-complementary online depth estimation and mesh reconstruction coupling method, such as... Figure 1 As shown, a two-way complementary approach is used for online depth estimation and mesh reconstruction based on TSDF. Each complementary process includes:

[0037] S101. Select a reference frame from the multi-view image.

[0038] Specifically, a scoring function can be set based on the multi-view baseline length and camera tilt angle to evaluate the impact of different camera configurations on the accuracy of 3D reconstruction; images with a view score greater than a threshold are selected as reference frames based on the scoring function.

[0039] Figure 1 In the framework shown, for a given set of multi-view images with camera pose, two frames with F(B,θ) > 0.8 are selected as the preceding reference frames f of the current frame. i-1 and subsequent reference frame f i+1 .

[0040] S102. Perform downsampling depth estimation on the reference frame and generate a plane-complete prior based on the TSDF field ray projection feedback from the incremental mesh reconstruction process of historical frame depth fusion. Calculate the multi-view matching cost under the plane-complete prior constraint and obtain the depth map through local optimization and cost aggregation.

[0041] Specifically, the effective prior generated by the TSDF field ray projection can be injected into the downsampling depth estimation process to obtain complete image prior information; the plane prior of the current frame can be estimated using the plane condition equation.

[0042] The Raycasting method can be used to compute effective prior information of the current frame from the TSDF fusion field reconstructed by incremental meshing. This effective prior information includes an effective mask map, a depth map, and a normal prior map, such as... Figure 2 The area shown by the blue dashed line. Considering that the depth and normal priors obtained from the TSDF field of historical frame depth fusion may contain some missing information, in order to fill in these missing areas, the image plane prior calculation process in this embodiment is as follows: Figure 2 As shown, the raycasting results of the incremental TSDF field are injected into the downsampling depth estimation process of online depth estimation, thereby obtaining complete and reliable image prior information. Finally, the plane condition equation n can be used. x +n x +n x +z l =0 solves for the planar prior of the current frame and uses it as the image planar prior in the high-resolution depth estimation process to constrain the cost calculation and cost aggregation process.

[0043] After obtaining the planar prior, the multi-view matching cost under the complete planar prior constraint is calculated and can be designed to include:

[0044] The discrete sampling range of image points is calculated using a non-uniformly spaced discrete sampling strategy, and the discrete sampling range is constrained using a plane prior.

[0045] The local homography transformation matrix between multiple view frames is calculated using planar priors, and the consistency of the matching cost calculation window between the current frame and the adjacent reference frames is ensured by local image deformation of adjacent reference frames.

[0046] The matching cost is calculated using the centrally symmetric Census matching cost.

[0047] like Figure 3 As shown, firstly, the discrete sampling depth range [Z] of the image points is calculated using a non-equidistant discrete sampling strategy. min Z max By using the solved planar prior depth to constrain the original discrete sampling range, the depth search range for matching cost calculation is effectively reduced. The constrained depth search range significantly reduces the computation time for matching costs. Secondly, considering that image points with large disparity variations are prone to generating incorrect matching costs during multi-view frame matching cost calculation, this embodiment utilizes planar priors to calculate the local homography transformation matrix between multi-view frames. By using local image deformation of adjacent reference frames to reduce the forward parallelism bias in the matching cost calculation, the current frame f can be optimized. i With adjacent reference frame f j The consistency of the matching cost calculation window is improved, thereby enhancing the accuracy of the matching calculation. The formula for calculating the local homography transformation matrix of multi-view frames can be expressed as follows:

[0048]

[0049] In the formula, p′ represents the image point p in the adjacent reference frame f. j The homogeneous coordinates of the projection point on the z-axis, p Let n be the discrete sampling depth corresponding to image point p. p Let be the normal vector of the plane prior, K be the intrinsic parameter of the camera, and [R t] represent the relative transformation matrix of image point p in the current frame with respect to the adjacent reference frame p′.

[0050] Finally, in order to calculate the matching cost of image points at different sampling depths, the centrally symmetric Census matching cost is used to calculate the matching cost, which saves half the cost calculation time while avoiding the interference of noise at the central image point.

[0051]

[0052] In the formula, WCT n,m Match the bit chain corresponding to the center image point (u,v) on the matching window. This is a bit-level concatenation symbol used to connect bit values ​​calculated by the symbol judgment function s(u,v) = 0 if u ≥ v; s(u,v) = 1 otherwise. P(ui,vj) and P(u+i,v+j) are I... n×m The grayscale values ​​of image points in all symmetrical regions of the neighborhood window.

[0053] In online depth estimation, depth maps are obtained through local optimization and cost aggregation, and can be designed to include:

[0054] The local cost aggregation and cost penalty are constrained by the color similarity function. In the local cost aggregation, the aggregation optimization order of horizontal first and then vertical is followed. The local cost optimization of the image point is performed by aggregation weighted fusion. The cost aggregation process is constrained by the planar prior. The range of cost aggregation is limited by the depth variation within the tolerance range. A uniform penalty is applied to the depth position that exceeds the tolerance range. The discrete depth jump step of the image point can be used to shift the original depth position of the image point to a depth position consistent with the surface direction, so as to perform zero-cost penalty transition of the image point.

[0055] The winner-takes-all algorithm is used to select the optimal solution from multiple depth values, and a complete depth map is obtained based on the optimal solution.

[0056] Considering that in areas with weak texture and large parallax variations, pixels in the same region are prone to aggregation cost errors due to a lack of local correlation information. For example... Figure 4 As shown, the flood filling segmentation results are incorporated into the cost aggregation process. The color similarity function Label(p) is used to constrain local cost aggregation and cost penalty, as follows: Figure 4 As shown in (a) above. The calculation formula is as follows:

[0057] Label(p) = I p ·Γ(|I p -I q |≤ε),q∈N p (3)

[0058] In the formula, N p Let I be the four neighborhood of image point p. p Let Γ(·) be the gray value of pixel p, Γ(·) be the judgment function, and ε be the color tolerance threshold. Specifically, ε can be set to 5, and can be adjusted according to the needs in actual applications.

[0059] To improve the smoothness of the depth map, based on the image segmentation results, a weighted fusion formula is used to optimize the local cost of image points. To reduce computation, cost aggregation is performed every 5 pixels. The specific aggregation process can be represented as follows:

[0060]

[0061] In the formula, Let represent the aggregation cost of image point p in the horizontal and vertical directions, respectively. This paper follows the aggregation optimization order of horizontal first and then vertical. w is the weight between image point p and image points in the neighborhood of step. Based on the principle of "weighting is only allowed in the same region", neighboring image points are assigned a weight value of 0 or 1 to optimize the aggregation cost.

[0062] To solve for depth online, this embodiment utilizes a planar prior depth constraint cost aggregation process, by setting a certain tolerance range. The depth variation within the range is used to limit the range of cost aggregation, such as Figure 4 As shown in (b), a P2 penalty is uniformly applied to depth locations exceeding the tolerance range, thereby optimizing the cost aggregation time. Secondly, considering that traditional semi-global matching strategies may apply incorrect cost penalties to the same region of pixels in scenes with weak textures and large disparity variations, such as... Figure 4 As shown in (c), the original depth position l is determined by using the discrete depth jump step size ΔL of the image point. p Offset to a depth position l aligned with the surface direction p +ΔL, such as Figure 4 As shown in (d), a zero-cost penalty transition is achieved for image point p, improving depth estimation accuracy without increasing time consumption. The aggregation cost for each r direction is calculated as follows:

[0063]

[0064] For each pixel, the cost is aggregated for each depth position from the eight basic directions r. The aggregated cost for each direction is summed according to equation (6) to obtain the final cost aggregation cost value S(p,l). p ).

[0065]

[0066] The optimal depth sampling position is determined using the WTA (Winner Takes All) algorithm. The optimal depth value of a pixel is determined using formula (7). Iterate through each pixel to generate an accurate and complete depth map.

[0067]

[0068] S103. Project the depth map onto the voxel coordinate system, update the TSDF field through TSDF voxel fusion and obtain the effective priors in ray projection for optimizing depth estimation, and generate a triangular mesh by extracting the zero isosurface from the TSDF field.

[0069] In this embodiment, an incremental TSDF mesh reconstruction method can be used to optimize the mesh reconstruction process by utilizing the incremental weighted fusion algorithm of TSDF, avoiding repeated calculation of historical frame points, and effectively reducing computation time and storage consumption.

[0070] Given an initialized 3D voxel grid, the multi-view depth map is projected onto the 3D voxel grid using the image pose information according to the coordinate transformation relationship in (8), thereby obtaining the depth value z corresponding to each voxel. v .

[0071]

[0072] For each voxel V, the TSDF value and weight value of each voxel within the cutoff distance [-τ,τ] (τ=3cm in this paper) are incrementally calculated and updated using the weighted average fusion formula of Equation (9):

[0073]

[0074] In the formula, T t p (V) W t (V) and W t-1 (V) represents the TSDF value and weight value of voxel V at time t and time t-1, respectively. t (V)=W t-1 (V)+1, sgn(·) is the sign judgment function, min(·) is the auxiliary function, when z v When the value is less than or equal to τ, the voxels are updated incrementally according to Equation (9). Otherwise, the voxels will no longer be updated, thus avoiding invalid calculations and noise, and making the results more robust.

[0075] After all voxel values ​​have been updated, the Marching Cubes algorithm is used to extract isosurfaces with a distance weighted sum of zero from the updated voxel mesh, which are then used as the surfaces of the reconstructed scene to generate the final mesh reconstruction result.

[0076] This solution establishes a coupled framework for online depth estimation and mesh reconstruction, leveraging bidirectional optimization of both methods to effectively improve the speed and accuracy of mesh reconstruction. It utilizes image downsampling depth estimation and incremental TSDF field raycasting to rapidly calculate image depth and normal priors, avoiding the high complexity of traditional normal calculations and reducing computational time. Furthermore, it employs a matching cost calculation strategy guided by planar priors, significantly reducing computation time by constraining the depth search range for cost calculation and cost aggregation. Simultaneously, the introduction of image planar priors and segmentation results makes depth values ​​more accurate under weak textures and large disparity variations, effectively improving the global accuracy of the depth map.

[0077] Furthermore, based on the above method, this embodiment of the invention also provides a dual-complementary online depth estimation and mesh reconstruction coupled system, comprising: a depth estimation module and a mesh reconstruction module, wherein,

[0078] The depth estimation module performs downsampling depth estimation on the reference frame and generates a plane-complete prior based on the effective prior generated by the TSDF field ray projection feedback from the incremental mesh reconstruction process of historical frame depth fusion. It calculates the multi-view matching cost under the plane-complete prior constraint and obtains the depth map through local optimization and cost aggregation.

[0079] The mesh reconstruction module projects the depth map onto a voxel coordinate system, updates the TSDF field through TSDF voxel fusion and obtains effective priors from ray projection for optimizing depth estimation, and generates a triangular mesh by extracting zero isosurfaces from the TSDF field.

[0080] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0081] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0082] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0083] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0084] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A dual-complementary online depth estimation and mesh reconstruction coupling method, characterized in that, A two-way complementary process for online depth estimation and mesh reconstruction based on TSDF, each complementary process including: Select a reference frame from multi-view images; The effective prior generated by the TSDF field ray projection of the reference frame through downsampling depth estimation and incremental mesh reconstruction process based on the depth fusion of historical frames is used to generate a plane-complete prior. The multi-view matching cost under the plane-complete prior constraint is calculated, and the depth map is obtained through local optimization and cost aggregation. The depth map is projected onto a voxel coordinate system, the TSDF field is updated by TSDF voxel fusion, and the effective priors for optimizing depth estimation are obtained from ray projection. Triangular meshes are generated by extracting zero isosurfaces from the TSDF field.

2. The dual-end complementary online depth estimation and mesh reconstruction coupling method according to claim 1, characterized in that, Select a reference frame from the multi-view images, which includes the following: A scoring function is used to evaluate the impact of different camera configurations on 3D reconstruction accuracy based on the multi-view baseline length and camera tilt angle settings; Images with a view score greater than a threshold are selected as reference frames based on the scoring function.

3. The dual-end complementary online depth estimation and mesh reconstruction coupling method according to claim 1, characterized in that, The effective prior generated by the TSDF field ray projection, which is generated by downsampling depth estimation of the reference frame and incremental mesh reconstruction based on depth fusion of historical frames, includes: The effective priors generated by the TSDF field ray projection are injected into the downsampling depth estimation process to obtain complete image prior information; The plane prior of the current frame is estimated using the plane condition equation.

4. The dual-end complementary online depth estimation and mesh reconstruction coupling method according to claim 1 or 3, characterized in that, The calculation of the multi-view matching cost under planar complete prior constraints includes: The discrete sampling range of image points is calculated using a non-uniformly spaced discrete sampling strategy, and the discrete sampling range is constrained using a plane prior. The local homography transformation matrix between multiple view frames is calculated using planar priors, and the consistency of the matching cost calculation window between the current frame and the adjacent reference frames is ensured by local image deformation of adjacent reference frames. The matching cost is calculated using the centrally symmetric Census matching cost.

5. The dual-end complementary online depth estimation and mesh reconstruction coupling method according to claim 4, characterized in that, The process of calculating the local homography transformation matrix between multiple view frames using planar priors is expressed as follows: Where p′ and p are points in the two corresponding view frames, f i For the previous frame, f j For adjacent reference frames, This is the local homography transformation matrix between adjacent frames.

6. The dual-end complementary online depth estimation and mesh reconstruction coupling method according to claim 1, characterized in that, Depth maps are obtained through local optimization and cost aggregation, including: The local cost aggregation and cost penalty are constrained by the color similarity function. In the local cost aggregation, the aggregation optimization order of horizontal first and then vertical is followed. The local cost optimization of the image point is carried out by weighted fusion of aggregation. The cost aggregation process is constrained by the planar prior. The range of cost aggregation is limited by the depth variation within the tolerance range. A uniform penalty is applied to depth positions that exceed the tolerance range. The winner-takes-all algorithm is used to select the optimal solution from multiple depth values, and a complete depth map is obtained based on the optimal solution.

7. The dual-end complementary online depth estimation and mesh reconstruction coupling method according to claim 6, characterized in that, A uniform penalty is applied to depth locations exceeding the tolerance range, including: By using the discrete depth jump step size of the image point, the original depth position of the image point is offset to a depth position consistent with the surface direction, so as to perform zero-cost penalty transition of the image point.

8. A dual-end complementary online depth estimation and mesh reconstruction coupled system, characterized in that, It includes: a depth estimation module and a mesh reconstruction module, wherein, The depth estimation module performs downsampling depth estimation on each selected reference frame and generates a plane-complete prior based on the effective prior generated by the TSDF field ray projection feedback from the incremental mesh reconstruction process of historical frame depth fusion. It calculates the multi-view matching cost under the plane-complete prior constraint and obtains the depth map through local optimization and cost aggregation. The mesh reconstruction module projects the depth map onto a voxel coordinate system, updates the TSDF field through TSDF voxel fusion and obtains effective priors from ray projection for optimizing depth estimation, and generates a triangular mesh by extracting zero isosurfaces from the TSDF field.

9. An electronic device, characterized in that, include: At least one processor, and a memory coupled to said at least one processor; The memory stores a computer program that can be executed by the at least one processor to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, enables the implementation of the method as described in any one of claims 1 to 7.

Images