Fabric-like sewing type cavity self-healing endoscope scene reconstruction method
By employing a fabric-like stitching-type self-healing method for endoscopic scene reconstruction, and utilizing techniques such as 2D Gaussian scene initialization modeling and camera pose tracking, the problem of structural missingness and mapping voids in endoscopic 3D reconstruction was solved, achieving stable, continuous, and high-quality reconstruction in complex tubular organ environments.
Patent Information
- Application Number
- CN202610185088.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-01
AI Technical Summary
Existing endoscopic 3D reconstruction methods are prone to structural defects and mapping gaps in complex tubular organ environments, making it difficult to guarantee topological continuity and overall integrity, and lacking the ability to adaptively complete insufficiently observed areas.
An endoscope scene reconstruction method with a fabric-like stitch-like hole self-healing is adopted. Through two-dimensional Gaussian scene initialization modeling, camera pose tracking, Gaussian expansion and keyframe sampling, mapping and hole-aware completion, and bundle adjustment optimization with normal smoothing constraints, a stable three-dimensional reconstruction framework is constructed to achieve topological awareness and self-healing completion of the mapping process.
In the endoscopic environment with limited field of view, frequent occlusion and sparse texture, continuous and stable 3D reconstruction was achieved, effectively suppressing mapping holes and geometric discontinuities, and improving the integrity and geometric quality of the reconstruction results.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
A method for endoscopic scene reconstruction using fabric-like stitching-type self-healing voids Technical Field
[0003] This invention relates to an endoscopic scene reconstruction method, specifically a fabric-like sewn-up self-healing endoscopic scene reconstruction method, belonging to the field of medical imaging and computer vision technology. Background Technology
[0005] With the rapid development of minimally invasive surgical techniques and computer vision technology, 3D reconstruction and scene mapping methods based on endoscopic images have been widely used in medical navigation, intraoperative positioning, and intelligent diagnosis and treatment. Endoscopes typically move within long, narrow, curved, and sparsely textured tubular organs, where the imaging angle is limited and illumination changes drastically, leading to problems such as scale drift, structural breaks, and map discontinuities during reconstruction. Therefore, how to construct continuous, stable, and complete 3D scene models within complex physiological luminal environments has become an important research topic in the field of medical visual reconstruction.
[0006] Existing endoscopic mapping methods mostly rely on sparse feature points or dense depth estimation for geometric reconstruction, and combine bundle adjustment and local mapping mechanisms to jointly optimize camera pose and scene structure. In recent years, dense reconstruction methods based on two-dimensional Gaussian representation have gradually emerged. By representing the scene as a set of differentiable two-dimensional Gaussian distributions, high-quality rendering and continuous modeling are achieved, which improves reconstruction accuracy and robustness to some extent. However, these methods mainly rely on local viewpoint observation information for updates. When the endoscopic field of view is obstructed, rapidly rotated, or the sidewall texture is missing, structural holes or locally missing regions are easily generated during the mapping process, making it difficult to guarantee the continuity of the overall geometry and topological consistency.
[0007] Meanwhile, the loss function used in existing beam adjustment optimization processes mainly consists of color reconstruction error, structural similarity error, and geometric consistency error. Its optimization objective focuses on local reprojection accuracy, which is insufficient for constraining the overall geometric continuity of Gaussian scenes. After complex curved structures or multiple iterations of optimization, problems such as rough local geometric surfaces, unstable normal distribution, and lack of smooth transition between adjacent Gaussians can easily occur, resulting in jagged or discontinuous features in the detailed regions of the reconstructed model.
[0008] Furthermore, existing methods lack an explicit mechanism for determining the completeness of the mapping during the mapping process, making it impossible to actively perceive and address void regions. They typically rely on posterior smoothing or global optimization for indirect repair, making it difficult to adaptively complete missing regions and maintain topological consistency during the mapping phase. This makes the reconstruction process susceptible to insufficiently observed areas in complex luminal structures, limiting the model's stability and generalization ability in real-world endoscopic scenarios.
[0009] In summary, current 3D reconstruction and scene mapping technologies based on endoscopic images still face multiple technical challenges in complex tubular organ environments. On the one hand, existing endoscopic reconstruction methods mostly rely on local perspective observations and geometric consistency constraints for mapping updates. In areas with limited field of view, frequent occlusion, or sparse textures, structural defects and mapping holes are easily generated, making it difficult to guarantee the topological continuity and overall integrity of the reconstruction results. This is especially true in curved or long-distance tubular scenes, where the model is prone to breakage, discontinuity, or local missing parts. On the other hand, although some dense reconstruction methods based on 2D Gaussian representation have made some progress in rendering quality and local geometric representation, their mapping process is mainly driven by low-level observation data and lacks macroscopic geometric prior constraints on the overall morphological structure of the organ, making it difficult to reasonably complete insufficiently observed areas. Summary of the Invention
[0011] To address the aforementioned technical shortcomings, the purpose of this invention is to provide an endoscopic scene reconstruction method that uses a fabric-like stitching-type self-healing cavity mechanism. This method enables topological awareness and self-healing completion of the mapping process in complex tubular organ scenes, improving the continuity, stability, and geometric quality of endoscopic 3D reconstruction results. It is applicable to various application scenarios such as minimally invasive surgical navigation, medical 3D modeling, and intelligent diagnosis and treatment.
[0012] To achieve the above objectives, the present invention adopts the following technical solution:
[0013] A method for reconstructing endoscopic scenes using a fabric-like patch-like self-healing technique for creating voids includes the following steps:
[0014] S1. Initial modeling of a two-dimensional Gaussian scene: A set of two-dimensional Gaussian primitives is used to model the RGBD image sequence frames continuously acquired by the endoscope, resulting in the initial two-dimensional Gaussian scene. ;
[0015] S2, Camera pose tracking; the initial 2D Gaussian scene output in step S1. While keeping the position unchanged, perform camera pose estimation on the current image sequence frames to obtain the camera pose curve;
[0016] S3, Gaussian expansion and keyframe sampling strategy; the initial two-dimensional Gaussian scene output in step S1. Based on this, a Gaussian expansion strategy is used to supplement the incomplete regions observed due to the camera pose estimation transformation in step S2, and the supplemented 2D Gaussian scene is output. And key frames are sampled as subsequent inputs using a key frame sampling strategy;
[0017] S4, Graph Mapping and Hole-Aware Completion Module; in the completed 2D Gaussian scene Based on this, the supplemented two-dimensional Gaussian scene is further iteratively updated through graph mapping. Obtain the updated 2D Gaussian scene Furthermore, topological and continuity constraints are introduced for hole detection. When holes appear in the mapping, an adaptive completion mechanism is triggered, outputting a two-dimensional Gaussian scene with optimized mapping. To achieve continuous and consistent scene mapping;
[0018] S5. Bundle adjustment optimization with smoothing constraints; input the camera pose curve from step S2 and the 2D Gaussian scene after mapping optimization from step S4. Subsequently, a gradient descent-based bundle adjustment method was used to optimize the joint camera pose and the two-dimensional Gaussian scene.
[0019] The initial two-dimensional Gaussian scene obtained in step S1 of this invention for:
[0020]
[0021] Where: each two-dimensional Gaussian By location Rotation Scaling factor Opacity and RGB colors definition;
[0022] Rotation matrix Defined as , They are two orthogonal tangential vectors. These are the primitive normals and the scaling matrix. Defined as a diagonal matrix, where the last element is zero and the other two elements are zero. .
[0023] Step S2 of the present invention is as follows:
[0024] Camera pose is represented as Assuming the camera's initialization velocity for a new pose is constant, given two previous poses... Calculate the speed as and will Initialize, and then iteratively optimize the camera pose using gradient descent;
[0025] To perform geometrically robust tracking, a method with projected point-to-point distance is used. and distance from point to plane Surface normal sensing tracking regularization method: Let It's a point from the rendering depth. It is a point derived from the ground truth depth;
[0026] definition for
[0027] in: for value; The truth depth; For rendering depth;
[0028] definition for
[0029] in: , It is the true ground normal obtained by performing finite difference on nearby points. They are shaft and Difference on the axis;
[0030] The final loss function for camera pose tracking is defined as follows:
[0031]
[0032] By minimizing the loss function for camera pose tracking, a continuous, fixed, optimized camera pose curve is output.
[0033] in: It is the loss in rendering the image;
[0034] It is the rendered color after affine exposure adjustment;
[0035] It is the true color of the ground.
[0036] Step S3 of the present invention specifically includes:
[0037] S31: Gaussian expansion strategy: Add all RGBD image sequences continuously acquired by the endoscope in step S1 to the keyframe candidate list, and update the initial two-dimensional Gaussian scene every K frames. In the initial two-dimensional Gaussian scene The text supplements the real tissues that were not observed due to new visual effects generated by changes in camera pose.
[0038] Using the rendered outline This represents the observation area of the current camera pose in each frame, and the contour value is less than a threshold. The region and the region above the reconstructed tissue surface with the ground truth geometry are used to extend the initial two-dimensional Gaussian scene. The new 2D Gaussian scene is initialized using a keyframe candidate list to obtain the supplemented 2D Gaussian scene. ;
[0039] S32: Keyframe sampling strategy: Sampling from the keyframe candidate list in step S31 For each keyframe, a pose consistency strategy that simultaneously considers translation and rotation errors is employed. The sampling probability of each candidate frame is defined as follows:
[0040]
[0041] in: It is the distance between locations;
[0042] It is the magnitude error of the rotation quaternion;
[0043] It is a time interval relative to the current frame;
[0044] It is a scaling constant set to 0.2;
[0045] The sampling probability of each candidate frame is normalized so that ,in It is the predefined probability of the current frame. All candidate frames are sorted and sampled according to the cumulative probability distribution function.
[0046] Step S4 of the present invention specifically includes:
[0047] S41: Mapping: After outputting the fixed and optimized camera pose curve in step S2, map the 2D Gaussian scene supplemented in step S31. The updated 2D Gaussian scene is obtained by iteratively updating the scene using the gradient descent method. Define depth distortion loss to minimize the distance between ray-sheet intersections:
[0048]
[0049] in: , It is the first The weights and depths of the intersections;
[0050] , It is the first The weights and depths of the intersections;
[0051] We also define a normal consistency loss function. To constrain the surface normals of a two-dimensional Gaussian:
[0052]
[0053] in: It is the index along the ray and the intersection plate;
[0054] It is the normal of the image relative to the camera;
[0055] S42: Hole-aware completion: The updated 2D Gaussian scene is generated after the mapping output in step S41. Subsequently, a cylindrical topology constraint framework based on the Frenet frame is introduced as a hole detection and triggering mechanism; the fixed and optimized camera pose curve obtained in step S2 is regarded as a spatial curve. and at each sampling location on the trajectory Constructing a dynamic Frenet frame coordinate system:
[0056] Where: tangent vector Pointing in the direction of camera movement;
[0057] Principal Normal Vector Indicates the direction of the trajectory's curvature;
[0058] binormal vector Perpendicular to the oscillating plane;
[0059] For the updated 2D Gaussian scene Two-dimensional Gaussian points First, in the space curve Find the projection point that has the shortest Euclidean distance from it. :
[0060]
[0061] Establish a corresponding Frenet coordinate system at this projection point, and convert the Gaussian point in the world coordinate system... Projecting onto the current Frenet frame coordinate system yields a local cylindrical coordinate representation:
[0062]
[0063] Where: radial distance is defined as ;
[0064] In a local sliding window Internal statistical mean radius Construct the topological constraint loss function for the cylinder. :
[0065]
[0066] in: The confidence weighting function assigns a larger weight to the observation density region.
[0067] The topological constraint loss function of the cylinder With a given threshold If a comparison is made, If a hole structure is found in the current mapping region, an adaptive completion mechanism is triggered, and points corresponding to the projection points are selected within the neighborhood of the hole. The shortest set of edge Gaussian points Along the Frenet frame normal vector , The radial direction spanned generates new two-dimensional Gaussian initial parameters to fill the holes:
[0068] like Then, the adaptive completion mechanism is stopped, and only the routine update of the two-dimensional Gaussian parameters in step S41 (mapping mapping) is performed. The final mapping loss function is defined as follows:
[0069]
[0070] Output mapping optimized 2D Gaussian scene
[0071] in: It is the loss in color reconstruction;
[0072] It is the structural similarity index loss. .
[0073] Step S5 of the present invention specifically includes:
[0074] Keyframes are selected according to the keyframe sampling strategy in step S32. A bundle adjustment optimization is performed once every 200 iterations of camera pose tracking in step S2 to initially update the camera pose and the mapped and optimized 2D Gaussian scene. ;
[0075] Given camera pose And the 2D Gaussian scene after mapping optimization The loss function of the bundle adjustment method is adopted.
[0076] Based on this, a normal smoothing constraint is introduced to address the issues of rough local geometric features and discontinuities in Gaussian scenes caused by the bundle adjustment method. A normal smoothing loss function combining normal consistency and depth gradient anisotropy is designed:
[0077]
[0078] Where: weighted normal consistency loss function , It is a mask function, if adjacent points If it is located at the edge of a wrinkle (the difference in depth gradient or normal exceeds a set threshold), then That is, mutation is allowed, otherwise That is, forced smoothing; while edge-protected deep smoothing loss function
[0079] Finally, a loss function incorporating normal smoothing constraints is used for bundle adjustment optimization:
[0080]
[0081] The output completed the precise correction of camera pose. High-quality 3D scene representation with geometric feature optimization.
[0082] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0083] This invention proposes a fabric-like, stitch-like, self-healing method for reconstructing endoscopic scenes. By constructing a unified reconstruction framework consisting of 2D Gaussian scene initialization modeling, camera pose tracking, Gaussian expansion and keyframe sampling, mapping and hole-aware completion, and bundle adjustment optimization incorporating normal smoothing constraints, it achieves continuous, complete, and high-quality reconstruction of scene structures in complex tubular organ scenes. This method can operate stably in endoscopic environments with limited field of view, frequent occlusion, and sparse textures, effectively suppressing mapping holes and geometric discontinuities.
[0084] The two-dimensional Gaussian scene initialization model constructed in this invention utilizes a set of two-dimensional Gaussian vectors to uniformly model a continuously acquired RGBD image sequence, enabling explicit representation of the scene surface and thus establishing a differentiable and optimizable initial scene model. Compared to traditional point cloud or mesh models, this representation method has stronger continuous expressive capabilities, providing a stable and consistent geometric foundation for subsequent pose tracking, mapping, and bundle adjustment.
[0085] Based on a fixed initial 2D Gaussian scene, this invention constructs a camera pose tracking method that combines surface normal-aware tracking regularization with point-to-point distance and point-to-plane distance to iteratively optimize the camera pose. This method fully utilizes the geometric constraints between the rendered depth and the true depth, making the pose estimation process more robust to noise and local errors. It can output continuous, stable, and accurate camera motion trajectories, providing a reliable center axis reference for subsequent mapping and completion.
[0086] Furthermore, the Gaussian expansion and keyframe sampling strategy proposed in this invention gradually incorporates newly observed real tissue regions due to camera pose changes into the 2D Gaussian scene model through the Gaussian expansion strategy, effectively compensating for areas not covered in the initial modeling process and continuously enriching and improving the scene representation as the observation process progresses. Simultaneously, the keyframe sampling strategy selects representative keyframes from candidate frames as input for subsequent optimization, avoiding the computational burden caused by redundant data and improving the overall system's operating efficiency and stability.
[0087] In the mapping and hole-aware completion module, this invention introduces a cylindrical topological constraint based on a Frenet frame as a hole-aware and triggering mechanism, building upon conventional mapping optimization. By treating the camera pose trajectory as a spatial curve and constructing a moving Frenet frame coordinate system, a consistent constraint is applied to the radial distribution of 2D Gaussian points to the central axis. When a hole structure is detected in the mapping region, an adaptive completion mechanism is automatically triggered. Using the Gaussian parameters at the hole edge as a reference, new 2D Gaussian parameters are gradually generated radially, allowing the missing region to progressively close like stitching a piece of fabric. This mechanism significantly improves the integrity and continuity of the reconstruction results in terms of topological structure, effectively avoiding the mapping breakage problem caused by limited field of view in traditional methods.
[0088] In the bundle adjustment optimization module incorporating smoothing constraints, this invention introduces normal smoothing constraints that combine normal consistency and depth gradient anisotropy while jointly optimizing camera pose and 2D Gaussian scene parameters. This smoothing process continuously enhances and smooths the local geometry. While maintaining the edges of wrinkles and abrupt changes in the real structure, it effectively suppresses noise accumulation and unnecessary geometric oscillations, resulting in a smoother and more natural surface morphology and a more stable and reliable geometric representation of the final 2D Gaussian scene.
[0089] In summary, this invention has the following significant advantages over existing technologies: First, it achieves continuous and dense representation of endoscopic scenes through two-dimensional Gaussian scene modeling and Gaussian expansion mechanisms, improving the stability of the reconstruction process. Second, it enables the mapping process to have a self-healing ability similar to fabric patching through a hole-sensing and completion mechanism based on the Frenet frame, significantly improving the problems of mapping holes and topological discontinuities. Third, it effectively alleviates local geometric roughness and Gaussian scene discontinuities by introducing normal smoothing constraints for bundle adjustment optimization, improving the geometric quality of the reconstructed model. Fourth, the overall framework is modularly designed and can be seamlessly integrated with existing two-dimensional Gaussian mapping and bundle adjustment systems, exhibiting good engineering feasibility. Fifth, this method is applicable to complex tubular organ scenes and has broad practical value in applications such as endoscopic navigation, medical 3D modeling, and intelligent assisted surgery. Attached Figure Description
[0091] Figure 1 is a schematic diagram of the overall structure of the present invention;
[0092] Figure 2 is an overall flowchart of the method of the present invention;
[0093] Figure 3 shows the RGBD rendering effect output by this invention;
[0094] Figure 4 shows a comparison of the three-dimensional reconstruction results of different methods on the same endoscopic sequence, where (a) is a comparison of the three-dimensional reconstruction models of the intestinal scene, and (b) is the reconstruction result of the corresponding camera trajectory. Detailed Implementation
[0096] The invention will now be further described with reference to the accompanying drawings.
[0097] This invention addresses the problems of limited field of view, frequent occlusion, and holes and geometric discontinuities in endoscopic imaging during mapping. It proposes a fabric-like stitching-type hole self-healing endoscopic scene reconstruction method. The overall structure is shown in Figure 1, and the overall flowchart is shown in Figure 2. The system includes a unified planning framework for two-dimensional Gaussian scene initialization modeling, camera pose tracking, Gaussian expansion and keyframe sampling, mapping and hole-aware completion, and bundle adjustment optimization with normal smoothing constraints. The system takes continuously acquired RGBD image sequences from an endoscope as input. First, an initial 2D Gaussian scene representation is constructed using a 2D Gaussian scene initialization modeling module. Then, camera pose tracking is performed on this scene to obtain a continuous and stable camera motion trajectory. Further, through Gaussian expansion and keyframe sampling strategies, newly observed real tissue regions are gradually added to the 2D Gaussian scene. Based on this, a mapping and hole-aware completion module is introduced to iteratively optimize and update the 2D Gaussian scene. Automatic detection and triggering of holes in the mapping are achieved using cylindrical topological constraints based on a Frenet frame, enabling missing areas to progressively self-heal in a fabric-like stitching manner. Finally, through bundle adjustment optimization incorporating normal smoothing constraints, joint fine-tuning of the camera pose and the 2D Gaussian scene is performed, outputting a structurally continuous, geometrically smooth, and topologically complete 3D endoscopic scene model.
[0098] The present invention proposes a method for reconstructing endoscopic scenes using a fabric-like stitch-mending self-healing technique for creating voids, comprising the following steps:
[0099] S1. Initial modeling of a two-dimensional Gaussian scene: A set of two-dimensional Gaussian primitives is used to model the RGBD image sequence frames continuously acquired by the endoscope, resulting in the initial two-dimensional Gaussian scene. ;
[0100] The initial two-dimensional Gaussian scene for:
[0101]
[0102] Where: each two-dimensional Gaussian By location Rotation Scaling factor Opacity and RGB colors definition;
[0103] Rotation matrix Defined as , They are two orthogonal tangential vectors. These are the primitive normals and the scaling matrix. Defined as a diagonal matrix, where the last element is zero and the other two elements are zero. .
[0104] The two-dimensional Gaussian scene initialization model constructed in this invention utilizes a set of two-dimensional Gaussian vectors to uniformly model a continuously acquired RGBD image sequence, enabling explicit representation of the scene surface and thus establishing a differentiable and optimizable initial scene model. Compared to traditional point cloud or mesh models, this representation method has stronger continuous expressive capabilities, providing a stable and consistent geometric foundation for subsequent pose tracking, mapping, and bundle adjustment.
[0105] S2, Camera pose tracking; the initial 2D Gaussian scene output in step S1. While keeping the position unchanged, perform camera pose estimation on the current image sequence frames to obtain the camera pose curve;
[0106] Step S2 is as follows:
[0107] Camera pose is represented as Assuming the camera's initialization velocity for a new pose is constant, given two previous poses... Calculate the speed as and will Initialize, and then iteratively optimize the camera pose using gradient descent;
[0108] To perform geometrically robust tracking, a method with projected point-to-point distance is used. and distance from point to plane Surface normal sensing tracking regularization method: Let It's a point from the rendering depth. It is a point derived from the ground truth depth;
[0109] definition for
[0110] in: for value; The truth depth; For rendering depth;
[0111] definition for
[0112] in: , It is the true ground normal obtained by performing finite difference on nearby points. They are shaft and Difference on the axis;
[0113] The final loss function for camera pose tracking is defined as follows:
[0114]
[0115] By minimizing the loss function for camera pose tracking, a continuous, fixed, optimized camera pose curve is output.
[0116] in: It is the loss in rendering the image;
[0117] It is the rendered color after affine exposure adjustment;
[0118] It is the true color of the ground.
[0119] Based on a fixed initial 2D Gaussian scene, this invention constructs a camera pose tracking method that combines surface normal-aware tracking regularization with point-to-point distance and point-to-plane distance to iteratively optimize the camera pose. This method fully utilizes the geometric constraints between the rendered depth and the true depth, making the pose estimation process more robust to noise and local errors. It can output continuous, stable, and accurate camera motion trajectories, providing a reliable center axis reference for subsequent mapping and completion.
[0120] S3, Gaussian expansion and keyframe sampling strategy; the initial two-dimensional Gaussian scene output in step S1. Based on this, a Gaussian expansion strategy is used to supplement the incomplete regions observed due to the camera pose estimation transformation in step S2, and the supplemented 2D Gaussian scene is output. And key frames are sampled as subsequent inputs using a key frame sampling strategy;
[0121] Step S3 is as follows:
[0122] S31: Gaussian expansion strategy: Add all RGBD image sequences continuously acquired by the endoscope in step S1 to the keyframe candidate list, and update the initial two-dimensional Gaussian scene every K frames. In the initial two-dimensional Gaussian scene The text supplements the real tissues that were not observed due to new visual effects generated by changes in camera pose.
[0123] Using the rendered outline This represents the observation area of the current camera pose in each frame, while contour values less than a threshold are used. ( This represents the region (representing the threshold) and the region above the reconstructed tissue surface with the ground truth geometry to extend the initial two-dimensional Gaussian scene. The new 2D Gaussian scene is initialized using a keyframe candidate list to obtain the supplemented 2D Gaussian scene. ;
[0124] S32: Keyframe sampling strategy: Sampling from the keyframe candidate list in step S31 For each keyframe, a pose consistency strategy that simultaneously considers translation and rotation errors is employed. The sampling probability of each candidate frame is defined as follows:
[0125]
[0126] in: It is the distance between locations;
[0127] It is the magnitude error of the rotation quaternion;
[0128] It is a time interval relative to the current frame;
[0129] It is a scaling constant set to 0.2;
[0130] The sampling probability of each candidate frame is normalized so that ,in It is the predefined probability of the current frame. All candidate frames are sorted and sampled according to the cumulative probability distribution function.
[0131] This invention employs a Gaussian expansion strategy to progressively incorporate newly observed real tissue regions, resulting from changes in camera pose, into a 2D Gaussian scene model. This effectively compensates for areas not covered during the initial modeling process, allowing the scene representation to be continuously enriched and improved as observation progresses. Simultaneously, a keyframe sampling strategy selects representative keyframes from candidate frames as input for subsequent optimization, avoiding the computational burden of redundant data and improving the overall system's efficiency and stability.
[0132] S4, Graph Mapping and Hole-Aware Completion Module; in the completed 2D Gaussian scene Based on this, the supplemented two-dimensional Gaussian scene is further iteratively updated through graph mapping. Obtain the updated 2D Gaussian scene Furthermore, topological and continuity constraints are introduced for hole detection. When holes appear in the mapping, an adaptive completion mechanism is triggered, outputting a two-dimensional Gaussian scene with optimized mapping. To achieve continuous and consistent scene mapping;
[0133] Step S4 is as follows:
[0134] S41: Mapping: After outputting the fixed and optimized camera pose curve in step S2, map the 2D Gaussian scene supplemented in step S31. The updated 2D Gaussian scene is obtained by iteratively updating the scene using the gradient descent method. Define depth distortion loss to minimize the distance between ray-sheet intersections:
[0135]
[0136] in: , It is the first The weights and depths of the intersections;
[0137] , It is the first The weights and depths of the intersections;
[0138] We also define a normal consistency loss function. To constrain the surface normals of a two-dimensional Gaussian:
[0139]
[0140] in: It is the index along the ray and the intersection plate;
[0141] It is the normal of the image relative to the camera;
[0142] S42: Hole-aware completion: The updated 2D Gaussian scene is generated after the mapping output in step S41. Subsequently, a cylindrical topology constraint framework based on the Frenet frame is introduced as a hole detection and triggering mechanism; the fixed and optimized camera pose curve obtained in step S2 is regarded as a spatial curve. and at each sampling location on the trajectory Constructing a dynamic Frenet frame coordinate system:
[0143] Where: tangent vector Pointing in the direction of camera movement;
[0144] Principal Normal Vector Indicates the direction of the trajectory's curvature;
[0145] binormal vector Perpendicular to the oscillating plane;
[0146] For the updated 2D Gaussian scene Two-dimensional Gaussian points First, in the space curve Find the projection point that has the shortest Euclidean distance from it. :
[0147]
[0148] Establish a corresponding Frenet coordinate system at this projection point, and convert the Gaussian point in the world coordinate system... Projecting onto the current Frenet frame coordinate system yields a local cylindrical coordinate representation:
[0149]
[0150] Where: radial distance is defined as ;
[0151] In a local sliding window Internal statistical mean radius Construct the topological constraint loss function for the cylinder. :
[0152]
[0153] in: The confidence weighting function assigns a larger weight to the observation density region.
[0154] The topological constraint loss function of the cylinder With a given threshold If a comparison is made, If a hole structure is found in the current mapping region, an adaptive completion mechanism is triggered, and points corresponding to the projection points are selected within the neighborhood of the hole. The shortest set of edge Gaussian points Along the Frenet frame normal vector , The radial direction spanned generates new two-dimensional Gaussian initial parameters to fill the holes:
[0155] like Then, the adaptive completion mechanism is stopped, and only the routine update of the two-dimensional Gaussian parameters in step S41 (mapping mapping) is performed. The final mapping loss function is defined as follows:
[0156]
[0157] Output mapping optimized 2D Gaussian scene
[0158] in: It is the loss in color reconstruction;
[0159] It is the structural similarity index loss. .
[0160] In the mapping and hole-aware completion module, this invention introduces a cylindrical topological constraint based on a Frenet frame as a hole-aware and triggering mechanism, building upon conventional mapping optimization. By treating the camera pose trajectory as a spatial curve and constructing a moving Frenet frame coordinate system, a consistent constraint is applied to the radial distribution of 2D Gaussian points to the central axis. When a hole structure is detected in the mapping region, an adaptive completion mechanism is automatically triggered. Using the Gaussian parameters at the hole edge as a reference, new 2D Gaussian parameters are gradually generated radially, allowing the missing region to progressively close like stitching a piece of fabric. This mechanism significantly improves the integrity and continuity of the reconstruction results in terms of topological structure, effectively avoiding the mapping breakage problem caused by limited field of view in traditional methods.
[0161] S5. Bundle adjustment optimization with smoothing constraints; input the camera pose curve from step S2 and the 2D Gaussian scene after mapping optimization from step S4. Subsequently, a gradient descent-based bundle adjustment method was used to optimize the joint camera pose and the two-dimensional Gaussian scene.
[0162] Step S5 is as follows:
[0163] Keyframes are selected according to the keyframe sampling strategy in step S32. A bundle adjustment optimization is performed once every 200 iterations of camera pose tracking in step S2 to initially update the camera pose and the mapped and optimized 2D Gaussian scene. ;
[0164] Given camera pose And the 2D Gaussian scene after mapping optimization The loss function of the bundle adjustment method is adopted.
[0165] Based on this, a normal smoothing constraint is introduced to address the issues of rough local geometric features and discontinuities in Gaussian scenes caused by the bundle adjustment method. A normal smoothing loss function combining normal consistency and depth gradient anisotropy is designed:
[0166]
[0167] Where: weighted normal consistency loss function , It is a mask function, if adjacent points If it is located at the edge of a wrinkle (the difference in depth gradient or normal exceeds a set threshold), then That is, mutation is allowed, otherwise That is, forced smoothing; while edge-protected deep smoothing loss function
[0168] Finally, a loss function incorporating normal smoothing constraints is used for bundle adjustment optimization:
[0169]
[0170] The output completed the precise correction of camera pose. High-quality 3D scene representation with geometric feature optimization.
[0171] In the bundle adjustment optimization module incorporating smoothing constraints, this invention introduces normal smoothing constraints that combine normal consistency and depth gradient anisotropy while jointly optimizing camera pose and 2D Gaussian scene parameters. This smoothing process continuously enhances and smooths the local geometry. While maintaining the edges of wrinkles and abrupt changes in the real structure, it effectively suppresses noise accumulation and unnecessary geometric oscillations, resulting in a smoother and more natural surface morphology and a more stable and reliable geometric representation of the final 2D Gaussian scene.
[0172] Example
[0173] In this embodiment, to verify the endoscopic scene reconstruction method of fabric-like sewn-up self-healing cavity proposed in this invention, a complete experimental system was constructed to perform functional verification and performance evaluation of this invention, and the functional verification and performance evaluation were performed on the C3VD dataset.
[0174] Figure 1 shows a schematic diagram of the overall structure of the endoscopic scene reconstruction method of the present invention, which uses a fabric-like patching-style self-healing technique for voids. This method constructs a unified planning framework that includes two-dimensional Gaussian scene initialization modeling, camera pose tracking, Gaussian expansion and keyframe sampling, mapping and void-aware completion, and bundle adjustment optimization with fused normal smoothing constraints. The system takes continuously acquired RGBD image sequences from the endoscope as input. First, an initial two-dimensional Gaussian scene representation is constructed through the two-dimensional Gaussian scene initialization modeling module. Then, camera pose tracking is performed on this scene to obtain a continuous and stable camera motion trajectory. Further, through Gaussian expansion and keyframe sampling strategies, newly observed real tissue regions are gradually added to the two-dimensional Gaussian scene. Based on this, a mapping and void-aware completion module is introduced to iteratively optimize and update the two-dimensional Gaussian scene. Finally, the automatic perception and triggering of voids in the mapping is achieved using cylindrical topological constraints based on the Frenet frame, enabling the missing regions to achieve progressive self-healing in a fabric-like patching manner. Finally, by integrating normal smoothing constraints into the beam adjustment optimization, the camera pose and the two-dimensional Gaussian scene are jointly and finely optimized to output a structurally continuous, geometrically smooth, and topologically complete endoscopic three-dimensional scene model.
[0175] To verify the feasibility and effectiveness of the method of this invention, this embodiment uses C3VD as the experimental dataset, selecting 10 endoscopic video sequences for testing. Each video has a resolution of 675*540 pixels and contains an average of 638 frames. This dataset also provides corresponding depth information, which can be used to evaluate the accuracy and completeness of scene reconstruction. In the comparative experiment, the method of this invention is compared with various existing endoscopic mapping and localization methods, including traditional feature point generation methods, implicit neural representation methods, endoscopic depth estimation methods, and the EndoGSLAM method based on three-dimensional Gaussian representation. To ensure the fairness of the comparison, all methods are tested using RGBD input data. The experimental results are shown in Figures 3 and 4. Figure 3 shows the endoscopic scene rendering result image output by the method of this invention, which shows that the reconstructed scene has high quality in terms of color consistency, texture continuity, and overall geometric structure.
[0176] Figure 4 shows a comparison of the 3D reconstruction results of different methods on the same endoscopic sequence. As can be seen from the comparison of the 3D reconstruction models of the intestinal scene in Figure 4(a), traditional methods such as Co-SLAM, ESLAM, ENERF-SLAM, and iMap all exhibit varying degrees of geometric structure loss or local voids during the reconstruction of the intestinal surface, especially in curved and sparsely textured regions, resulting in discontinuous or fragmented reconstructed surfaces. In contrast, the method of our invention (Ours), by introducing cylindrical topological constraints based on Frenet frames and a void-aware completion mechanism, can adaptively fill in missing regions during the mapping process, making the overall morphology of the reconstruction result closer to the real intestinal structure. Furthermore, the beam adjustment optimization, which incorporates normal smoothing constraints, maintains a continuous, smooth, and complete geometric surface. Meanwhile, as shown in the corresponding camera trajectory reconstruction results in Figure 4(b), it can be seen that our invention maintains good stability and consistency in camera pose estimation, providing a reliable geometric basis for scene reconstruction. Our invention can automatically detect this region and trigger the adaptive completion mechanism, effectively filling the missing region and ultimately forming a continuous and complete intestinal surface structure.
[0177] The experimental results above demonstrate that the fabric-like stitching-type cavity self-healing mechanism proposed in this invention can effectively identify structural cavities caused by occlusion, changes in viewing angle, or insufficient observation during endoscopic scene reconstruction, and compensate for missing areas along local topological directions, thereby significantly improving the integrity and continuity of the reconstruction results. Furthermore, by incorporating beam adjustment optimization with normal smoothing constraints, the problems of local geometric roughness and Gaussian scene discontinuities are further suppressed, achieving high-quality endoscopic 3D scene reconstruction.
[0178] In summary, this embodiment verifies the feasibility and advancement of the present invention in achieving hole perception and adaptive completion during endoscopic scene reconstruction. Compared with existing technologies, the present invention introduces a unified planning framework that includes 2D Gaussian scene initialization modeling, camera pose tracking, Gaussian expansion and keyframe sampling, mapping and hole perception completion, and bundle adjustment optimization with fused normal smoothing constraints. This significantly improves the completeness, continuity, and geometric consistency of mapping results in complex tubular organ scenes. This method possesses good engineering feasibility and scalability, and is applicable to applications such as endoscopic navigation, medical 3D modeling, and surgical assistance, providing a reliable technical solution for real-time scene reconstruction in complex biological tissue environments.
[0179] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.
Claims
1. A method for reconstructing endoscopic scenes using a fabric-like stitch-mending self-healing technique for creating voids, characterized in that... Includes the following steps: S1. Initial modeling of a two-dimensional Gaussian scene: A set of two-dimensional Gaussian primitives is used to model the RGBD image sequence frames continuously acquired by the endoscope, resulting in the initial two-dimensional Gaussian scene. S2, Camera pose tracking; The initial 2D Gaussian scene output in step S1 Without changing the position of the camera, perform camera pose estimation on the current image sequence frames to obtain the camera pose curve; S3, Gaussian spread and keyframe sampling strategy; the initial two-dimensional Gaussian scene output in step S1. Based on this, a Gaussian expansion strategy is used to supplement the incomplete regions observed due to the camera pose estimation transformation in step S2, and the supplemented 2D Gaussian scene is output. And sample keyframes as subsequent input using a keyframe sampling strategy; S4, mapping and hole-aware completion module; in the completed 2D Gaussian scene Based on this, the supplemented two-dimensional Gaussian scene is further iteratively updated through graph mapping. Obtain the updated 2D Gaussian scene Furthermore, topological and continuity constraints are introduced for hole detection. When holes appear in the mapping, an adaptive completion mechanism is triggered, outputting a two-dimensional Gaussian scene with optimized mapping. Achieve continuous and consistent scene mapping; S5, optimize bundle adjustment by fusing smooth constraints; input the camera pose curve from step S2 and the 2D Gaussian scene after mapping optimization from step S4. Subsequently, a gradient descent-based bundle adjustment method was used to optimize the joint camera pose and the two-dimensional Gaussian scene.
2. The endoscopic scene reconstruction method with fabric-like stitch-mending self-healing cavity as described in claim 1, characterized in that, The initial two-dimensional Gaussian scene obtained in step S1 for: Where: each two-dimensional Gaussian By location Rotation Scaling factor Opacity and RGB colors Definition; Rotation matrix Defined as ， They are two orthogonal tangential vectors. These are the primitive normals and the scaling matrix. Defined as a diagonal matrix, where the last element is zero and the other two elements are zero. 。 3. The endoscopic scene reconstruction method with fabric-like stitch-mending self-healing cavity as described in claim 2, characterized in that, Step S2 is as follows: The camera pose is represented as follows Assuming the camera's initialization velocity for a new pose is constant, given two previous poses... Calculate the speed as and will Initialization is performed, followed by iterative optimization of the camera pose using gradient descent; for geometrically robust tracking, a method with projected point-to-point distance is employed. and distance from point to plane Surface normal sensing tracking regularization method: Let It's a point from the rendering depth. It is a point originating from the ground truth depth; definition for ,in: for value; The truth depth; For rendering depth; define for ,in: ， It is the true ground normal obtained by performing finite difference on nearby points. They are shaft and The difference on the axis; finally, the loss function for camera pose tracking is defined as: By minimizing the loss function of camera pose tracking, a continuous, fixed-optimization camera pose curve is output, where: It is the loss in rendering the image; It is the rendered color after affine exposure adjustment; It is the true color of the ground.
4. The endoscopic scene reconstruction method with fabric-like stitch-mending self-healing cavity as described in claim 2, characterized in that, Step S3 specifically involves: S31: Gaussian expansion strategy: Add all RGBD image sequences continuously acquired by the endoscopic device in step S1 to the keyframe candidate list, and update the initial two-dimensional Gaussian scene every K frames. In the initial two-dimensional Gaussian scene The text supplements the real tissues that were not observed due to new visual effects generated by camera pose changes; it also utilizes rendered contour maps. This represents the observation area of the current camera pose in each frame, and the contour value is less than a threshold. To extend the initial two-dimensional Gaussian scene, the region above the reconstructed tissue surface with the ground truth geometry is also included. The new 2D Gaussian scene is initialized using a keyframe candidate list to obtain the supplemented 2D Gaussian scene. S32: Keyframe sampling strategy: Sample from the keyframe candidate list in step S31 For each keyframe, a pose consistency strategy that simultaneously considers translation and rotation errors is employed. The sampling probability of each candidate frame is defined as follows: ,in: It is the distance between locations; It is the magnitude error of the rotation quaternion; It is a time interval relative to the current frame; It is a scaling constant set to 0.2; the sampling probability of each candidate frame is normalized so that ,in It is the predefined probability of the current frame. All candidate frames are sorted and sampled according to the cumulative probability distribution function.
5. The endoscopic scene reconstruction method with fabric-like stitch-mending self-healing cavity as described in claim 4, characterized in that, Step S4 specifically involves: S41: Mapping: After outputting the fixed and optimized camera pose curve in step S2, the 2D Gaussian scene supplemented in step S31 is mapped... The updated 2D Gaussian scene is obtained by iteratively updating the scene using the gradient descent method. Define the depth distortion loss function. To minimize the distance between ray-sheet intersections: ,in: ， It is the first The weights and depths of the intersections; ， It is the first The weights and depths of the intersection points are defined; and the normal consistency loss function is defined. To constrain the surface normals of a two-dimensional Gaussian: ,in: It is the index along the ray and the intersection plate; It is the normal of the image relative to the camera; S42: Hole-aware completion: the updated 2D Gaussian scene output after mapping in step S41. Subsequently, a cylindrical topology constraint framework based on the Frenet frame is introduced as a hole detection and triggering mechanism; the fixed and optimized camera pose curve obtained in step S2 is regarded as a spatial curve. and at each sampling location on the trajectory Constructing a dynamic Frenet frame coordinate system: , where: tangent vector Pointing in the direction of camera movement; principal normal vector Pointing to the direction of trajectory curvature; binormal vector Perpendicular to the oscillating plane; for the updated 2D Gaussian scene Two-dimensional Gaussian points First, in the space curve Find the projection point that has the shortest Euclidean distance from it. ： And establish the corresponding Frenet coordinate system at this projection point, and convert the Gaussian point in the world coordinate system... Projecting onto the current Frenet frame coordinate system yields a local cylindrical coordinate representation: Where: radial distance is defined as In a local sliding window Internal statistical mean radius Construct the topological constraint loss function for the cylinder. ： ,in: The confidence weighting function assigns a larger weight to the observation density region; the cylindrical topology constraint loss function... With a given threshold If a comparison is made, If a hole structure is found in the current mapping region, an adaptive completion mechanism is triggered, and points corresponding to the projection points are selected within the neighborhood of the hole. The shortest set of edge Gaussian points Along the Frenet frame normal vector , The radial direction spanned generates new two-dimensional Gaussian initial parameters to fill the holes: ,like Then, the adaptive completion mechanism is stopped, and only the routine update of the two-dimensional Gaussian parameters in step S41 (mapping mapping) is performed. The final mapping loss function is defined as follows: Output the 2D Gaussian scene with optimized mapping. ,in: It is the loss in color reconstruction; It is the structural similarity index loss. 。 6. The endoscopic scene reconstruction method with fabric-like stitch-mending self-healing cavity as described in claim 5, characterized in that, Step S5 specifically involves selecting keyframes based on the keyframe sampling strategy of step S32, and performing bundle adjustment optimization once every 200 camera pose tracking iterations in step S2 to initially update the camera pose and the mapped and optimized 2D Gaussian scene. ； Given camera pose And the 2D Gaussian scene after mapping optimization The loss function of the bundle adjustment method is adopted. Based on this, a normal smoothing constraint is introduced to address the issues of rough local geometric features and discontinuities in Gaussian scenes caused by the bundle adjustment method. A normal smoothing loss function combining normal consistency and depth gradient anisotropy is designed: Where: weighted normal consistency loss function ， It is a mask function, if adjacent points If it is located at the edge of a fold, then That is, allow mutations, otherwise That is, forced smoothing; while edge-protected deep smoothing loss function Finally, a loss function incorporating normal smoothing constraints for bundle adjustment optimization is adopted: The output completed the precise correction of the camera pose. High-quality 3D scene representation with geometric feature optimization.