Endoscope monocular dense reconstruction and real-time positioning method based on deformation modeling

Abstract

The invention discloses an endoscope monocular dense reconstruction and real-time positioning method based on deformation modeling. The method comprises the following steps: constructing a hybrid map model combining a sparse key point graph and three-dimensional Gaussian point representation; performing initial three-dimensional structure reconstruction and non-rigid camera attitude estimation through the sparse point map; fine pose adjustment and high-quality reconstruction are carried out on the Gaussian map in combination with micro-rendering optimization; efficient conversion from sparse geometry to dense geometry is realized through a sparse prior guided depth alignment mechanism, and tracking stability and reconstruction density are both considered. According to the method, three-dimensional mapping and tracking of the deformation scene can be realized in real time under the monocular camera, the method is suitable for an endoscopic surgery scene with complex tissue deformation, and the visualization quality and the navigation precision in the surgery are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, medical image processing and robot navigation technology, and in particular to an endoscope monocular dense reconstruction and real-time positioning method based on deformation modeling, which is suitable for visual enhancement and three-dimensional navigation in soft tissue surgery. BACKGROUND

[0002] With the wide application of minimally invasive surgery, endoscopes have become an indispensable visual tool in modern medical diagnosis and treatment. Endoscopic technology acquires images inside the body cavity through a long probe and a camera, which is used to assist doctors in diagnosis, surgical positioning and navigation. However, since the environment in which the endoscope is located is mostly soft tissue structure, its inherent properties (such as elastic deformation, occlusion, blood coverage) make the image quality unstable, and the camera movement is complex and has non-rigid characteristics. This poses a great challenge to traditional visual SLAM (Simultaneous Localization and Mapping) systems that rely on rigid assumptions.

[0003] Traditional SLAM systems (such as ORB-SLAM, DSO, LSD-SLAM, etc.) are mainly used in static environments, assuming that the scene remains unchanged during camera motion, and are commonly used for indoor three-dimensional mapping or mobile robot navigation. These systems usually achieve positioning and map construction through feature point extraction, camera motion estimation, re-projection optimization, etc. However, in the endoscopic surgery scene, the tissue surface continuously deforms due to physiological factors such as breathing, heartbeat, tool contact, etc., leading to feature point matching errors, camera pose drift, map distortion, and even system failure.

[0004] In order to solve the above problems, some medical SLAM methods have been proposed in recent years to achieve more robust visual reconstruction in soft tissue environments. For example, MIS-SLAM (Minimally Invasive Surgery SLAM) introduces a local deformation map to model tissue deformation, improving tracking stability in laparoscopic surgery; Def-SLAM and SD-DefSLAM methods use triangular meshes to reconstruct the surface of the surgical scene with monocular images, supporting a certain degree of non-rigid geometric mapping; NR-SLAM integrates deformation into the optimization framework in a physical way by introducing viscoelastic physical modeling, greatly improving non-rigid tracking accuracy. However, these methods either rely on binocular cameras or depth sensors for initialization, or have obvious trade-offs between visualization accuracy and running efficiency, limiting their feasibility in clinical real-time applications.

[0005] Parallel to this, the field of neural rendering has been rapidly developing, with methods such as NeRF (Neural Radiance Fields) and 3D Gaussian Splatting (3DGS) demonstrating unparalleled image reconstruction capabilities, generating high-fidelity, multi-view consistent three-dimensional reconstruction results. Among them, 3DGS models the scene as a set of Gaussian kernels with transparency, scale, and color attributes, and uses differentiable rendering techniques for training and optimization, and has been widely used in various fields. In the field of surgical visual enhancement, some studies have attempted to use 3DGS for endoscopic image reconstruction, such as EndoGSLAM, SurgicalGaussian, and other methods have achieved preliminary results.

[0006] However, it is worth noting that neural reconstruction methods are essentially more concerned with image quality than real-time positioning, and without precise camera pose input, it is difficult to achieve stable and real-time application in surgical navigation. In addition, most of its solutions still rely on external depth estimators for geometric initialization or registration, increasing system complexity and error sources.

[0007] Therefore, the existing technology has the following key bottlenecks:

[0008] 1. Lack of a unified system with robust tracking ability and dense reconstruction accuracy in dynamic non-rigid scenes;

[0009] 2. Dependence on depth or binocular initialization, limiting monocular endoscopic deployment;

[0010] 3. Existing 3DGS reconstruction framework fails to effectively integrate geometric constraints in SLAM, lacking scale consistency and physical interpretation ability;

[0011] 4. Traditional SLAM methods fail to handle local non-rigid disturbances caused by tissue deformation, leading to reconstruction drift or misjudgment.

[0012] To address the above challenges, the present application proposes a hybrid map modeling method combining sparse keypoint maps and Gaussian dense representations, which unifies the two types of geometric representations through a sparse-dense depth alignment module, optimizes camera poses with physical constraints, and optimizes reconstruction quality using differentiable rendering, thereby achieving accurate mapping and stable positioning of tissue dynamics in real scenes under monocular conditions. SUMMARY

[0013] To overcome the problems of insufficient dense reconstruction capability, poor non-rigid tracking accuracy, and strong dependence on depth in the prior art under the dynamic scene of an endoscope, the present application proposes an endoscopic monocular dense reconstruction and real-time positioning method based on a deformation-aware monocular endoscopic SLAM system D2SLAM (Deformation-aware Dense SLAM), aiming to achieve robust camera tracking under monocular endoscopic conditions, high-fidelity and scale-consistent dense reconstruction, and lightweight deployment without depth map or binocular initialization.

[0014] To this end, the technical solution of the present application is as follows:

[0015] An endoscopic monocular dense reconstruction and real-time positioning method based on deformation modeling, characterized in that it comprises the following steps:

[0016] S1. Detect sparse key points of an input endoscopic image sequence by a Shi-Tomasi operator, construct a dynamic deformable graph, and form a sparse key point graph; simultaneously, perform triangulation based on initial matching points to generate an initial three-dimensional sparse point cloud;

[0017] S2. Obtain an initial depth map of the current frame using a depth estimation network, project the sparse key points to the image plane to obtain geometric depth, realize sparse-dense depth alignment through polynomial fitting, and generate a scale-consistent dense depth map;

[0018] S3. Initialize a three-dimensional Gaussian point map according to the aligned dense depth map and RGB image, wherein each Gaussian point contains position, color, scale, and opacity parameters;

[0019] S4. Jointly optimize the re-projection error and the DDG deformation energy on the sparse key point graph to solve the initial camera pose;

[0020] S5. Take the coarse estimated pose as the initial value, generate a synthetic image and a depth map through differentiable rendering of the Gaussian map, optimize the difference between rendering and observation, and obtain a high-precision camera pose;

[0021] S6. Dynamically expand the Gaussian map according to the current pose and the visible area, and jointly optimize the geometric properties and rendering quality of the map by fusing the newly observed depth and color information;

[0022] After the initialization steps S1-S3 are completed, S4-S6 are executed in a loop to realize real-time dense reconstruction and positioning of the dynamic scene of an endoscope.

[0023] The specific scheme is as follows:

[0024] (1) Proposed a hybrid map modeling method combining sparse key point graph and three-dimensional Gaussian point cloud map. Among them, the sparse point graph constructs a non-rigid tracking structure through DDG (dynamic deformable graph), and the Gaussian map is used to support the differentiable rendering, generate dense reconstruction image and depth map. The two types of maps complement each other and jointly support the tracking and mapping process.

[0025] (2) Design a sparse-dense geometric alignment mechanism. Use the sparse triangulation result in the initial stage as a geometric prior to perform polynomial fitting and scale alignment on the monocular depth estimation result, thereby establishing a reliable dense initial depth map and realizing the transition from sparse geometry to dense reconstruction.

[0026] (3) Adopt a coarse-to-fine two-stage pose estimation strategy. The coarse stage optimizes the viscoelastic deformation energy and reprojection error on the sparse graph; the fine stage uses the differentiable rendered image and depth map on the Gaussian map for backpropagation optimization to further improve the accuracy of camera pose estimation.

[0027] (4) In the map expansion process, a fusion strategy based on visibility mask and remapped depth map is used to add new points in the Gaussian graph and jointly optimize color, depth, visibility and other attribute parameters to ensure the consistency and visual quality of the reconstruction result.

[0028] (5) Through the joint training of SSIM perceptual loss and L1 color / depth consistency loss, the structure and pixel details of the reconstructed image are optimized, and higher visual fidelity is obtained.

[0029] The system has been verified on multiple synthetic and real endoscope datasets, including C3VD and EndoMapper, and has shown better tracking robustness and reconstruction accuracy than existing methods. At the same time, the system can run at a speed of 3-6 fps on an RTX4090 platform, and has potential for clinical practical application.

[0030] Compared with the prior art, the beneficial effects of the present application are as follows:

[0031] 1) By combining sparse-dense hybrid map and differentiable rendering, the tracking accuracy and reconstruction quality are considered; the sparse geometry is used to scale the dense depth, which is free from the dependence on the real depth map; the viscoelastic modeling and Gaussian rendering are introduced, making the system adapt to dynamic non-rigid tissue scenes;

[0032] 2) Achieve industry-leading visual quality and stability in real data such as EndoMapper, and promote the development of endoscopic surgery navigation system. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is the flowchart of D2SLAM.

[0034] Figure 2 is a contrast result map on real clinical endoscope data. DETAILED DESCRIPTION

[0035] The application will be described in detail below with specific examples. The following content is helpful for those skilled in the art to understand the application more clearly, but should not limit the protection scope of the application. It should be understood that various modifications and replacements can be made without departing from the spirit of the application.

[0036] As shown in Figure 1 , the system of the application generally includes five main stages of sparse initialization, depth alignment, hybrid map construction, camera pose estimation and map extension.

[0037] (I) Sparse initialization stage

[0038] At the beginning of system startup, sparse feature points of the input image sequence are extracted, corner points are detected using the Shi-Tomasi algorithm, and optical flow tracking is performed using the pyramid Lucas-Kanade method. Matched points between multiple frames are selected, and triangulation is achieved by weighted midpoint method to obtain initial sparse three-dimensional point cloud

[0039] A dynamic deformable graph (DDG) is constructed for these sparse points, and each key point is connected with its spatial neighbors to form an elastic network. This graph introduces viscoelastic displacement modeling in the subsequent tracking stage to capture non-rigid tissue deformation, and defines a deformation energy term:

[0040]

[0041] Where δ i represents the displacement of key point i at time t, d ij represents the Euclidean distance between key points i and j, and L δ is a robust Huber loss function.

[0042] (II) Depth alignment and dense initialization

[0043] For the current frame image, a depth estimation network is used to estimate its relative depth map At the same time, the sparse points in the initialization stage are projected to the current image plane, and their real geometric depth z(u) is extracted, and a polynomial fitting is used for scale alignment:

[0044]

[0045] Where polyfit n represents an n-order polynomial fitter, and S 2DSparse projection points set. Construct initial 3D Gaussian point map using aligned dense depth and input image Each Gaussian point parameter is G i = (μ i , C i , s i , o i ).

[0046] (III) Pose estimation stage

[0047] Camera pose estimation adopts a coarse-to-fine two-stage strategy:

[0048] 1. Coarse estimation: jointly minimize re-projection error and DDG viscoelastic deformation energy on sparse map to estimate initial pose

[0049]

[0050] 2. Fine optimization: perform differentiable rendering on Gaussian map to generate current frame image and depth Optimization objective function is:

[0051]

[0052] where I(u) is the real image, and are rendered image and depth. D s (u) is the aligned depth. V(u) is the pixel visibility mask defined as follows:

[0053]

[0054] (IV) Map expansion and optimization

[0055] After completing camera pose estimation, new key frames are selected according to visible regions, and new Gaussian points are added in significant regions. To improve geometric consistency, a dense depth remapping strategy is adopted to align the newly estimated depth with the current map depth:

[0056]

[0057] Finally, image color consistency, structural similarity, and depth difference are fused as the overall optimization objective to optimize the quality of rendered image and depth reconstruction accuracy:

[0058]

[0059] where λ is the SSIM weight coefficient. D d (u) is the depth map after secondary alignment.

[0060] The system verifies its real-time performance and reconstruction accuracy on multiple synthetic and real endoscopic datasets, and the reconstruction results have good visual consistency and geometric restoration ability, which is suitable for visual navigation and structure modeling in dynamic intraoperative environment.

[0061] On the Colonoscopy 3D Video Dataset (C3VD) dataset, compared with mainstream methods such as SLAM3R, EndoGSLAM, Endo-2DTAM, D2SLAM achieves the highest reconstruction quality. The quantitative indicators are shown in Table 1: D2SLAM achieves a PSNR of 22.84 dB, an SSIM of 0.77, and the lowest LPIPS error (0.26), while having a real-time processing speed of 3.09 fps, which is better than the comprehensive performance of other methods.

[0062] Table 1 C3VD

[0063]

[0064]

[0065] Table 2 EndoMapper

[0066] In the EndoMapper dataset collected in real time, D2SLAM also shows significant advantages. Even in the real tissue deformation environment without depth prior, D2SLAM still achieves a PSNR of 20.11 dB and an SSIM of 0.66, which is significantly better than EndoGSLAM (19.22 dB, SSIM 0.60) and Endo-2DTAM (16.92 dB, SSIM 0.41). In addition, D2SLAM maintains a stable running speed of about 6 fps, demonstrating its ability and potential to process dynamic soft tissues in clinical applications.

[0067] The system also conducts ablation experiments to verify the impact of sparse-dense depth alignment, pose fine optimization, DDG deformation modeling and other modules on the final performance. Among them, removing the depth alignment module will cause the RMSE to rise significantly and the image structure to be blurred; disabling DDG will cause the tracking to fail in strong deformation scenarios. This shows that each sub-module is crucial to the overall performance of the system.

[0068] Figure 2are the contrast results on real clinical endoscope data, which can show the technical advantages of EndoSD-SLAM. Among them, D2SLAM (the method in this paper): the reconstruction quality is clear and sharp, the tissue texture details are rich, the geometric structure is accurate, and the depth information consistency is good, and it maintains stable visual quality in the tissue deformation area. EndoGSLAM contrast: there are obvious blur and ghosting phenomena, the tissue deformation processing capability is limited, which leads to the decline of reconstruction quality, and it lacks a special deformation modeling mechanism. Endo-2DTAM contrast: the reconstruction result appears serious geometric distortion and texture disorder, it is sensitive to depth estimation error, which leads to cumulative pose deviation, and it performs poorly on complex tissue surfaces.

Claims

1. An endoscope monocular dense reconstruction and real-time localization method based on deformation modeling, characterized in that, The method comprises the following steps: S1. Detect sparse key points of an input endoscope image sequence by a Shi-Tomasi operator, construct a dynamic deformable graph, and form a sparse key point graph; simultaneously, perform triangulation based on initial matching points to generate an initial three-dimensional sparse point cloud; S2. Obtain an initial depth map of a current frame by using a depth estimation network, project sparse key points to an image plane to obtain geometric depth, realize sparse-dense depth alignment by polynomial fitting, and generate a scale-consistent dense depth map; S3. Initialize a three-dimensional Gaussian point map according to the aligned dense depth map and an RGB image, wherein each Gaussian point contains position, color, scale, and opacity parameters; S4. Jointly optimize re-projection error and DDG deformation energy on the sparse key point graph to solve an initial camera pose; S5. Take a coarse estimation pose as an initial value, generate a synthesized image and a depth map by differentiable rendering of the Gaussian map, optimize differences between rendering and observation, and obtain a high-precision camera pose; S6. Dynamically expand the Gaussian map according to a current pose and a visible region, and jointly optimize geometric properties and rendering quality of the map by fusing newly observed depth and color information. After the initialization steps S1-S3 are completed, S4-S6 are executed in a loop to realize real-time dense reconstruction and positioning of an endoscope dynamic scene.

2. The endoscopic monocular dense reconstruction and real-time localization method based on morphable modeling according to claim 1, characterized in that, The target function of the construction and optimization of the dynamic deformable graph is: where δ i is the displacement of the i-th point of the subject, d ij denotes the distance of the point pair i, j in the current frame, is the initial distance of the point pair, L δ is the Huber loss function.

3. The endoscopic monocular dense reconstruction and real-time localization method based on morphable modeling according to claim 1, characterized in that, The Gaussian point map adopts a three-dimensional Gaussian kernel G i = {μ i , C i , s i , o i}, wherein μ i is a Gaussian center, C i is an RGB color, s i is a scale, and o i is an opacity, a visual mask of a pixel u is calculated, and the formula is as follows: wherein Gaussian kernel G i 2D projection on the image plane.

4. The endoscopic monocular dense reconstruction and real-time localization method based on morphable modeling of claim 1, wherein, The polynomial fitting realizes sparse-dense depth alignment, and its expression is: In the formula, polyfit n S represents an nth-order polynomial fitter. 2D For a sparse projection point set, z(u i ) is S 2D In, at a certain point u i The corresponding depth, This is the estimated depth corresponding to the pixel where the point is located.

5. The endoscopic monocular dense reconstruction and real-time localization method based on morphable modeling according to claim 1, characterized in that, In the coarse estimation stage S5, sparse re-projection error and DDG deformation energy are minimized: where L δ is a robust loss function used to suppress outliers and improve convergence stability, E i,rep , E i,def are the re-projection error term and the deformation constraint term, respectively. is the 2D tracked feature point position, u i is the projection of the 3D tracked point on the 2D plane.

6. The endoscopic monocular dense reconstruction and real-time localization method based on morphable modeling according to claim 1, characterized in that, The step S5. is based on the differentiable rendered image and depth is performed: where V(u) is a visibility mask for pixel u, I(u) is the input image, D s (u) is the aligned depth map.

7. The endoscopic monocular dense reconstruction and real-time localization method based on morphable modeling according to claim 1, characterized in that, In the dynamic expansion of the Gaussian map S6, a fusion strategy based on a visibility mask and depth remapping is adopted, and the optimization target function is: In the formula, D d (u) is the depth map after the second alignment.

Images