Endoscope monocular dynamic scene reconstruction method based on dynamic Gaussian splashing and motion tracking

Abstract

The invention discloses an endoscope monocular dynamic scene reconstruction method based on dynamic Gaussian splashing and motion tracking, and the method comprises the steps: firstly obtaining a dynamic scene image, obtaining an initial three-dimensional track through introducing auxiliary prior information, and then obtaining three-dimensional Gaussian; based on the auxiliary prior information, generating a group of compact Sim3 motion bases, and obtaining each three-dimensional Gaussian transformation function in the dynamic scene image by using the weighted combination of the Sim3 motion bases, the parameters of the three-dimensional Gaussian transformation function including the transformation of the position and the shape along with the time; and performing three-dimensional Gaussian transformation to each frame of image of the dynamic scene according to the three-dimensional Gaussian transformation function, and performing image rendering on each frame to obtain a rendered image. According to the method, monocular depth and dense displacement field prior are combined, and the problem of monocular scale drift is solved by utilizing depth relative sequence stability. According to the method, real-time rendering is ensured, and meanwhile, the dynamic scene reconstruction precision is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing and three-dimensional reconstruction, and in particular to an endoscope monocular dynamic scene reconstruction method based on dynamic Gaussian splashing and motion tracking. Background Art

[0002] As a core basic task of computer vision, three-dimensional reconstruction deeply integrates geometric reasoning and deep learning technology, aiming to restore the accurate three-dimensional geometric structure and topological relationship of the target scene from multi-view images or videos, and realize reverse modeling from two-dimensional observation to three-dimensional space. This technology has key application value in the fields of digital twins, virtual reality, industrial inspection and medical navigation. In minimally invasive surgery scenarios, the three-dimensional deformation process of dynamic organs and tissues can be reconstructed through endoscopic monocular video, which can provide doctors with accurate spatial posture guidance and understanding, and thus provide accurate spatial information support for surgical navigation, robot-assisted operations, etc. This type of technology has irreplaceable clinical value in scenarios such as minimally invasive surgery planning and operational risk warning by accurately restoring the spatial topological relationship and dynamic deformation process of organs and tissues.

[0003] Endoscopic surgery involves a significant amount of non-rigid tissue deformation, placing high demands on accurate modeling. For such methods, ensuring tissue reconstruction accuracy in dynamic scenarios is a key issue. Furthermore, due to the limited viewing angle in these scenarios, many methods can only reconstruct dynamic surgical scenes from a fixed perspective, resulting in poor accuracy. Implementing an accurate dynamic 3D reconstruction algorithm is crucial.

[0004] Currently, dynamic scene reconstruction based on monocular endoscopes still has the following key issues:

[0005] (1) Limited scene perspective: In the 3D reconstruction task of monocular video or fixed camera perspective, the limited scene perspective is a key challenge. Traditional 3D reconstruction methods often rely on multi-view consistency to infer depth information and 3D geometric structure. However, when the camera perspective is limited, many key geometric features cannot be fully observed, resulting in a decrease in the accuracy of depth estimation and structure reconstruction. Traditional reconstruction methods need to rely on structure from motion (SfM) technology to recover the initial geometric structure. However, the core principle of the SfM method is to estimate the position of 3D points through multi-view consistency. Therefore, when the input is only a monocular video, it is prone to initialization failure. Especially in scenes with less texture and large dynamic changes, SfM has difficulty in inferring stable 3D structure from limited perspective information, resulting in a decrease in the quality of subsequent reconstruction. Due to the instability of the geometric information in the initialization stage, the 3D model optimized based on this geometry is also susceptible to the accumulation of errors, which ultimately leads to inaccurate or even collapsed reconstruction results.

[0006] (2) Insufficient modeling accuracy of dynamic tissue deformation: The non-rigid motion of soft tissue is highly complex and uncertain, and is usually accompanied by large deformation and topological changes. However, many existing methods rely mainly on the depth prior information of a single frame during the modeling process, and fail to fully utilize the dynamic feature information of the displacement field between frames. This method may be effective in short-term prediction, but because it ignores the long-range correlation in the time series, it is prone to drift, distortion, and even structural errors when predicting long-range motion trajectories. In the dynamic reconstruction of the human body or biological tissue, the deformation of a certain frame may be related to the motion state of multiple time steps before. If prediction is based only on single-frame information, it is difficult to capture the continuous deformation pattern, which affects the overall reconstruction accuracy.

[0007] To address the above issues, some methods use Neural Radiance Fields (NeRF) to optimize the process and improve the quality of reconstruction. Although these methods have achieved significant improvements in reconstruction accuracy, they face the problem of slow training and rendering speed, and their implicit representation limits the applicability of downstream tasks. As the latest major breakthrough in the field of computer graphics, 3D Gaussian Splatting (3D-GS) combines the flexibility of implicit neural networks with the structured representation of explicit methods. Its highly parallel rendering pipeline achieves a double improvement in speed and quality. Recent studies have overcome the limitations of methods based on neural radiance fields (NeRF) by using 3D-GS. It constructs a 3D Gaussian representation in a canonical space and fuses the deformation field to model deformable surgical scenes. The initialization of the 3D Gaussian is crucial to training efficiency and quality: EndoGS uses SfM for initialization, but multi-view Figure 1 Consistency dependence often leads to poor initialization, prolonging training time and compromising model accuracy. To address this issue, several studies have introduced deep priors to achieve 3D Gaussian initialization by backprojecting the first endoscopic image frame into 3D space. Among them, EndoGaussian and Endo-4DGS use HexPlane to construct the deformation field, while Deform 3D GS explicitly models 3D Gaussian deformations through linear combinations of Gaussian functions. These methods significantly shorten training cycles, improve rendering speed, and optimize reconstruction quality.

[0008] Currently, the following methods are used to achieve accurate reconstruction of dynamic surgical scenes under monocular endoscopes:

[0009] (1) Introducing a single-frame depth prior: By using the depth information of the first frame to initialize the 3D point cloud, the initialization failure problem that may occur in SfM can be avoided. The advantage is that it can quickly provide initial geometric information and improve training efficiency. The disadvantage is that inconsistent scales between frames may cause depth drift, affecting the stability of long time series.

[0010] (2) Implicit deformation field modeling: The HexPlane structure is used to encode spatiotemporal changes, enabling the deformation field to efficiently learn the deformation patterns of dynamic organizations. The advantage is that continuous time series can be interpolated, improving the accuracy of time series modeling. The disadvantage is that it relies on a large amount of training data and has poor interpretability of deformation.

[0011] (3) Explicit motion modeling: Explicitly modeling the deformation of the tissue through a combination of Gaussian basis functions (e.g., Deform 3D GS) to enhance the interpretability of the motion field. The advantage is that it can provide clear motion trajectory information and improve the stability of deformation optimization. The disadvantage is that it may be difficult to fit complex deformations when the topology changes drastically, and it also requires a high level of accuracy for the initial geometric structure.

[0012] Existing methods suffer from the following major issues: First, they underutilize inter-frame relationships, resulting in inaccurate dynamic reconstruction of non-rigid tissue deformation. Non-rigid motion of soft tissue is highly complex and uncertain, often accompanied by large deformations and topological changes. However, many existing methods rely primarily on prior depth information from a single frame during modeling, failing to fully utilize the dynamic characteristics of the inter-frame displacement field. This approach may be effective for short-term predictions, but because it ignores long-range correlations in the time series, it is prone to drift, distortion, and even structural errors when predicting long-range motion trajectories. In dynamic reconstruction of the human body or biological tissue, the deformation in a single frame may be related to the motion state of multiple previous time steps. Relying solely on single-frame information for prediction makes it difficult to capture continuous deformation patterns, thus compromising overall reconstruction accuracy. Second, there is the issue of depth scale drift. Monocular depth estimation algorithms often face the problem of inter-frame scale inconsistency. Due to the lack of a true absolute scale reference, depth estimates between frames may drift in scale, leading to geometric instability during 3D reconstruction. This drift often causes jitter in the reconstruction, making it difficult to maintain a smooth trajectory and compromising the final visual quality. The traditional L1 depth loss focuses primarily on local pixel-level errors and cannot effectively constrain global scale consistency, making it difficult to eliminate depth scale drift in dynamic scenes. This is especially true in monocular scenes, where the lack of additional depth supervision signals further exacerbates the impact of this problem.

[0013] Based on the advantages and disadvantages of the various models mentioned above, this paper proposes an endoscopic monocular reconstruction method based on dynamic Gaussian splattering and motion tracking. Its innovations include:

[0014] (1) Dual prior fusion mechanism: combining monocular depth and dense displacement field priors to provide good assistance for accurate reconstruction;

[0015] (2) Compact Sim(3) Motion basis representation: constructing a low-dimensional motion basis through K-means clustering and Procrustes alignment to achieve efficient dynamic modeling;

[0016] (3) Depth Order Loss Function: This method uses the relative order stability of depth to solve the monocular scale drift problem. This method significantly improves the accuracy of dynamic scene reconstruction while ensuring real-time rendering. Summary of the Invention

[0017] This paper aims to overcome the shortcomings of existing technologies by proposing an endoscopic monocular reconstruction method based on dynamic Gaussian splatting and motion tracking. This method combines monocular depth with a dense displacement field prior to provide excellent support for accurate reconstruction. It also constructs a low-dimensional motion basis through K-means clustering and Procrustes alignment to achieve efficient dynamic modeling. It also exploits depth relative order stability to address the monocular scale drift problem. This method significantly improves the accuracy of dynamic scene reconstruction while ensuring real-time rendering.

[0018] In order to achieve the above object, the technical solutions specifically adopted by the present invention are as follows:

[0019] A method for reconstructing a monocular dynamic scene in an endoscope based on dynamic Gaussian splashing and motion tracking comprises the following steps:

[0020] Step 1: Obtain a dynamic scene image, obtain the initial three-dimensional trajectory by introducing auxiliary prior information, and then obtain a three-dimensional Gaussian;

[0021] Step 2: Based on the auxiliary prior information, a set of compact Sim(3) motion bases are generated, and each three-dimensional Gaussian transformation function in the dynamic scene image is obtained by weighted combination of the Sim(3) motion bases, wherein the parameters of the three-dimensional Gaussian transformation function include the transformation of position and shape over time;

[0022] Step 3: transform the three-dimensional Gaussian transform to each frame image of the dynamic scene according to the three-dimensional Gaussian transform function, and render each frame to obtain a rendered image;

[0023] Step 4: Use the total loss composed of reconstruction loss and tracking loss to train the endoscope monocular dynamic scene reconstruction model constructed in steps 1 to 3;

[0024] Step 5: Use the trained endoscope monocular dynamic scene reconstruction model to reconstruct the scene.

[0025] Optionally, the three-dimensional Gaussian and auxiliary priors described in step 1 are specifically:

[0026] 1) Unlike traditional methods of representing 3D scenes using meshes, voxels, or point clouds, 3D Gaussian Splatting (GS) uses a large number of 3D Gaussian distributions (directional ellipsoids) with specific properties (center position, shape, opacity, and color) as basic units to describe the scene. The scene is represented as a collection of thousands of 3D Gaussian ellipsoids. When rendering an image, all 3D Gaussian ellipsoids associated with a given pixel are projected onto the 2D image plane, forming a 2D Gaussian distribution. These 2D Gaussian "blobs" are sorted by depth and then blended using alpha blending to produce a high-quality, realistic image.

[0027] 2) Introduce complementary image depth estimation prior and dense displacement field prior, and fuse them to enhance 3D optical flow.

[0028] Depth estimation priors utilize monocular or stereo vision methods to infer scene geometry from input RGB video frames. Deep learning-based monocular depth estimation can learn rich scene features from large amounts of data and provide stable depth predictions in areas with sparse texture or low contrast. The depth prior provides strong structural constraints for the calculation of 3D optical flow, ensuring that motion estimation is more consistent with the actual scene geometry.

[0029] The dense displacement field prior is derived from traditional optical flow calculation methods or learning-driven optical flow networks. Classical optical flow methods estimate the motion trajectory of image pixels in a time series based on the assumption of constant brightness and local smoothness. Modern deep learning methods combine spatiotemporal feature extraction with an end-to-end learning framework to predict more accurate displacement fields in complex scenes. The dense displacement field provides global information about object motion, which can make up for the shortcomings of depth estimation in dynamic areas and enhance the robustness of optical flow estimation in the presence of rapid motion or occlusion. By fusing depth prior with dense displacement field prior, we have constructed a more robust 3D optical flow estimation method, achieving high-precision 3D motion estimation in complex surgical scenarios, effectively improving the accuracy and performance of dynamic 3D reconstruction.

[0030] Optionally, the motion base described in step 2) is specifically:

[0031] In three-dimensional space, the similarity transformation group Sim(3) is used to describe the rigid motion and scale change of objects, which includes three-dimensional rotation, translation and global scale transformation. Compared with the Euclidean transformation group SE(3), Sim(3) introduces an additional scale factor to adapt to the scale changes in the scene. Specifically, the Sim(3) transformation consists of the rotation matrix R∈SO(3), the translation vector and scale factor Composition, its matrix is ​​expressed as follows The rotation matrix R describes the change in the orientation of the rigid body, the translation vector t represents the displacement of the object, and the scale factor s allows the object to be scaled proportionally. The Sim(3) motion basis defines the basic transformation method of this group and has wide applications in dynamic 3D scene reconstruction, visual SLAM, and surgical navigation.

[0032] Optionally, the optimization framework described in step 4) is specifically:

[0033] Two sets of losses are used to supervise the dynamic Gaussian. The first set of losses is the reconstruction loss, which includes the RGB loss and the sequential depth loss to match the pixel-level color and depth order of each frame. The second set of losses is the tracking loss, which is used to supervise the movement of the Gaussian between different frames.

[0034] Optionally, the sequential depth loss in step 4) is specifically:

[0035] Image-based relative depth maps usually contain richer details, but lack inter-frame depth consistency, which is generally manifested as flickering depth maps between adjacent frames. Although video-based monocular depth estimation maintains inter-frame depth consistency, it is computationally expensive and lacks practicality. We observe that despite the inconsistency of the values, the order of depth values ​​of different pixels in different frames is stable, which motivates us to propose an order-based depth loss L order .

[0036] The present invention has the following characteristics and beneficial effects:

[0037] The reconstruction model proposed in this paper uses a monocular depth prior and a dense displacement field prior to generate pixel-level three-dimensional trajectories. A compact and low-dimensional Sim(3) motion basis is designed to represent the scene motion. The motion of each point is represented as a weighted combination of these motion bases. In addition, a novel ordinal depth loss function is developed to address the scale inconsistency problem inherent in the monocular depth prior. At the same time, the reconstruction loss (including texture loss and ordinal depth loss) and tracking loss (including 2D position and depth loss) are combined in the optimization stage to achieve high-quality and high-precision dynamic surgical scene reconstruction in terms of handling complex tissue deformation, achieving leading reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a model architecture diagram of an embodiment of the present invention.

[0039] Figure 2 Schematic diagram of qualitative comparison results of frame extraction in an embodiment of the present invention.

[0040] Figure 3A schematic diagram of qualitative comparison results generated from a new perspective of an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention is described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0042] A method for endoscope monocular dynamic scene reconstruction based on dynamic Gaussian splashing and motion tracking, such as Figure 1 As shown, the following steps are included:

[0043] Step 1: Obtain a dynamic scene image, obtain the initial three-dimensional trajectory by introducing auxiliary prior information, and then obtain a three-dimensional Gaussian.

[0044] It should be noted that a global three-dimensional Gaussian ellipsoid is used in the canonical space to model the dynamic surgical scene. Each three-dimensional Gaussian is defined by parameters g≡(μ, r, s, o, c), where represents the center of the ellipsoid, represents the ellipsoid rotation, Represents the ellipsoid scaling, these parameters will change over time; represents opacity, represents the color, and these parameters remain consistent across all time steps.

[0045] Among them, the auxiliary prior information includes image depth estimation prior and dense displacement field prior.

[0046] Specifically, in this embodiment, the monocular depth prior is introduced As one of the key auxiliary inputs, these depth priors are calculated by feeding endoscopic surgical videos into the monocular depth estimation model Depth AnythingV2. Depth Anything V2 has strong zero-shot generalization capabilities, enabling it to generate relatively stable and finely detailed depth maps for complex endoscopic scenes.

[0047] Introducing dense displacement field prior As another key auxiliary input, these dense displacement field priors are calculated by feeding endoscopic surgical videos into the optical flow tracking model TAPIR. TAPIR can track pixel-by-pixel trajectories with high speed and precision, generating stable and accurate pixel-by-pixel displacement fields for complex endoscopic scenes.

[0048] Furthermore, the monocular depth prior provides a pixel-by-pixel depth estimate for each frame in the surgical video sequence, and the dense displacement field prior provides the pixel-level correspondence between two frames. These two priors are combined to estimate the 3D trajectory of pixels in the scene as follows:

[0049] 1) Promote two-dimensional pixels to three-dimensional space: For each frame t in the video sequence and each pixel (u, v) on the frame, query its corresponding depth prior value D t (u, v), and then through back projection calculation, the two-dimensional pixel coordinates (u, v) are upgraded to the three-dimensional point P t (u,v).

[0050] 2) Connecting 3D points across frames: After the 2D pixels of each frame are elevated to 3D points using the depth prior, the next step is to use the dense displacement field prior To track the movement of these points in the time stream. Specifically, for frame t i A pixel in and its corresponding 3D point By calculating the dense displacement field prior at frame t j Then, we also use the frame t to get the corresponding two-dimensional pixel (u′, v′). j Deep prior Lift the corresponding two-dimensional pixel (u′, v′) to its three-dimensional coordinates In this way, in frame t i and t j Three-dimensional points are established between and The three-dimensional displacement connection forms a segment of the three-dimensional trajectory. By continuously applying this process along the video sequence, we can obtain the three-dimensional trajectory of the pixels changing over time in the dynamic scene.

[0051] Step 2: Based on the auxiliary prior information, a set of compact Sim(3) motion bases are generated, and each three-dimensional Gaussian transformation function in the dynamic scene image is obtained by weighted combination of the Sim(3) motion bases, wherein the parameters of the three-dimensional Gaussian transformation function include the transformation of position and shape over time.

[0052] In order to reconstruct the dynamic 3D surgical scene, N 3D Gaussians {G} are defined in the canonical space and their positions, orientations and scales are changed according to the time step. The 3D Gaussian distribution in any time step t is based on the canonical frame t c The three-dimensional Gaussian in Calculated.

[0053] In this embodiment, the frame with the most visible three-dimensional trajectories is selected as the canonical frame, and the Gaussian center point μ0 is initialized using N three-dimensional trajectories randomly sampled from the frame.

[0054] Modeling the 3D motion trajectory for each Gaussian independently would be computationally expensive and difficult to train. We use the initial noisy 3D trajectory {X(t)} to obtain a set of globally shared learnable basis trajectories The transformation T of each Gaussian point in any time step t is calculated by weighted combination of these basis trajectories c→t .

[0055] First, vectorized velocity analysis is performed on these noisy 3D trajectories {X(t)}, and K-means clustering is used to divide them into B types of trajectory sets. Then, for the trajectory set {X(t)} belonging to the b-th cluster b , we use weighted Procrustes alignment to normalize the frame point set {X(c)} at all times τ=0,…,T b and {X(τ)} b Aligned to get the initialization base conversion Next, we let the weight w corresponding to each Gaussian be (b) Initialize in an exponentially decaying manner with respect to the distance from the Gaussian corresponding cluster center in the canonical frame. At each time step t, the basis coefficient w of each point is used. (b) , weighted combination of this global basis trajectory set is used to calculate the transformation T containing each Gaussian c→t :

[0056]

[0057] Based on the above representation method, we now describe how to use any two frames t and t′ to track and optimize pixel-by-pixel 3D motion trajectories. Due to the characteristics of alpha blending rendering, tracking pixel-by-pixel 3D trajectories must also track a set of 3D Gaussians associated with that pixel. We use the canonical frame to establish a dense pixel-by-pixel displacement field between two given frames. Specifically, for any pixel p at time t and the set of Gaussians H(p) that intersect with that pixel, the Gaussian mean distribution at that time is obtained from the canonical space through basis transformation, and then the corresponding desired 3D world coordinates are obtained through alpha blending rendering:

[0058]

[0059] Then transform the Gaussian set H(p) to time t′ to obtain the expected 3D world coordinates of pixel p at the target time t′:

[0060]

[0061] Finally, the 2D corresponding position and depth of pixel p at t′ can be expressed as:

[0062]

[0063] where K t With E t is the camera intrinsic parameter and camera extrinsic parameter at time t′, Π represents the perspective projection operation, (·) [3] is the third element of a vector.

[0064] Step 3: transform the three-dimensional Gaussian transform to each frame image of the dynamic scene according to the three-dimensional Gaussian transform function, and render each frame to obtain a rendered image.

[0065] It should be noted that the core task of this embodiment is to project the 3D Gaussian onto the 2D image plane of each frame and generate the final rendered image through the rendering pipeline. This process involves geometric transformations, color and opacity calculations, and ultimately synthesizing the pixel colors of a single frame.

[0066] Specifically, first, we need to transform the 3D Gaussian representing the scene geometry and appearance into the 2D image space. A 3D Gaussian consists of its center position and a covariance matrix However, in order to optimize and transform more efficiently, the covariance matrix is ​​usually decomposed into a scaling matrix S and a rotation matrix R, that is,

[0067] For each frame in a dynamic scene, the corresponding camera perspective is defined by the world-to-camera transformation matrix W (view matrix) and the projection matrix P. The process of transforming a 3D Gaussian center point μ to a point μ′ in the camera coordinate system is as follows:

[0068] μ′=Wμ

[0069] Next, in order to project the three-dimensional Gaussian onto the two-dimensional image plane, it is necessary to calculate its covariance matrix Σ in two-dimensional space 2D . Through the properties of affine transformation, the Jacobian matrix J is used for approximate calculation. The Jacobian matrix J describes the local linear approximation of the perspective projection transformation near the point μ′. Finally, the calculation formula of the two-dimensional covariance matrix is:

[0070]

[0071] This two-dimensional covariance matrix describes the shape, size, and orientation of the elliptical Gaussian projected onto the image plane.

[0072] After obtaining the 2D representation of all 3D Gaussians on the current frame's image plane, the next step is to perform alpha blending to synthesize the final image. For any pixel in the image, its final color C is calculated by blending all Gaussians covering that pixel, sorted by depth, from front to back.

[0073] First, sort all N Gaussians from front to back along the line of sight. Then, for a pixel position p, the calculation formula for its final color C(p) is:

[0074]

[0075] Among them, c i is the color of the i-th Gaussian, usually calculated by spherical harmonics according to the viewing direction. i is the final opacity after modulation. It is composed of the opacity α learned by Gaussian itself i and its two-dimensional Gaussian distribution value at pixel p:

[0076]

[0077] Here μ 2D,i is the center point of the i-th Gaussian projection onto the two-dimensional image. It represents the cumulative transmittance, that is, the remaining energy after the light passes through the previous i-1 Gaussians.

[0078] By repeating the above projection and rendering steps for each frame of the dynamic scene, a series of coherent, high-quality rendered images can be generated.

[0079] Step 4: Use the total loss consisting of reconstruction loss and tracking loss to train the endoscope monocular dynamic scene reconstruction model constructed in steps 1 to 3.

[0080] Specifically, in this embodiment, two sets of losses are used to supervise the dynamic Gaussian. The first set of losses includes reconstruction losses, which are used to match the pixel-level color and depth order of each frame:

[0081]

[0082] Among them, the first calculation renders the RGB image The pixel-level absolute error loss between the real RGB image I and the second term calculates the depth order loss of the image and uses the sequential depth loss λ order Adjust the weight of this item.

[0083] Furthermore, the sequential depth loss proposed in this embodiment is:

[0084]

[0085] Among them, sign is the sign function, D t represents the predicted depth value, Represents the rendering depth value. The above formula represents the value of and The depth difference between the two is converted to 1 or -1. Next, it is forced to render the depth map The depth order and predicted depth map D t Keep the order consistent.

[0086] In areas with significant occlusion, motion blur, or weak textures, depth estimation is inherently unstable and may not accurately estimate optical flow. We use the reverse consistency check of optical flow to filter out these areas. This method randomly samples 140,000 (p1, p2) pairs in the valid area and calculates the sequential depth loss.

[0087] The second set of losses is used to supervise the motion of Gaussians between frames. We use the absolute error loss to fit the observed 3D trajectory under the constraint of temporal smoothness, and c 、w (b) And the basis function set To optimize:

[0088]

[0089] Among them, L track-2d Supervise the 2D pixel trajectory at time t′ starting from time t, L track-depth Supervise the depth trajectory at time t′ starting from time t.

[0090] Furthermore, the training strategy used in this embodiment is the Adam strategy, which combines the advantages of the other two optimization algorithms (RMSprop and stochastic gradient descent with momentum). Adam calculates an adaptive learning rate for each parameter, which means that it can dynamically adjust the learning step size of each parameter based on the estimated first-order moment (mean) and second-order moment (uncentered variance) of the parameter's past gradient. This makes Adam generally more efficient and stable during training, able to converge to better results faster, and is particularly outstanding when processing complex models and large data sets.

[0091] Step 5: Use the trained endoscope monocular dynamic scene reconstruction model to reconstruct the scene.

[0092] Specifically, in this example, the proposed Endo-GSMT method was evaluated on two public stereo endoscopy video datasets:

[0093] 1) EndoNeRF dataset: Contains six stereoscopic surgical videos, mainly showing moderate tissue deformation. Two publicly available scenes were used in the experiment.

[0094] 2) StereoMIS dataset: Contains eleven stereoscopic surgical videos, characterized by diverse scenes and complex tissue deformation. Five of these videos were selected for evaluation.

[0095] To verify the performance of this method, we compared it with the state-of-the-art methods in the current field, including:

[0096] 1) NeRF-based methods: including EndoNeRF and LerPlane.

[0097] 2) 3DGS-based methods: EndoGS, EndoGaussian, Endo-4DGS, Deform3DGS.

[0098] This example uses two different evaluation strategies to comprehensively evaluate model performance:

[0099] 1) Frame Extraction Evaluation: Following the previous model comparison method in this field, video frames were randomly divided into training and test sets in a ratio of 7:1. This method mainly evaluates the model's reconstruction ability under known viewpoints.

[0100] 2) Novel View Synthesis Evaluation: Using only the training view for frame extraction evaluation cannot determine whether the model is overfitting to the training view. Therefore, we use the left view (main view) as training data and the right view as test data to ensure a more comprehensive performance evaluation. This allows us to assess whether the model has true 3D reconstruction capabilities.

[0101] The evaluation indicators used in this embodiment are: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS).

[0102] Using two evaluation strategies, frame extraction and novel view synthesis, our method was evaluated against two NeRF (Neural Radiance Field)-based methods and four 3DGS (3D Gaussian Splatter)-based methods (comparisons for novel view synthesis were performed only between 3DGS-based methods). As shown in Table 1, our method outperformed all baseline methods in both frame extraction and novel view synthesis. On the EndoNeRF dataset, our method achieved modest improvements in PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity), and a significant 24% improvement in LPIPS (Learning Perceptual Patch Similarity) over the next-best method. On the more challenging StereoMIS dataset, our method's superiority was even more pronounced: compared to the next-best method, our method achieved 1.9dB and 2.2dB improvements in PSNR, 1% and 7% improvements in SSIM, and 39% and 41% improvements in LPIPS. These results demonstrate that our method is able to better capture non-rigid motion and texture details in dynamic surgical scenes, thereby improving reconstruction fidelity and synthesis quality.

[0103] Table 1. Quantitative comparison results. Our method (Ours) is quantitatively evaluated against two existing NeRF-based methods and four 3DGS-based methods. The best results are shown in bold.

[0104]

[0105] To further demonstrate the effectiveness of our approach, a qualitative visual comparison is provided below. Figure 2 shows the results of frame extraction, and Figure 3 The results of new view synthesis are shown. In order to evaluate the reconstruction quality more clearly, the key areas are highlighted with green boxes. In the comparison of frame extraction ( Figure 2 ), our method accurately recovers regions with complex textures and tissue deformations, avoiding the blurring artifacts typically associated with 3DGS methods. In a more challenging new view synthesis comparison ( Figure 3 ), our method always maintains excellent texture details and geometric structures. These results further confirm the robust performance of our method in enhancing 3D reconstruction quality and dynamic scene representation.

[0106] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for reconstructing monocular dynamic scenes in an endoscope based on dynamic Gaussian splashing and motion tracking, characterized in that: The steps include: Step 1: Obtain a dynamic scene image, obtain the initial three-dimensional trajectory by introducing auxiliary prior information, and then obtain a three-dimensional Gaussian; Step 2: Based on the auxiliary prior information, a set of compact Sim(3) motion bases are generated, and each three-dimensional Gaussian transformation function in the dynamic scene image is obtained by weighted combination of the Sim(3) motion bases, wherein the parameters of the three-dimensional Gaussian transformation function include the transformation of position and shape over time; Step 3: transform the three-dimensional Gaussian transform to each frame image of the dynamic scene according to the three-dimensional Gaussian transform function, and render each frame to obtain a rendered image; Step 4: Use the total loss composed of reconstruction loss and tracking loss to train the endoscope monocular dynamic scene reconstruction model constructed in steps 1 to 3; Step 5: Use the trained endoscope monocular dynamic scene reconstruction model to reconstruct the scene.

2. The method according to claim 1, characterized in that The auxiliary prior information includes image depth estimation prior and dense displacement field prior.

3. The method according to claim 2, characterized in that The image depth estimation prior is to use a monocular vision method to infer the geometric structure information of the scene from the input RGB video frame image; the dense displacement field prior is an optical flow calculation method or a learning-driven optical flow network to estimate the motion trajectory of image pixels in a time series.

4. The method according to claim 1, wherein The method for obtaining the initial three-dimensional trajectory is: For each frame t in the dynamic scene image and each pixel (u, v) on the frame, query the image depth estimation prior D corresponding to each pixel (u, v) t (u, v), and then the two-dimensional pixel (u, v) is upgraded to a three-dimensional point P through back projection calculation t (u,v); Using dense displacement field prior To track the movement of three-dimensional points in the time stream to obtain the initialized three-dimensional trajectory of the pixels changing over time in the dynamic scene 5. The method according to claim 4, characterized in that The method for tracking 3D points is: For a certain frame t i A pixel in and its corresponding 3D point The dense displacement field prior is used to calculate the 3D point in another frame t j The two-dimensional corresponding pixel (u′, v′) in frame t is then j Image depth estimation prior Lift the corresponding 2D pixel (u′, v′) to its 3D point 6. The method according to claim 5, characterized in that By tracking frame t i and t j The three-dimensional point in frame t i and t j Three-dimensional points are established between and The three-dimensional displacement connection forms a segment of the three-dimensional trajectory. By continuously applying this process along the video sequence, we can obtain the three-dimensional trajectory of the pixels changing with time in the dynamic scene.

7. The method according to any one of claims 1 to 6, characterized in that In step 1, the method for obtaining the initialized three-dimensional Gaussian is: selecting a frame with the most visible three-dimensional trajectories as a standard frame, and using N three-dimensional trajectories randomly sampled from the standard frame, using the coordinates of the N sampled three-dimensional trajectories in the frame as the center point of the three-dimensional Gaussian, and initializing other attributes of the three-dimensional Gaussian, and then obtaining the three-dimensional Gaussian based on the center point and the initialized attributes, wherein the other attributes include opacity, shape and color.

8. The method according to claim 7, characterized in that The method for generating the Sim(3) motion basis is: First, the three-dimensional trajectory {X(t)} is vectorized and analyzed for velocity. K-means clustering is used to divide the three-dimensional trajectory into B types of trajectory sets. Then, for the trajectory set {X(t)} belonging to the b-th cluster b , at all times τ=0,…,T, the point set {X(c)} of the canonical frame is aligned using weighted Procrustes b and the point set {X(τ)} at time τ b Align, thus obtaining the Sim(3) motion basis 9. The method according to claim 8, characterized in that In step 2, let the weight w corresponding to each three-dimensional Gaussian be (b) Initialize in an exponentially decaying manner with the distance from the Gaussian corresponding cluster center in the standard frame, and at each time step t, pass the weight w of each point (b) , weighted combination of this global basis trajectory set is used to calculate each three-dimensional Gaussian transformation function T c→t :

10. The method according to claim 9, characterized in that In step 3, the image rendering method is: projecting a three-dimensional Gaussian onto a two-dimensional image plane to form a two-dimensional Gaussian distribution, sorting the two-dimensional Gaussian distribution according to its distance from the two-dimensional image plane and mixing it through α blending rendering to obtain a rendered image.

11. The method according to claim 9, characterized in that The optimization method of the three-dimensional Gaussian transformation function is: Using any two frames t and t′ to track and optimize the pixel-by-pixel motion trajectory, for any pixel p at time t and the Gaussian set H(p) intersecting with the pixel, the Gaussian mean distribution at that moment is obtained from the canonical space through basis transformation, and then the corresponding desired 3D world coordinates are obtained through alpha blending rendering: Then transform the Gaussian set H(p) to time t′ to obtain the expected three-dimensional world coordinates of pixel p at the target time t′: Finally, at the two-dimensional corresponding position of t′ about pixel p and depth It can be expressed as: where K t With E t is the camera intrinsic parameter and camera extrinsic parameter at time t′, Π represents the perspective projection operation, (·) [3] is the third element of a vector; Then, the absolute error loss is used to adjust μ under the time smoothness constraint using the tracking loss. c 、w (b) And the three-dimensional Gaussian transformation function T in the basis function set c→t For optimization, the expression is as follows: L track-2d Supervise the two-dimensional trajectory from time t to time t′, L track-depth Supervise the depth trajectory at time t′ starting from time t.

12. The method according to claim 10, characterized in that In step 3, the reconstruction loss is applied to optimize the image rendering, and the expression is as follows: Among them, the first calculation renders the RGB image The pixel-level absolute error loss between the real RGB image I and the second term is the depth order loss used to calculate the image, λ order is the weight used to adjust the depth order loss.

13. The method according to claim 12, characterized in that The depth order loss expression is as follows: Among them, sign is the sign function, D t represents the predicted depth value, Indicates the rendering depth value, and The rendered depth value for two pixels.

Images