Multi-scale 3d gaussian reconstruction method for endoscopic scene under extreme deformation

Abstract

This invention relates to the field of Gaussian reconstruction technology and discloses a multi-scale 3D Gaussian reconstruction method for endoscopic scenes under extreme deformation. The method decomposes the dynamic deformation of each 3D Gaussian in a normal Gaussian space into global low-frequency motion and local high-frequency motion in the time dimension, and models them using coarse and fine-scale linear basis functions respectively to calculate the various deformation variables of each 3D Gaussian at corresponding times, thus obtaining a deformed Gaussian space. The deformed and non-deformed regions in the deformed Gaussian space are then subjected to radical densification and compaction respectively to obtain an optimized deformed Gaussian space. This optimized space is projected onto a two-dimensional plane to obtain the rendered RGB image and depth map, and RGB and depth map rendering losses are constructed. A total variational loss is constructed based on the differences between adjacent pixels in the rendered RGB image. A total loss function is constructed based on the aforementioned losses, and the 3D Gaussian attributes and linear basis function parameters in the Gaussian space are trained to achieve multi-scale 3D Gaussian reconstruction.

Description

Technical Field

[0001] This invention relates to the field of Gaussian reconstruction technology, and in particular to a multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenarios under extreme deformation. Background Technology

[0002] In modern minimally invasive surgery, the real-time and high-fidelity reconstruction of three-dimensional dynamic soft tissue from endoscopic video is the foundation for realizing cutting-edge clinical applications such as augmented reality (AR) surgical navigation, building virtual surgical training systems, and downstream precision medical surgery.

[0003] Early traditional 3D reconstruction methods, such as Structure from Motion (SFM) and Simultaneous Localization and Mapping (SLAM), were very mature in handling rigid scenes with good texture. However, the environments in which endoscopic images are acquired generally exhibit significant non-rigid deformations caused by the movement of soft organs and instrument manipulation, coupled with weak texture and high reflectivity, posing serious challenges to the stability and accuracy of traditional methods.

[0004] In recent years, with the development of neural rendering, Neural Radiance Field (NeRF) has used implicit neural representations to reconstruct soft tissue surgical scenes, significantly improving the reconstruction quality in these scenes. However, its high training cost and extremely slow rendering speed hinder its application in surgeries requiring real-time feedback. Neural implicit representation methods, represented by NeRF, have achieved revolutionary breakthroughs in the fidelity of viewpoint synthesis by learning continuous functions that map 3D coordinates to color and density. EndoNeRF pioneered the introduction of this paradigm into dynamic endoscopic scenes, employing a dual multilayer perceptron (MLP) network to model the deformation of soft tissue and the canonical representation of the scene, achieving significant reconstruction quality. Building on this, EndoSurf further introduces the Signed Distance Field (SDF) function to enhance geometric constraints and improve surface reconstruction accuracy. Despite these significant effects, a common bottleneck for these methods is their extremely high computational complexity. To improve efficiency, works such as Lerplane and Forplane have adopted the idea of ​​low-rank decomposition, decomposing the four-dimensional spatiotemporal field into multiple low-dimensional feature planes, thereby accelerating the training process. However, even with the acceleration strategy of introducing low-rank decomposition, NeRF-like methods still rely on expensive volumetric rendering and implicit neural network queries, resulting in training times typically lasting hours and rendering speeds that are difficult to achieve the 30 FPS standard required for clinical real-time processing. This inherent bottleneck in computational efficiency of implicit representation has prompted researchers to turn to more efficient explicit representation methods, namely 3D Gaussian sputtering 3DGS.

[0005] As a pioneering real-time rendering technology, 3D Gaussian Splatting (3DGS) offers a new approach to overcoming this bottleneck. Researchers have rapidly extended it to dynamic scenes by combining learnable deformation fields or linear basis functions to model dynamic soft tissues. As an emerging explicit scene representation method, 3DGS directly optimizes 3D Gaussian primitives through differentiable rasterization, achieving unprecedented real-time rendering speed and high-fidelity reconstruction quality. It has quickly become a research hotspot in dynamic scene reconstruction, especially in endoscopic scene reconstruction for surgical needs. Endo-4DGS and subsequently EndoGaussian introduced 3DGS into endoscopic scene reconstruction, following the design philosophy of NeRF, by introducing an implicit MLP deformation field to predict the displacement of the 3D Gaussian. While this hybrid architecture of explicit 3D Gaussian and implicit MLP inherits the powerful nonlinear fitting capabilities of MLP, it does not completely eliminate the computational burden of neural networks, resulting in training and rendering speeds still being limited by the query efficiency of the MLP. To further unleash the real-time potential of 3DGS, subsequent work began exploring neural network-free modeling paths. Deform3DGS innovatively uses learnable linear basis functions to explicitly regress Gaussian motion; this method eliminates the dependence on high-dimensional feature planes and MLPs, compressing single-scene training time from hours to minutes, achieving true real-time rendering. Having established the foundation for real-time performance, recent research has shifted towards addressing the specific challenges of endoscopic scenarios. To address the high storage consumption of explicit representations, LGS proposed strategies based on deformation-aware pruning and Gaussian property pruning; almost simultaneously, EH-SurGS introduced the concept of Gaussian time periods to address the irreversible topological deformation of soft tissues caused by surgical cutting; while Endo-4DGX focuses on solving the problem of drastic changes in intraoperative illumination. Although these methods have shown great potential in various aspects of endoscopic scenarios (such as storage efficiency, irreversible deformation, and illumination changes), they still have limitations in extreme deformation dynamic scenarios. Specifically, even methods like Deform3DGS and EH-SurGS, which boast high-precision reconstruction quality and excellent real-time computational efficiency, struggle to simultaneously account for low-frequency global respiratory motion and high-frequency local instrument traction due to their use of a single-scale basis function. This makes them unable to accurately simulate the deformation of highly non-rigid soft tissues. Furthermore, existing uniform or gradient-based densification strategies lack awareness of areas of intense deformation, easily generating artifacts and redundancy in these regions.

[0006] In summary, most current methods typically rely on high-dimensional spatiotemporal feature fields and small multilayer perceptrons (MLPs) to regress point motion. This approach is not only computationally expensive but also limits training and rendering speed. Therefore, to overcome this limitation, some existing methods employ linear basis functions to model soft tissue deformation, effectively improving training and rendering speed. However, limitations remain in extreme deformation scenarios, such as those encountered in endoscopy. Highly non-rigid soft tissue deformation is high-frequency, localized, and nonlinear. Neither learnable deformation fields nor linear basis functions are sufficient to accurately represent the large displacements required for each 3D Gaussian, leading to geometric registration failure. Furthermore, due to the simplistic density strategy, the gradient of the 3D Gaussian's movement within extreme deformation regions is easily diluted or misled over time, resulting in artifacts and redundancy, leading to high reconstruction errors and poor accuracy. Summary of the Invention

[0007] Therefore, the technical problem to be solved by the present invention is to overcome the problems in the prior art when facing the drastic and complex scene changes caused by surgical instruments acting on soft tissues, such as the difficulty in modeling highly non-rigid soft tissue deformation and the easy generation of artifacts and redundancy in extreme deformation areas.

[0008] To address the aforementioned technical problems, this invention provides a multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenes under extreme deformation, comprising: Initialize the normalized Gaussian space based on the input real RGB image, real depth map and surgical instrument mask; The dynamic deformation of each 3D Gaussian in the normal Gaussian space in the time dimension is decomposed into global low-frequency motion and local high-frequency motion, and modeled by coarse-scale Gaussian function and fine-scale Gaussian function respectively to calculate the deformation of each 3D Gaussian at the corresponding time, including position, rotation, scale and opacity, and obtain the deformed Gaussian space. Based on the deformation mask, the deformed and non-deformed regions in the deformed Gaussian space are obtained, and radical compaction and compaction are performed respectively to obtain the optimized deformed Gaussian space. The optimized deformed Gaussian space is projected onto a two-dimensional plane to obtain the rendered RGB image and depth map. These are then compared with the real RGB image and real depth map to construct the RGB image rendering loss and the depth map rendering loss. The total variational loss is constructed based on the difference between adjacent pixels in the rendered RGB image. Based on RGB image rendering loss, depth map rendering loss and total variational loss, a total loss function is constructed. The function is then trained on various 3D Gaussian functions, coarse-scale Gaussian functions and fine-scale Gaussian functions in the normalized Gaussian space to obtain the target Gaussian space and achieve multi-scale 3D Gaussian reconstruction.

[0009] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: The multi-scale 3D Gaussian reconstruction method for endoscopic scenes under extreme deformation described in this invention selects the Gaussian function, which has good temporal locality and smoothness, as the linear time basis function. The Gaussian function allows the model to activate specific deformation modes within a specific time window, which is highly consistent with the transient nature of soft tissue deformation in endoscopic scenes. By learning parameters... and The model can adaptively adjust the distribution of each basis function on the time axis, thereby enabling continuous modeling of long time-series videos without the need for rigid time segmentation.

[0010] Meanwhile, a more aggressive densification strategy is adopted for deformable regions. First, a gain weight is introduced during the gradient accumulation stage, multiplying the original accumulated gradient by a weight greater than 1 to prevent time averaging from diluting these critical transient gradient signals, thus allowing Gaussians in deformable regions to make a greater gradient contribution. Second, during the densification judgment stage, a lower densification threshold is used for Gaussians in deformable regions. This makes it easier for Gaussians in severely deformed regions to trigger splitting or cloning, thereby quickly supplementing local high-frequency geometric details. For non-deformable regions, the standard densification strategy based on view space location gradient in the original 3DGS is still used. That is, when the accumulated gradient exceeds the default threshold, cloning or splitting operations are performed according to the Gaussian scale. For Gaussians with small gradients, low opacity, or those that have been invisible for a long time, pruning operations are performed to control the number of Gaussians and reduce redundancy. This embodiment, which applies a more aggressive densification strategy to extremely deformable regions, can better describe the complex changes in the deformation of non-rigid soft tissues, effectively suppress the trailing shadows left by rapid tissue displacement, and further ensure the accuracy of 3D reconstruction.

[0011] Meanwhile, because this embodiment employs an aggressive densification strategy to capture rapid deformation, it may lead to excessive 3D Gaussianization in areas with weak texture, manifesting as high-frequency noise or floating objects in the rendered image. This embodiment introduces a total variational loss for the image, which effectively smooths these unstructured high-frequency noises by penalizing the norm of the rendered image gradient, while preserving the main geometric edges of the soft tissue surface. This spatial regularization complements the linear Gaussian function of this embodiment, jointly ensuring the consistency of the reconstruction results in the spatiotemporal dimensions. Attached Figure Description

[0012] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart of the steps of the multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenarios under extreme deformation according to the present invention; Figure 2 This is a flowchart illustrating the principle of a multi-scale 3D Gaussian reconstruction method for endoscopic scenarios under extreme deformation. Figure 3 This is a comparison chart of the experimental results of PSNR and rendering speed FPS between the present invention and the comparison method; Figure 4 This is a comparison chart of the experimental results of PSNR and training time (Time) between the present invention and the comparison method; Figure 5 This is a comparison chart of the rendering results of the present invention and the comparison method on the EndoNeRF and StereoMIS datasets; Figure 6 These are the results of the rendering process of the present invention and the comparison method on the EndoNeRF and StereoMIS datasets over time; Figure 7 The PSNR index follows The curve variation graph; Figure 8 The PSNR index follows The curve variation graph; Figure 9 This is a comparison chart of PSNR metrics for different frame intervals on the StereoMIS dataset in this embodiment; Figure 10 This is a comparison chart of the SSIM metrics for different frame intervals on the StereoMIS dataset in this embodiment; Figure 11 This is a comparison chart of the LPIPS metrics for different frame intervals on the StereoMIS dataset in this embodiment; Figure 12 These are comparison images of rendered images from the StereoMIS dataset. Figure 12 (a) is the rendered image of group (a) in Table 11 on the StereoMIS dataset. Figure 12 (h) is the rendered image of the experiment in group (h) in Table 11 on the StereoMIS dataset. Detailed Implementation

[0013] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0014] Specifically, 3D Gaussian sputtering is an explicit representation of a 3D scene that uses a set of anisotropic 3D Gaussian primitives. Composed of Gaussian To model a static scene, each 3D Gaussian element... From the mean Covariance matrix Opacity Spherical Harmonic Function Composition; among which, This indicates the center position of Gauss in world coordinates. This represents the shape of Gauss's ellipsoid. Represents color as seen from a specific viewpoint. Decompose the covariance matrix. ,in Represented as spatial scale. This is represented as a rotation. All properties of each Gaussian are optimized through backpropagation, and a single Gaussian function is described as follows: 3D Gaussian sputtering first performs rasterization during the rendering process, projecting the 3D Gaussian sputtering onto the 2D image plane. This process involves setting the covariance matrix... The projection is: ; in, For view transformation matrix, This is the Jacobian matrix for the affine approximation in the projection transformation. Next, we render the colors. and depth Given a set of 3D Gaussians, first sort them by their depth, then sequentially sputter the projections of each 3D Gaussian onto a 2D image in pixel space. Rendering using compositing, represented as: ; pixel final color and depth , represented as: , ;in, For the first A 3D Gaussian Center Axial depth.

[0015] Reference Figure 1 The flowchart shown is a step-by-step diagram of the En-MGS (Endoscope Multi-scale Gaussian Splatting) method for multi-scale three-dimensional Gaussian reconstruction of endoscopic scenes under extreme deformation, as described in S101 to S106.

[0016] S101: Initialize the Gaussian space based on the input real RGB image, real depth map and surgical instrument mask.

[0017] Specifically, the Gaussian point cloud needs to be initialized before training begins, based on the input real RGB image. True depth map and surgical instrument mask Dense Gaussian point clouds are generated as a normalized Gaussian space. This point cloud is used to normalize 3D Gaussian properties in Gaussian space.

[0018] S102: Decompose the dynamic deformation of each 3D Gaussian in the normalized Gaussian space in the time dimension into global low-frequency motion and local high-frequency motion, and model them using coarse-scale Gaussian functions and fine-scale Gaussian functions respectively to calculate the deformation of each 3D Gaussian at the corresponding time point in terms of position, rotation, scale, and opacity, and obtain the deformed Gaussian space, including: S102-1: Position in each 3D Gaussian. Rotation angle ,size With Opacity Multiple learnable Gaussian functions are preset to simulate the dynamic deformation of each 3D Gaussian in the time dimension, and the dynamic deformation is decomposed into global low-frequency motion and local high-frequency motion. S102-2: At each time step Multiple learnable Gaussian functions are explicitly decomposed into coarse-scale Gaussian functions. With fine-scale Gaussian function , respectively represented as: ; ; S102-3: Global low-frequency motion is simulated using a coarse-scale Gaussian function, and local high-frequency motion is simulated using a fine-scale Gaussian function. The deformations of the position, rotation angle, size, and opacity of each 3D Gaussian are obtained, expressed as follows: ; S102-4: Based on the deformed 3D Gaussians, a deformed Gaussian space is formed; in, This represents the natural exponential function. and Let represent the learnable center and variance of the Gaussian function, respectively; This represents the coarse-scale scaling factor, designed to capture large-scale, low-frequency global displacements, giving it a wider temporal receptive field. This represents a fine-scale reduction factor, which focuses on capturing local, high-frequency fine residuals, enabling them to respond to rapidly changing transient deformations. , express Time parameters Based on its attribute set The shape variable, , , and Representing parameters respectively The learnable weight set, center set, and variance set of multiple Gaussian functions; This represents the total number of coarse-scale Gaussian functions. , , and Representing parameters respectively The Learnable weights, center, and variance of a coarse-scale Gaussian function; This represents the total number of fine-scale Gaussian functions. , , and Representing parameters respectively The Learnable weights, center, and variance of a fine-scale Gaussian function.

[0019] Specifically, the deformation of non-rigid soft tissues in endoscopic scenarios under extreme deformation is complex, encompassing both smooth global motion caused by physiological activities such as organ peristalsis and localized, intense, high-frequency deformation caused by surgical instrument traction and cutting. Existing methods based on linear basis functions typically apply uniform scale constraints to all basis functions, often leading to oversmoothing when fitting high-frequency, intense motion or unnecessary jitter when fitting low-frequency motion. Furthermore, some methods based on high-dimensional spatiotemporal feature fields and small multilayer perceptrons (MLPs) not only fail to model highly non-rigid deformations but also incur significant computational overhead during training, failing to meet the real-time processing requirements of surgery; their key challenge lies in multi-scale modeling of Gaussian deformations.

[0020] Therefore, this embodiment takes into account that endoscopic soft tissue deformation includes periodic low-frequency movements caused by respiration and sudden high-frequency movements caused by instrument manipulation, and uses a set of multiple learnable Gaussian functions. and weight To simulate the deformation of Gaussians in a scene, and to apply this set of Gaussian functions... Explicitly decomposed into coarse-scale components and fine-scale components .

[0021] To achieve multi-scale operation, this embodiment sets 20 basis functions, logically divided into two groups: the first 10 basis functions are used to construct the coarse-scale layer, and the last 10 basis functions are used to construct the fine-scale layer. Subsequently, to describe each Gaussian at time... Location Rotation angle ,size With Opacity This embodiment introduces an additional set of learnable parameters. , , and , by location For example, the deformation variable A set of parameters can be used Represented as: Finally, the deformed position can be obtained as follows: .

[0022] In addition, rotation angle ,size With Opacity Deformation amount , and Similarly, we can obtain the following.

[0023] The core advantage of using the Gaussian function as the linear time basis function in this embodiment lies in its excellent temporal locality and smoothness. Unlike global polynomial basis functions or frequency-domain Fourier basis functions, the Gaussian function allows the model to activate specific deformation modes within a specific time window, which is highly compatible with the transient nature of soft tissue deformation in endoscopic scenarios (such as the instantaneous fluctuations of surgical instruments). Through learning parameters... and The model can adaptively adjust the distribution of each basis function on the time axis, thus enabling continuous modeling of long-term video without the need for rigid time segmentation. Simultaneously, this multi-scale design effectively decouples complex surgical scene dynamics, better describing the complex changes in the deformation of non-rigid soft tissues.

[0024] S103: Based on the deformation mask, obtain the deformed and non-deformed regions in the deformed Gaussian space, and perform radical compaction and compaction respectively to obtain the optimized deformed Gaussian space, including: S103-1: Obtain the average deformation of each 3D Gaussian in the deformed Gaussian space compared to its normal Gaussian space, and compare it with a first preset threshold: If the average deformation of the 3D Gaussian is greater than the first preset threshold, the 3D Gaussian is assigned to the first candidate dynamic region; otherwise, the 3D Gaussian is assigned to the first candidate static region. S103-2: Obtain the difference in local rendering loss between each 3D Gaussian in deformed Gaussian space and normalized Gaussian space, and compare it with the second preset threshold: If the difference is less than the second preset threshold, the 3D Gaussian is assigned to the second candidate static region; otherwise, the 3D Gaussian is assigned to the second candidate dynamic region. S103-3: Obtain the intersection of the first candidate dynamic region and the second candidate dynamic region as the deformation region; S103-4: Obtain the intersection of the first candidate static region and the second candidate static region as the non-deformable region; S103-5: For each 3D Gaussian in the non-deformable region, when its cumulative position gradient exceeds the basic gradient threshold, perform a densification operation on it to obtain a dense non-deformable region; the densification operation is cloning or splitting. S103-6: Reduce the basic gradient threshold by a preset ratio to obtain the low-order gradient threshold; S103-7: For each 3D Gaussian in the deformation region, a gain weight is introduced when accumulating the position gradient. When the gain accumulation of the position gradient exceeds the low-order gradient threshold, a densification operation is performed on it to obtain a dense deformation region. S103-8: Based on the dense deformable region and the dense non-deformable region, the optimized deformable Gaussian space is obtained.

[0025] Specifically, when dividing the deformed and non-deformed regions, regions where the two criteria conflict are further divided into four smaller sub-regions. The above division steps are repeated to obtain a more refined dynamic mask. After obtaining the region-level dynamic mask, the 3D Gaussian center at the current moment is projected onto the image plane, and each Gaussian is assigned a dynamic label based on whether its projection position falls within the dynamic region. This dynamic label is then used to guide the gradient weighted accumulation and differential densification processing of the deformed region.

[0026] Specifically, in endoscopic scenarios with extreme deformation, traditional 3DGS densifies based on the average gradient of the view space location. However, in this region, the moving gradient of the 3D Gaussian is easily diluted over time, leading to misleading results and causing blurring or artifacts in the dynamic region reconstruction. To address this, this embodiment proposes an adaptive Gaussian perception strategy that employs different processing methods for deformed and non-deformed regions.

[0027] First, this embodiment uses gradient statistical augmentation to... The covered Gaussian points use gain weights when accumulating gradients. To prevent its gradient information from being diluted during time averaging, the gain accumulates the position gradient, expressed as: ;in, For the accumulated position gradient, For the current iteration The position gradient magnitude of a 3D Gaussian; For indicator functions, if In the deformation zone If the value is 1, the indicator function value is 1; otherwise, it is 0.

[0028] In traditional 3DGS densification strategies, the accumulated view space location gradient Typically, the gradient is averaged over a number of iterations. However, in regions of extreme deformation, a 3D Gaussian may only produce a large gradient within a very short time window (e.g., the instant a machine rapidly passes by), remaining stationary for the rest of the time. Simple time averaging would significantly dilute these critical transient gradient signals, causing the 3D Gaussian that should trigger splitting to fail to split in time. The gain weights used... Essentially, it is an importance sampling factor that forces the model to assign higher weights to 3D Gaussians in active deformation states when calculating the average gradient, thereby ensuring that high-frequency deformation details can trigger the necessary Gaussian splitting or cloning operations.

[0029] Secondly, this embodiment employs a differentiated splitting threshold, maintaining the default gradient threshold for non-deformable regions during densification operations. However, for deformable regions, the splitting threshold can be reduced; threshold , represented as: ;in, This is the scaling factor. This means that areas with severe deformation are more likely to trigger 3D Gaussian splitting and cloning, thus quickly filling geometric holes.

[0030] For deformable regions, this embodiment employs a more aggressive densification strategy. First, a gain weight is introduced during the gradient accumulation phase, multiplying the original accumulated gradient by a weight greater than 1. This prevents time averaging from diluting these critical transient gradient signals, allowing Gaussians in deformable regions to contribute more gradient. Second, during the densification judgment phase, a lower densification threshold is used for Gaussians in deformable regions. This makes it easier for Gaussians in severely deformed regions to trigger splitting or cloning, thereby quickly supplementing local high-frequency geometric details. For non-deformable regions, the standard densification strategy based on view-space location gradients in the original 3DGS is still used. That is, when the accumulated gradient exceeds the default threshold, cloning or splitting operations are performed according to the Gaussian scale. For Gaussians with small gradients, low opacity, or those that are not visible for a long time, pruning operations are performed to control the number of Gaussians and reduce redundancy. This more aggressive densification strategy applied to extremely deformable regions effectively suppresses the trailing artifacts left by rapid tissue displacement, further ensuring the accuracy of 3D reconstruction.

[0031] S104: Project the optimized deformed Gaussian space onto a 2D plane to obtain the rendered RGB image and depth map. Compare these with the real RGB image and real depth map to construct the RGB image rendering loss and depth map rendering loss, including: RGB image rendering loss , represented as: ; Depth map rendering loss , represented as: ; in, Indicates surgical instrument mask, This indicates element-wise multiplication. and These represent the rendered RGB image and the actual RGB image, respectively. and These represent the rendered depth map and the actual depth map, respectively.

[0032] S105: Construct the total variational loss based on the differences between adjacent pixels in the rendered RGB image. , represented as: ; in, This represents the rendered RGB image. Indicates batch size, Indicates the number of channels. and These represent the height and width of the actual RGB image, respectively. Indicates batch as The number of channels is Position in the rendered RGB image Image pixel values ​​at that location.

[0033] Because this embodiment employs an aggressive densification strategy to capture rapid deformation, it may result in excessive 3D Gaussianization in areas with weak texture, manifesting as high-frequency noise or floating objects in the rendered image. This embodiment introduces a total variational loss for the image. By penalizing the norm of the rendered image gradient, these unstructured high-frequency noises are effectively smoothed out, while preserving the main geometric edges of the soft tissue surface. This spatial regularization complements the temporal regularization (linear basis functions) in this embodiment, together ensuring the consistency of the reconstruction results in the spatiotemporal dimensions.

[0034] S106: Based on RGB image rendering loss, depth map rendering loss and total variational loss, a total loss function is constructed. The function is trained on various 3D Gaussian functions, coarse-scale Gaussian functions and fine-scale Gaussian functions in the normalized Gaussian space to obtain the target Gaussian space and realize multi-scale 3D Gaussian reconstruction.

[0035] Specifically, the total loss function is expressed as: ; The weights are for the total variational loss. The position, rotation angle, scale, opacity, and spherical harmonic coefficients of each 3D Gaussian in the normalized Gaussian space are trained based on the total loss function; the center, variance, and weights of the coarse-scale and fine-scale Gaussian functions are also trained.

[0036] Reference Figure 2 The diagram shows the principle flowchart of a multi-scale 3D Gaussian reconstruction method for endoscopic scenes under extreme deformation. First, this embodiment uses 3D Gaussian as the scene representation for this method. A normalized Gaussian space is initialized using the image of the first frame, the instrument mask, and the depth map. This establishes a static normalized space for subsequent dynamic deformation, containing the initial 3D Gaussian properties. Then, based on all 3D Gaussians in the normalized Gaussian space, the complex dynamic deformation is decomposed into global low-frequency motion and local high-frequency residuals, and two linear basis functions with different time scales (such as...) are used. Figure 2 The fine-scale and coarse-scale Gaussian functions (shown in the lower left box) model soft tissue deformation, dynamically updating the properties of the standard 3D Gaussian based on time-driven principles. Next, an adaptive Gaussian perception strategy is used to distinguish between dynamic and static regions using a deformation mask. A more aggressive densification strategy is applied to extremely deformed regions, achieving fine-tuning of the 3D Gaussian lifetime density to suppress artifacts and redundancy, resulting in an optimized deformed 3D Gaussian. Finally, the optimized deformed 3D Gaussian is projected onto a 2D plane using differentiable rasterization. The rendered RGB image and depth map are supervised based on the input real RGB image and depth map, and a total variational loss is introduced. This ensures high-fidelity dynamic 3D reconstruction of extremely deformed endoscopic scenes while maintaining real-time rendering speed.

[0037] Based on the above embodiments, in order to verify the above method in extremely challenging scenarios, this embodiment constructs an extreme deformation evaluation benchmark with significantly increased inter-frame displacement by performing temporal downsampling of the original sequence at intervals. Quantitative and qualitative results on this benchmark and multiple endoscopic datasets demonstrate that En-MGS achieves state-of-the-art performance in reconstruction quality while ensuring real-time rendering speed.

[0038] The experimental environment for this embodiment was implemented on a system with a single NVIDIA RTX 4090 (24GB) GPU and a 16 vCPU Intel(R) Xeon(R) Gold 6430 CPU. The method in this embodiment uses 20 learnable linear basis functions, trained for 3000 iterations using the Adam optimizer, with a denserization every 30 iterations, and an initial learning rate of 0.0016. Hyperparameters were set to... , , , In this embodiment, each dataset is divided into a training set and a test set in a 7:1 ratio. All experiments are based on the PyTorch framework, run three times on a single dataset, and the average results are reported.

[0039] Specifically, this embodiment evaluates the performance of En-MGS on two publicly available endoscopic datasets. The datasets used for evaluation include the EndoNeRF dataset and the StereoMIS dataset, which are widely used to evaluate the performance of 3D reconstruction for deformation of non-rigid soft tissues.

[0040] Specifically, the EndoNeRF dataset consists of stereoscopic videos captured by the DaVinci robot during prostatectomy surgery. It includes two publicly available videos: video 1, pulling_soft_tissues, shows the process of soft tissue being pulled by surgical instruments, and video 2, cutting_tissues_twice, shows the process of soft tissue being cut by surgical instruments. In addition to binocular RGB images, the dataset also includes manually annotated surgical instrument masks and STTR-Light estimated depth maps, 63 frames and 156 frames respectively. This dataset is widely used for benchmarking algorithms for endoscopic scene reconstruction. The StereoMIS dataset consists of stereoscopic videos captured by the DaVinci Xi robot from live pigs. In this embodiment, sequences 9750-9900 were selected from P2_6, and sequences 10060-10360 were selected from P3. This embodiment uses the methods provided in [the document / reference] for preprocessing to obtain RGB images, surgical instrument masks, and depth maps. Furthermore, this embodiment performs temporal downsampling on the two original sequences at two-frame intervals to create highly non-rigid soft tissue deformation, constructing an extreme deformation evaluation benchmark with significantly increased inter-frame displacement. This processing multiplies the soft tissue displacement amplitude between adjacent frames, thus simulating rapid motion or frame drops that may occur in conventional endoscopic videos. This greatly tests the algorithm's ability to capture transient extreme deformations, referred to below as SM1 and SM2, which are 76 and 75 frames respectively. This dataset presents a challenge for reconstructing extreme deformation endoscopic scenes.

[0041] For the StereoMIS dataset, this embodiment employs a preprocessing workflow similar to that of Deform3DGS and EH-SurGS. First, a specified sequence segment is selected from the original binocular endoscopic video, and the left-eye RGB image is extracted as the primary training view. Then, the corresponding depth map is estimated using the binocular image, and the automatically generated surgical instrument mask provided in the StereoMIS dataset is read to eliminate interference from instrument-occluded areas on soft tissue reconstruction. Next, the RGB image, depth map, and instrument mask are sized and normalized to maintain consistent spatial resolution. Finally, the data in the StereoMIS dataset is organized into the same format as that in the EndoNeRF dataset.

[0042] This embodiment uses a standard set of image quality evaluation metrics to assess the reconstruction quality of the method, including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS).

[0043] PSNR is used to measure the mean squared error (MSE) between images. The higher the PSNR, the better the image quality. The expression for PSNR is: , ;in, and These are the width and height of the image, respectively. and These are the input real RGB image and the rendered RGB image, respectively. This represents the maximum possible value of the image pixels.

[0044] SSIM is used to consider the similarity between images at multiple scales. Values ​​range from 0 to 1, with values ​​closer to 1 indicating higher similarity and better image quality. The SSIM expression is: ;in, , , Calculated using Gaussian window convolution. For variance, For covariance.

[0045] LPIPS is used to measure the perceptual quality of an image, with values ​​ranging from 0 to 1. A lower value indicates higher perceptual quality. The expression is: ;in, For feature extraction layer, For the first The feature activation values ​​of the layer after channel normalization. The weight vector used to scale the activation channel. This is a dot product operation.

[0046] In addition, this embodiment also reports the training time (Second) and rendering speed (Frames Per Second) to evaluate the computational efficiency of the method.

[0047] This embodiment demonstrates a comparative analysis of En-MGS and other methods, including quantitative and qualitative analysis. The methods compared include not only the NeRF-based state-of-the-art (SOTA) method Forplane, but also 3DGS-based methods such as EndoGaussian, Deform3DGS, EH-SurGS, and Endo-4DGX, all representative methods for current endoscopic scene reconstruction. EndoGaussian uses a similar architecture to Forplane—a decomposed feature plane plus a small MLP—to simulate soft tissue motion; Deform3DGS was the first to introduce linear basis functions into endoscopic scene reconstruction, eliminating the dependence on high-dimensional feature planes and MLPs, reducing single-scene training time to the minute level, making it a state-of-the-art (SOTA) method in terms of training time and rendering speed; EH-SurGS can handle irreversible dynamic soft tissue deformation; and Endo-4DGX is specifically designed for drastic lighting changes in endoscopic scenes, such as low light or overexposure.

[0048] First, to evaluate the performance metrics of the method in this embodiment, the experimental comparison results of En-MGS and other methods in terms of PSNR, SSIM, LPIPS, rendering speed (FPS), and training time (Time) are shown in Tables 1 and 2. The symbol ↑ indicates that the higher the value, the better the performance, and the symbol ↓ indicates that the lower the value, the better the performance.

[0049] Table 1 Comparison of experimental results for evaluation metrics on the EndoNeRF dataset.

[0050] Table 2 Comparison of experimental results for evaluation metrics on the StereoMIS dataset

[0051] As shown in Tables 1 and 2, the method in this embodiment exhibits the best performance in PSNR, SSIM, and LPIPS across all datasets, demonstrating that En-MGS achieves high-fidelity reconstruction quality in endoscopic scenarios. However, while the rendering speed (FPS) and training time (Time) of this embodiment are still lower than Deform3DGS and EH-SurGS, this is because the adaptive Gaussian perception strategy proposed in this embodiment requires more frequent densification, resulting in a slightly higher number of Gaussians and more optimized parameters, leading to increased training time and decreased rendering speed (FPS). However, this speed reduction is worthwhile given the improved rendering quality, and the average FPS reaches 225.5, fully meeting the real-time requirements of endoscopic surgery. (Refer to...) Figure 3 The figure shown is a comparison of the experimental results of PSNR and rendering speed FPS between the present invention and the comparison method; see reference. Figure 4The figure shown is a comparison of the PSNR and training time (Time) of the present invention and the comparison method; based on Figure 3 and Figure 4 The PSNR metrics of all methods were visualized and compared with the experimental results of rendering speed and training time, which more clearly shows the superiority of En-MGS in endoscopic image scene reconstruction.

[0052] Reference Figure 5 The image shows a comparison of the rendering results of the present invention and the comparison method on the EndoNeRF and StereoMIS datasets. Figure 5 Each row displays the rendering results of different methods on a single dataset. The first four columns show the rendering results of other methods: EndoGaussian, Deform3DGS, EH-SurGS, and Endo-4DGX. The fifth column shows the En-MGS method used in this embodiment. The last column, Reference, shows the input image. The PSNR metric is displayed in the upper left corner of each rendered image. Furthermore, this embodiment specifically indicates the rendering gap areas for each method and magnifies these areas for better identification. Figure 5 As can be seen from the comparison of the results rendered on the EndoNeRF-Pulling and StereoMIS-SM2 datasets, using the input image as a reference, it is clear that the En-MGS of this embodiment can render details that other methods cannot, proving that the method of this embodiment has the best high-fidelity reconstruction quality in endoscopic scenarios.

[0053] Reference Figure 6 The image shown illustrates the rendering results of the present invention and the comparison method on the EndoNeRF and StereoMIS datasets over time. Each method is presented with two rows of images: the first row shows the reconstructed point cloud, and the second row shows the rendered image, both indicating the rendering progress at a given time step. Furthermore, Figure 6 The last line is the input image. From Figure 6 It can be seen that En-MGS's rendering performance on the EndoNeRF dataset is not significantly different from Deform3DGS and EH-SurGS, but it has advantages over EndoGaussian and Endo-4DGX. Furthermore, En-MGS's rendering performance on the StereoMIS dataset is superior to Deform3DGS and EH-SurGS, indicating that the method in this embodiment has better adaptability to highly non-rigid soft tissue deformation caused by surgical instruments, and also proving the effectiveness of the multi-scale Gaussian deformation modeling method proposed in this embodiment. Moreover, the image rendered from the Gaussian point cloud by the method in this embodiment is free of artifacts, demonstrating the effectiveness of the adaptive Gaussian perception strategy proposed in this embodiment.

[0054] The above embodiment illustrates the explicit decomposition of all Gaussian functions into coarse-scale and fine-scale components, which includes two parameters. and , are the scaling factor in the coarse-scale component and the shrinking factor in the fine-scale component, respectively. It should be noted that Deform3DGS and EH-SurGS use a uniform-scale Gaussian function. Therefore, this embodiment discusses the experimental results of these two parameters on the StereoMIS dataset to explain why this embodiment uses... and Set to 3.5 and 1.5. This embodiment will... Set them to 2.5, 3.0, 3.5, and 4.0 respectively. Sixteen groups of experiments were set to values ​​of 0.5, 1.0, 1.5, and 2.0, respectively, and named as group (a) to group (p). The method in this embodiment is group (k). Other conditions in each group of experiments were kept consistent to ensure the reliability and comparability of the experimental results. The evaluation indicators, mainly PSNR, SSIM, and LPIPS, were the focus, as shown in Table 3.

[0055] Table 3 Components at different scales and Comparison of experimental results on evaluation metrics on the StereoMIS dataset

[0056] Reference Figure 7 As shown, the PSNR index varies with... The curve variation graph; refer to Figure 8 As shown, the PSNR index varies with... The curve variation graph; from Table 3, Figure 7 and Figure 8 It can be seen from this that, When set to 3.5, PSNR, SSIM, and LPIPS increase accordingly. As the value increases, the performance generally shows a trend of first increasing and then decreasing. Optimal performance is achieved when the value is set to 1.5 because... It focuses on capturing fine residuals at local, high frequencies, and can respond to rapidly changing transient deformations. If If the scale is too large, the support of the fine-scale layer widens, and the effect of the fine-scale layer overlaps with that of the coarse-scale layer, resulting in the loss of high-frequency characteristics, smoothing out detailed deformations, and potentially leading to insufficient fitting of transient deformations; if If the scale is too small, the support of the fine-scale layer becomes narrow, and only very recent time points participate. The gradient may be sharp and unstable, easily leading to overfitting and training oscillations. If time sampling is sparse, the contribution of the fine-scale layer is almost zero. When set to 1.5, at the same time When the value is set to 3.5, the performance of almost all metrics is optimal because... Designed to capture large-scale, low-frequency global displacements, it has a wider temporal receptive field. If If the size is too large, the coarse-scale layer supports widen, causing Gaussian layers to alias over time, potentially spanning multiple dynamic stages and resulting in unrelated motion features being incorrectly coupled together; if If the size is too small, the support of the coarse-scale layer becomes narrow, and the effect of the coarse-scale layer overlaps with that of the fine-scale layer, failing to provide long-range motion constraints and resulting in a loss of global consistency. Therefore, this embodiment ultimately... Set to 3.5 and Set it to 1.5.

[0057] Secondly, this embodiment discusses the number of linear basis functions used in the multi-scale Gaussian deformation modeling method. The experimental results of parameter variations on the EndoNeRF and StereoMIS datasets are used to illustrate why this embodiment uses... Let 20 be the number of linear basis functions. This embodiment will... The experiments were set to 14, 16, 18, 20, 22 and 24, for a total of 6 groups, named group (a) to group (f), with the method in this embodiment being group (d). Other conditions were kept consistent for each group of experiments to ensure the reliability and comparability of the experimental results. This embodiment evaluated the reconstruction quality and computational efficiency of each group of experiments, focusing on evaluation metrics such as PSNR, SSIM, LPIPS, rendering speed (FPS) and training time (Time), as shown in Tables 4 and 5.

[0058] Table 4 Number of different linear basis functions Comparison of experimental results on evaluation metrics on the EndoNeRF dataset

[0059] Table 5 Number of different linear basis functions Comparison of experimental results on evaluation metrics on the StereoMIS dataset

[0060] As can be seen from Tables 4 and 5, we should first focus on the two metrics of rendering speed (FPS) and training time (Time), and then... As the number of linear basis functions increases, the FPS decreases while the time increases, indicating that increasing the number of linear basis functions reduces the computational efficiency of the method. This is because... The larger the value, the more parameters need to be optimized. Then, for the two metrics SSIM and LPIPS, as... As the value of PSNR continues to increase, the actual value remains relatively stable. Finally, the main focus is on the PSNR metric, which... As the value increases, the performance of PSNR initially increases and then remains almost constant, with the key saturation point being... Set to 20, when When it is greater than 20, PSNR will The value fluctuated around 20 without any upward trend. Therefore, this embodiment comprehensively balanced reconstruction quality and computational efficiency, ultimately reducing the number of linear basis functions. Set it to 20.

[0061] Then, this embodiment discusses the number of scales in the multi-scale Gaussian deformation modeling method. The experimental results of parameter variations on the StereoMIS dataset will be used to explain why this embodiment chooses a dual-scale approach instead of more scales. This embodiment will... Four groups of experiments were designated as 1, 2, 3, and 4, respectively, and named as group (a) to group (d). Group (a) used a uniform scale; group (b) used a dual-scale method with parameters of 3.5 and 1.5; group (c) used a triple-scale method with parameters of 3.5, 2.5, and 1.5; and group (d) used a quadruple-scale method with parameters of 3.5, 2.5, 1.5, and 0.5. All other conditions were kept consistent across the groups to ensure the reliability and comparability of the results. This embodiment evaluated the reconstruction quality and computational efficiency of each group, focusing primarily on evaluation metrics such as PSNR, SSIM, LPIPS, rendering speed (FPS), and training time (Time), as shown in Table 6.

[0062] Table 6 Number of different scales Comparison of experimental results on evaluation metrics on the StereoMIS dataset

[0063] As shown in Table 6, group (a) using a uniform scale has better rendering speed (FPS) and training time (Time) compared to groups using multiple scales (b), (c), and (d), while the differences in rendering speed (FPS) and training time between groups using different scales (b), (c), and (d) are not significant. Furthermore, with... As the value increases, the three indices PSNR, SSIM, and LPIPS show a trend of first increasing and then decreasing, with group (b) achieving the best reconstruction quality. Therefore, this embodiment will... Let 2 be the scale number of the method in this embodiment.

[0064] Next, this embodiment discusses the scaling factor in densification. The experimental results of parameter changes on the StereoMIS dataset are used to illustrate why this embodiment sets... This embodiment will The experiments were set to 1.0, 1.5, and 2.0 respectively, for a total of three groups, named group (a) to group (c). Other conditions were kept consistent for each group of experiments to ensure the reliability and comparability of the experimental results. This embodiment evaluated the reconstruction quality and computational efficiency of each group of experiments, focusing on the evaluation metrics of PSNR, SSIM, LPIPS, rendering speed (FPS), and training time (Time), as shown in Table 7.

[0065] Table 7 Different scaling factors Comparison of experimental results on evaluation metrics on the StereoMIS dataset

[0066] As can be seen from Table 7, with With the continuous increase of [the data type], the four metrics—PSNR, SSIM, LPIPS, and training time—show a consistently increasing trend, while the rendering speed (FPS) shows a consistently decreasing trend. This demonstrates that more aggressive densification leads to better reconstruction quality but results in reduced computational efficiency. Therefore, this embodiment takes all factors into consideration and ultimately [determines the optimal approach]. Set it to 1.5.

[0067] In addition, this embodiment discusses the total variation of the image in Gaussian rendering optimization. Loss coefficient The experimental results of parameter changes on the StereoMIS dataset are used to illustrate why this embodiment sets... This embodiment designed 5 groups of experiments, named group (a) to group (e). Among them, group (a) contains... The value is 0.01; in group (b) It is 0.05; in group (c) The value is 0.10; in group (d) It is 0.50; in group (e) The value is 1.00. Other conditions for each group of experiments were kept consistent to ensure the reliability and comparability of the experimental results. This embodiment evaluated the reconstruction quality and computational efficiency of each group of experiments, focusing mainly on the evaluation metrics of PSNR, SSIM, and LPIPS, as shown in Table 8.

[0068] Table 8 Total variation for different images Loss coefficient Comparison of experimental results on evaluation metrics on the StereoMIS dataset

[0069] As can be seen from Table 8, with As the value increases, PSNR and SSIM show a trend of first increasing and then decreasing. It reaches its maximum at a certain time. LPIPS, however, shows a continuous increasing trend, due to the total variation of the image. Loss makes the image smoother within the pixel neighborhood. The LPIPS metric is sensitive to the loss of high-frequency information in the image, so when the coefficient... As the LPIPS increases, the LPIPS will gradually increase. Therefore, this embodiment will... Set it to 0.1.

[0070] Finally, the experimental results of this embodiment under extreme deformation are discussed. This embodiment simulates different degrees of deformation of non-rigid soft tissue by adjusting the frame interval of the temporal downsampling to verify the reliability of the method under extreme deformation. In this embodiment, the frame interval is set to 1, 2, 3 and 4 respectively, and experiments with the same sequence interval but different number of frames and different sequence intervals but the same number of frames are set, for a total of 7 groups of experiments, named group (a) to group (g). Group (a) includes 151 frames in the 9750-9900 sequence of P2_6 and 150 frames in the 10060-10360 sequence of P3 with a frame interval of 1; group (b) includes 76 frames in the 9750-9875 sequence of P2_6 and 75 frames in the 10060-10210 sequence of P3 with a frame interval of 1; group (c) is the dataset with a frame interval of 2 used in this embodiment; group (d) includes Group (e) contains 51 frames in the 9750-9900 sequence of P2_6 and 50 frames in the 10060-10360 sequence of P3, with a frame interval of 3; Group (f) contains 38 frames in the 9750-9900 sequence of P2_6 and 38 frames in the 10060-10360 sequence of P3, with a frame interval of 4; Group (g) contains 76 frames in the 9750-10050 sequence of P2_6 and 75 frames in the 10060-10660 sequence of P3, with a frame interval of 4. All other conditions were kept consistent for each group of experiments to ensure the reliability and comparability of the experimental results. In this embodiment, the reconstruction quality of each group of experiments was evaluated, mainly focusing on the evaluation indicators of PSNR, SSIM and LPIPS, as shown in Table 9.

[0071] Table 9 Comparison of experimental results for evaluation metrics on the StereoMIS dataset with different frame intervals.

[0072] Reference Figure 9 The image shows a comparison of PSNR metrics for different frame intervals on the StereoMIS dataset in this embodiment; refer to... Figure 10The image shows a comparison of the SSIM metrics for different frame intervals on the StereoMIS dataset in this embodiment; refer to... Figure 11 As shown, this is a comparison chart of LPIPS metrics for different frame intervals on the StereoMIS dataset in this embodiment; from Table 9, Figures 9 to 11 As can be seen, PSNR and SSIM increase with the increasing frame interval, while LPIPS decreases, indicating that the greater the degree of soft tissue deformation, the greater the challenge to the method in this embodiment. Therefore, the strategy of using temporal downsampling with a frame interval of 2 frames to form highly non-rigid soft tissue deformation is valid.

[0073] More importantly, this embodiment compares and analyzes the reconstruction quality of En-MGS with other methods at different sampling intervals, and conducts experiments on groups (b), (c), (e), and (g), as shown in Table 10.

[0074] Table 10 Comparison of experimental results of different methods on the StereoMIS dataset with different frame intervals.

[0075] As shown in Table 10, the reconstruction accuracy of all methods decreases with the increase of the frame interval. However, the experimental results of En-MGS are superior to those of other methods, which fully demonstrates the superiority and robustness of this embodiment in modeling the deformation of non-rigid soft tissues.

[0076] Based on the above embodiments, this embodiment discusses the effectiveness of the proposed multi-scale Gaussian deformation modeling method and adaptive Gaussian perception strategy. Furthermore, this embodiment also discusses the introduction of total variational loss for the image. The effectiveness of the experiment was assessed. This embodiment designed eight experiments on the StereoMIS dataset, named groups (a) to (h). All other conditions were kept consistent across the groups to ensure the reliability and comparability of the results. The reconstruction quality of each group was evaluated, primarily focusing on the evaluation metrics PSNR, SSIM, and LPIPS, as shown in Table 11.

[0077] Table 11 Impact of each key module on the reconstruction effect of the method in this embodiment

[0078] As shown in Table 11, group (b) used a multi-scale Gaussian deformation modeling method, which improved the reconstruction quality compared to group (a), proving its effectiveness; group (c) used an adaptive Gaussian perception strategy, which also improved the reconstruction quality compared to group (a), proving its effectiveness; group (d) used image total variational loss. Compared to group (a), PSNR and SSIM metrics improved, while LPIPS slightly decreased. This is due to the total variational loss of the image. This makes the image smoother within the pixel domain. The LPIPS metric detects the loss of high-frequency information in the image, thus worsening the metric, and it does not introduce total variational loss of the image. The LPIPS performance in group (e) is superior to that in group (h) which uses the loss, further demonstrating this point. Finally, when using group (h) with all key modules, the method achieves optimal PSNR and SSIM metrics, and the LPIPS performance is close to optimal, proving the effectiveness of all modules in this embodiment.

[0079] In addition, this embodiment discusses whether the proposed method can mitigate the problem of artifacts and redundancy easily generated by 3D Gaussians in regions of extreme deformation. Therefore, this embodiment visualizes a comparison of the rendered images of Table 11(a) and Table 11(h) on the StereoMIS dataset; refer to Figure 12 The image shown is a comparison of rendered images from the StereoMIS dataset. Figure 12 (a) is the rendered image of group (a) in Table 11 on the StereoMIS dataset. Figure 12 (h) represents the rendered images of group (h) in Table 11 on the StereoMIS dataset; artifacts and redundant parts are marked with green dashed boxes. It can be seen that on StereoMIS-SM1, Figure 12 Group (a) compared to Figure 12 Group (h) is more blurred in this area, with some visual floating objects, indicating that these 3D Gaussians were not removed, i.e., artifacts are present. On StereoMIS-SM2, Figure 12 Group (a) shows that the 3D Gaussian simulation of extreme deformation in this region is "stretched" to a longer scale and has a finer texture, indicating that these 3D Gaussians were not removed, i.e., there is redundancy. Figure 12 The texture of the rendered image in group (h) in this area is compared to Figure 12 Group (a) shows a clearer, high-fidelity reconstruction. This demonstrates that the method of this embodiment can mitigate the artifacts and redundancy issues that easily arise in 3D Gaussians in regions of extreme deformation.

[0080] This embodiment proposes En-MGS (Endoscope Multi-scale Gaussian Splatting), a novel dynamic 3D reconstruction method for endoscopic scenes under extreme deformation. First, this embodiment proposes a multi-scale Gaussian deformation modeling method. This method decomposes the dynamic deformation of soft tissues under extreme deformation into stable low-frequency global motion and local high-frequency residuals, and models them using learnable linear basis functions at different time scales. By decomposing the original single Gaussian linear basis function into coarse-scale and fine-scale parts, the coarse-scale basis function is responsible for modeling large-scale, smooth global deformation, while the fine-scale basis function is used to capture local, drastic deformation details, achieving accurate capture of local detailed deformation of highly non-rigid moving soft tissues. Secondly, to suppress artifacts and redundancy generated in extremely deformable regions, this embodiment proposes an adaptive Gaussian perception strategy. This strategy constructs a non-uniform gradient accumulation and splitting mechanism guided by deformation masks. By utilizing deformation masks to identify deformable regions, a more aggressive densification process is applied to these regions, achieving adaptive density control over the entire lifecycle of 3D Gaussians. This prevents the phenomenon of 3D Gaussian moving gradients being easily diluted over time, ensuring a more accurate geometric distribution and effectively suppressing artifacts and redundancy generated in extremely deformable regions. Extensive comprehensive experiments on the EndoNeRF and StereoMIS datasets demonstrate that, in endoscopic scenarios involving extremely deformed soft tissues, En-MGS outperforms existing state-of-the-art methods in reconstruction quality metrics (PSNR, SSIM, and LPIPS) while maintaining a real-time rendering speed of 240 FPS, providing a pathway for real-time tasks such as surgery.

[0081] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0083] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0085] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenes under extreme deformation, characterized in that, include: Initialize the normalized Gaussian space based on the input real RGB image, real depth map and surgical instrument mask; The dynamic deformation of each 3D Gaussian in the normal Gaussian space in the time dimension is decomposed into global low-frequency motion and local high-frequency motion, and modeled by coarse-scale Gaussian function and fine-scale Gaussian function respectively to calculate the deformation of each 3D Gaussian at the corresponding time, including position, rotation, scale and opacity, and obtain the deformed Gaussian space. Based on the deformation mask, the deformed and non-deformed regions in the deformed Gaussian space are obtained, and radical compaction and compaction are performed respectively to obtain the optimized deformed Gaussian space. The optimized deformed Gaussian space is projected onto a two-dimensional plane to obtain the rendered RGB image and depth map. These are then compared with the real RGB image and real depth map to construct the RGB image rendering loss and the depth map rendering loss. The total variational loss is constructed based on the difference between adjacent pixels in the rendered RGB image. Based on RGB image rendering loss, depth map rendering loss and total variational loss, a total loss function is constructed. The function is then trained on various 3D Gaussian functions, coarse-scale Gaussian functions and fine-scale Gaussian functions in the normalized Gaussian space to obtain the target Gaussian space and achieve multi-scale 3D Gaussian reconstruction.

2. The multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenes under extreme deformation as described in claim 1, characterized in that, Based on the input real RGB image, real depth map, and surgical instrument mask, the normalized Gaussian space is initialized and represented as: ； in, Represents a gauged Gaussian space. This represents the intrinsic parameter matrix of the camera used to obtain the true RGB image. The inverse matrix, Represents the true depth map. Represents a true RGB image. This indicates element-wise multiplication. This indicates a surgical instrument mask.

3. The multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenes under extreme deformation as described in claim 1, characterized in that, The dynamic deformation of each 3D Gaussian in the normalized Gaussian space over time is decomposed into global low-frequency motion and local high-frequency motion. Coarse-scale and fine-scale Gaussian functions are used to model these movements respectively, calculating the deformation of each 3D Gaussian at corresponding time points in terms of position, rotation, scale, and opacity. This yields the deformed Gaussian space, including: For each 3D Gaussian position Rotation angle ,size With Opacity Multiple learnable Gaussian functions are preset to simulate the dynamic deformation of each 3D Gaussian in the time dimension, and the dynamic deformation is decomposed into global low-frequency motion and local high-frequency motion. At every moment Multiple learnable Gaussian functions are explicitly decomposed into coarse-scale Gaussian functions. With fine-scale Gaussian function , respectively represented as: ； ； The global low-frequency motion is simulated using a coarse-scale Gaussian function, and the local high-frequency motion is simulated using a fine-scale Gaussian function. The deformations of the position, rotation angle, size, and opacity of each 3D Gaussian are obtained and expressed as follows: ； Based on the deformed 3D Gaussians, a deformed Gaussian space is formed; in, This represents the natural exponential function. and Let the center and variance of the learnable Gaussian function be represented, respectively. Indicates the coarse-scale scaling factor. Indicates the fine-scale reduction factor; , express Time parameters Based on its attribute set The shape variable, , , and Representing parameters respectively The learnable weight set, center set, and variance set of multiple Gaussian functions; This represents the total number of coarse-scale Gaussian functions. , , and Representing parameters respectively The Learnable weights, center, and variance of a coarse-scale Gaussian function; This represents the total number of fine-scale Gaussian functions. , , and Representing parameters respectively The Learnable weights, center, and variance of a fine-scale Gaussian function.

4. The multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenes under extreme deformation as described in claim 1, characterized in that, Based on the deformation mask, the deformed and non-deformed regions in the deformed Gaussian space are obtained, including: Obtain the average deformation of each 3D Gaussian in the deformed Gaussian space compared to its normal Gaussian space, and compare it with a first preset threshold: If the average deformation of the 3D Gaussian is greater than the first preset threshold, the 3D Gaussian is assigned to the first candidate dynamic region; otherwise, the 3D Gaussian is assigned to the first candidate static region. Obtain the difference in local rendering loss between the deformed Gaussian space and the normalized Gaussian space for each 3D Gaussian, and compare it with the second preset threshold: If the difference is less than the second preset threshold, the 3D Gaussian is assigned to the second candidate static region; otherwise, the 3D Gaussian is assigned to the second candidate dynamic region. Obtain the intersection of the first candidate dynamic region and the second candidate dynamic region, and use it as the deformation region; Obtain the intersection of the first candidate static region and the second candidate static region as the non-deformable region.

5. The multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenes under extreme deformation as described in claim 1, characterized in that, For the deformed and undeformed regions in the deformed Gaussian space, radical compaction and compaction are performed respectively to obtain the optimized deformed Gaussian space, including: For each 3D Gaussian in the undeformable region, when its cumulative position gradient exceeds the basic gradient threshold, a densification operation is performed on it to obtain a dense undeformable region; the densification operation is cloning or splitting. The basic gradient threshold is reduced by a preset ratio to obtain the low-order gradient threshold. For each 3D Gaussian in the deformable region, a gain weight is introduced when accumulating the position gradient. When the gain accumulation of the position gradient exceeds the low-order gradient threshold, a densification operation is performed on it to obtain a dense deformable region. Based on the dense deformable region and the dense non-deformable region, an optimized deformable Gaussian space is obtained.

6. The multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenes under extreme deformation as described in claim 5, characterized in that, Gain cumulative position gradient , represented as: ； in, Indicates an indicator function, if In the deformation zone If the value is 1, the indicator function value is 1; otherwise, it is 0. This represents the gain weight with a value greater than 1. This represents the position gradient magnitude of a 3D Gaussian.

7. The multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenes under extreme deformation as described in claim 1, characterized in that, The optimized deformed Gaussian space is projected onto a two-dimensional plane, and the projection onto the two-dimensional plane is then processed. Composite the RGB image and depth map to be rendered.

8. The multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenes under extreme deformation as described in claim 1, characterized in that, The optimized deformed Gaussian space is projected onto a 2D plane to obtain the rendered RGB image and depth map. These are then compared with the real RGB image and real depth map to construct the RGB image rendering loss and the depth map rendering loss, including: RGB image rendering loss , represented as: ; Depth map rendering loss , represented as: ; in, Indicates surgical instrument mask, This indicates element-wise multiplication. and These represent the rendered RGB image and the actual RGB image, respectively. and These represent the rendered depth map and the actual depth map, respectively.

9. The multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenes under extreme deformation as described in claim 1, characterized in that, The total variational loss is constructed based on the differences between adjacent pixels in the rendered RGB image. , represented as: ； in, This represents the rendered RGB image. Indicates batch size, Indicates the number of channels. and These represent the height and width of the actual RGB image, respectively. Indicates batch as The number of channels is Position in the rendered RGB image Image pixel values ​​at that location.

10. The multi-scale three-dimensional Gaussian reconstruction method for endoscopic scenes under extreme deformation as described in claim 1, characterized in that, Training is performed on various 3D Gaussian, coarse-scale Gaussian, and fine-scale Gaussian functions in the normalized Gaussian space, including: The position, rotation angle, scale, opacity, and spherical harmonic coefficients of each 3D Gaussian in the normalized Gaussian space are trained. The center, variance, and weights of the coarse-scale Gausky function and the fine-scale Gausky function are trained.

Images