Endoscopic scene reconstruction method and related apparatus

Abstract

The application provides an endoscope scene reconstruction method and related device; a first Gaussian cell is generated according to an endoscope video frame and a camera pose; a second Gaussian cell is obtained by locally deforming and modulating the first Gaussian cell; spatial features and temporal features of the second Gaussian cell stored in a shared mesh are extracted, the spatial features and the temporal features are fused to obtain fused features, the fused features are decoded to obtain decoded features, and the decoded features and the second Gaussian cell are linearly combined to obtain a third Gaussian cell; the third Gaussian cell is projected to a two-dimensional plane through the camera pose, the rendering color of an image pixel on the two-dimensional plane is calculated to obtain a rendering image, a loss function is calculated according to the difference between the rendering image and the endoscope video frame, and the parameters of a scene reconstruction model are adjusted according to the loss function to obtain a target scene reconstruction model; fine-grained local deformation and global space-time relationship can be captured, and the rendering speed and reconstruction quality are improved.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to endoscopic scene reconstruction methods and related apparatus. Background Technology

[0002] Existing endoscopic scene reconstruction techniques mainly employ NeRF-based methods (such as EndoNeRF and EndoSurf) and GS-based methods (such as EndoGS and Deform3DGS). However, they have shortcomings in handling local soft tissue deformation and rapid viewpoint changes, resulting in unstable reconstruction and low fidelity. Summary of the Invention

[0003] The following is an overview of the topics described in detail in this article.

[0004] The purpose of this application is to at least partially solve one of the technical problems existing in the related technologies. The embodiments of this application provide an endoscope scene reconstruction method and related apparatus, which can capture fine-grained local deformation and global spatiotemporal relationships, thereby improving rendering speed and reconstruction quality.

[0005] An embodiment of the first aspect of this application provides an endoscopic scene reconstruction method, comprising: Acquire endoscope video frames and camera poses, and input the endoscope video frames and camera poses into the scene reconstruction model; A first Gaussian element is generated based on the endoscope video frame and the camera pose. The second Gaussian element is obtained by locally deforming and modulating the first Gaussian element. Extract the spatial and temporal features of the second Gaussian element stored in the shared grid, fuse the spatial and temporal features to obtain the fused features, decode the fused features to obtain the decoded features, and linearly combine the decoded features with the second Gaussian element to obtain the third Gaussian element; The third Gaussian primitive is projected onto a two-dimensional plane through the camera pose, and the rendering color of the image pixels on the two-dimensional plane is calculated to obtain a rendered image. A loss function is calculated based on the difference between the rendered image and the endoscope video frame. The parameters of the scene reconstruction model are adjusted based on the loss function to obtain the target scene reconstruction model. The target endoscope scene image is obtained by performing an endoscope scene reconstruction operation using the target scene reconstruction model.

[0006] According to certain embodiments of the first aspect of this application, the first Gaussian element includes multiple attributes, namely: spatial center, rotation parameter, scale, opacity, and spherical harmonic coefficient.

[0007] According to certain embodiments of the first aspect of this application, a second Gaussian element is obtained by locally deforming and modulating the first Gaussian element, including: The deformation parameters are calculated based on the first Gaussian element by periodically modulating the Gaussian function; The deformation parameters and the first Gaussian element are linearly combined to obtain the second Gaussian element.

[0008] According to certain embodiments of the first aspect of this application, the deformation parameters are calculated based on the first Gaussian element by periodically modulating a Gaussian function, including: The time decay trend of the first Gaussian element is captured by the basic Gaussian function. According to the time decay trend, the first Gaussian element is injected with periodic dynamic characteristics by the cosine function based on the introduced frequency and amplitude to obtain the deformation parameters.

[0009] According to certain embodiments of the first aspect of this application, extracting the spatial and temporal features of the second Gaussian elements stored in the shared grid includes: The spatial features of the second Gaussian element are stored through a shared spatial grid, and the temporal features of the second Gaussian element are stored through a shared temporal grid. Determine the target second high-level element from among multiple second high-level elements; Spatial features of the target second Gaussian element are extracted from the spatially shared grid using bilinear interpolation, and temporal features of the target second Gaussian element are extracted from the temporally shared grid.

[0010] According to certain embodiments of the first aspect of this application, fusing the spatial features and the temporal features to obtain fused features includes: multiplying the spatial features and the temporal features by element-wise multiplication to obtain fused features.

[0011] According to certain embodiments of the first aspect of this application, the following steps are performed by the periodic modulation Gaussian function module of the scene reconstruction model: the first Gaussian element is locally deformed and modulated to obtain the second Gaussian element; The periodically modulated Gaussian function module is composed of a basic Gaussian function layer, a periodically modulated layer, and a first linear combination layer connected in series, and multiple periodically modulated layers are superimposed to form a composite function module.

[0012] According to certain embodiments of the first aspect of this application, the following steps are performed by the dual-plane module of the scene reconstruction model: extracting the spatial and temporal features of the second Gaussian element stored in the shared mesh, fusing the spatial and temporal features to obtain fused features, decoding the fused features to obtain decoded features, and linearly combining the decoded features and the second Gaussian element to obtain a third Gaussian element; The dual-plane module consists of a shared mesh initialization layer, a bilinear interpolation layer, a feature fusion layer, an MLP decoding layer, and a second linear combination layer.

[0013] According to a second aspect of this application, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the endoscopic scene reconstruction method as described in the first aspect of this application.

[0014] According to a third aspect of this application, a computer storage medium stores computer-executable instructions for performing the endoscopic scene reconstruction method as described in the first aspect of this application.

[0015] The above-mentioned scheme has at least the following beneficial effects: Periodic modulation can represent multiple deformation parameter change patterns within the same time window, accurately matching the complex and nonlinear qualitative change processes of tissues in endoscopic scenes; a dual-plane shared mesh design significantly reduces computational burden while ensuring the capture of Gaussian primitive spatiotemporal coupling relationships, adapting to the needs of real-time reconstruction of endoscopic videos; each module addresses the issues of initial primitive quality, local deformation modeling, global viewpoint adaptation, and accuracy optimization in endoscopic reconstruction, collectively achieving efficient and accurate endoscopic scene reconstruction; it can capture fine-grained local deformations and global spatiotemporal relationships, improving rendering speed and reconstruction quality. Attached Figure Description

[0016] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0017] Figure 1 This is a schematic diagram of the endoscopic scene reconstruction method; Figure 2 This is a step-by-step diagram of the endoscopic scene reconstruction method; Figure 3 This is a diagram showing the steps involved in using a periodically modulated Gaussian function module to locally deform and modulate the first Gaussian element to obtain the second Gaussian element. Figure 4 This is a structural diagram of the scene reconstruction model; Figure 5 This is a structural diagram of a periodically modulated Gaussian function module; Figure 6 This is a structural diagram of a dual-plane module. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] The embodiments of this application provide an endoscopic scene reconstruction method and related apparatus, which aim to realize real-time, robust 3D dynamic reconstruction of endoscopic videos, improve surgical navigation accuracy, and solve complex deformation and global adjustment problems.

[0021] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0022] Reference Figure 1 and Figure 2 The endoscopic scene reconstruction method includes the following steps: Step S100: Obtain endoscope video frames and camera poses, and input the endoscope video frames and camera poses into the scene reconstruction model; Step S200: Generate the first Gaussian element based on the endoscope video frame and camera pose; Step S300: Locally deform and modulate the first Gaussian element to obtain the second Gaussian element; Step S400: Extract the spatial and temporal features of the second Gaussian element stored in the shared grid, fuse the spatial and temporal features to obtain the fused features, decode the fused features to obtain the decoded features, and linearly combine the decoded features and the second Gaussian element to obtain the third Gaussian element. Step S500: Project the third Gaussian element onto a two-dimensional plane through the camera pose, calculate the rendering color of the image pixels on the two-dimensional plane to obtain the rendered image, calculate the loss function based on the difference between the rendered image and the endoscope video frame, and adjust the parameters of the scene reconstruction model based on the loss function to obtain the target scene reconstruction model. Step S600: Perform endoscopic scene reconstruction operation using the target scene reconstruction model to obtain the target endoscopic scene image.

[0023] In step S100, video is captured by the endoscopic camera. Endoscopic video frames are acquired. and camera pose H represents the number of frames. Endoscopic video frames and camera poses are input into the scene reconstruction model.

[0024] Each camera pose includes an intrinsic parameter matrix K and an extrinsic parameter matrix T_i.

[0025] Reference Figure 4 The scene reconstruction model includes a Gaussian primitive initialization module, a periodically modulated Gaussian function module, a biplane module, and a model optimization module.

[0026] For step S200, the first Gaussian element is generated by the Gaussian element initialization module based on the endoscope video frame and the camera pose.

[0027] The input data for the Gaussian initialization module are: the set of endoscopic video frames and the corresponding set of camera poses.

[0028] In the Gaussian primitive initialization module, a motion-aware point fusion scheme is adopted to generate an initial Gaussian primitive set based on the motion information and camera pose of multiple video frames. Where P is the total number of primitives. Each primitive It includes 5 core attributes: Space Center Rotation parameters ,scale Opacity spherical harmonic coefficient Among them, the scale and rotation parameters are determined by... Guarantee the positive semi-qualitative nature of the covariance matrix.

[0029] The output data of the Gaussian initialization module is: the set of the first Gaussian elements.

[0030] By leveraging the motion-aware point fusion scheme to fully utilize the motion continuity of the video, the generated initial Gaussian primitives are densely distributed in space, especially exhibiting higher primitive density in the deformable regions of the surgical scene. This provides high-quality initial input for subsequent deformation modeling and viewpoint adjustment, reducing the optimization burden on subsequent modules. It lays the foundation for reconstruction by providing high-quality initial primitives.

[0031] For step S300, the following steps are performed by the periodically modulated Gaussian function module: the first Gaussian element is locally deformed and modulated to obtain the second Gaussian element.

[0032] Reference Figure 5 The periodically modulated Gaussian function module is composed of a basic Gaussian function layer, a periodic modulation layer, and a first linear combination layer connected in series. Multiple periodic modulation layers are superimposed to form a composite function module.

[0033] The core function of the periodically modulated Gaussian function module is to address the problem that traditional Gaussian functions cannot capture the complex temporal dynamic deformation of tissues in endoscopic scenes. Its structure is centered on learnable parameterized function layers, without traditional convolutional or fully connected layers. It achieves dynamic modulation of Gaussian deformation parameters through adaptive learning of function parameters, modeling local deformations and enhancing temporal dynamic adaptability.

[0034] The input data for the periodically modulated Gaussian function module is: the set of first Gaussian elements. And the timestamp information t of the video frame.

[0035] Reference Figure 3 The second Gaussian element is obtained by locally deforming and modulating the first Gaussian element using a periodically modulated Gaussian function module, including the following steps: Step S310: Calculate the deformation parameters based on the first Gaussian element by periodically modulating the Gaussian function; Step S320: Linearly combine the deformation parameters and the first Gaussian element to obtain the second Gaussian element.

[0036] Specifically, in step S310, the time decay trend of the first Gaussian element is captured by the basic Gaussian function, and the periodic dynamic characteristics are injected into the first Gaussian element according to the introduced frequency and amplitude using the cosine function based on the time decay trend to obtain the deformation parameters.

[0037] Specifically, for each initial first Gaussian element By periodically modulating the Gaussian function Calculate its deformation parameter set (Including position, rotation, and scale offset). Specifically, the time decay trend of deformation parameters is first captured using the basic Gaussian function. Then, learnable frequencies ω and amplitudes β are introduced, and periodic dynamic characteristics are injected through a cosine function to achieve composite deformation modeling of "decay trend + periodic fluctuation". Finally, the deformation parameters are... With the first Gorski Yuan By performing a linear combination, the second Gaussian element is obtained. .

[0038] The core building block of the periodically modulated Gaussian function module consists of a basic Gaussian function layer and a periodic modulation layer connected in series. It contains four learnable parameters: time center θ, width σ, oscillation frequency ω, and amplitude β, which together form the parameter set Φ={θ,σ,ω,β}. The basic Gaussian function layer is responsible for constructing the time decay profile of the deformed parameters, while the periodic modulation layer (based on a cosine function) is responsible for injecting periodic dynamic characteristics.

[0039] The output of the basic Gaussian function layer is used as the input to the periodic modulation layer, and the basic Gaussian function layer calculates the output. The periodic modulation layer calculates the output. Element-wise multiplication of the two outputs yields a periodically modulated Gaussian function. .

[0040] To achieve more comprehensive deformation modeling, the module uses B periodically modulated Gaussian function units superimposed (B is a hyperparameter) to form a composite function module. The composite function of the composite function module is represented as follows: The parameters of each element are learned independently, and the superposition results are directly used to calculate the position and displacement of the Gaussian elements. It also extends to the calculation of deformation parameters such as rotation and scale.

[0041] The output data of the periodically modulated Gaussian function module is: a set of second Gaussian elements modulated by local deformation.

[0042] The periodically modulated Gaussian function module breaks through the limitation of traditional single-peak Gaussian functions that can only capture monotonically changing patterns. Through periodic modulation, it can represent multiple deformation parameter change patterns within the same time window. At the same time, the amplitude parameter β can adaptively adjust the deformation amplitude, accurately matching the complex and nonlinear qualitative change process of tissue locality in the endoscopic scene, such as organ peristalsis and local deformation caused by instrument operation.

[0043] For step S400, the following steps are performed by the dual-plane module: extract the spatial and temporal features of the second Gaussian element stored in the shared mesh, fuse the spatial and temporal features to obtain the fused features, decode the fused features to obtain the decoded features, and linearly combine the decoded features and the second Gaussian element to obtain the third Gaussian element.

[0044] Reference Figure 6 The dual-plane module consists of a shared mesh initialization layer, a bilinear interpolation layer, a feature fusion layer, an MLP decoding layer, and a second linear combination layer. It aims to address the global viewpoint variation caused by rapid camera movement in endoscopic scenarios. Global adjustment is achieved by modeling the spatiotemporal coupling of Gaussian units, while simultaneously reducing computational costs. It captures global coupling to enable adaptive viewpoint adjustment.

[0045] The input data for the dual-plane module are: the set of second Gaussian primitives and the spatiotemporal coordinates corresponding to each primitive.

[0046] Shared grid via shared grid space Storing the spatial characteristics of the second Gaussian element, and sharing the mesh temporally via a shared mesh. The temporal features of the second Gaussian primitive are stored. Subsequently, bilinear interpolation is used to extract the spatial and temporal features of the corresponding primitives from the spatially and temporally shared grids. Element-wise multiplication fusion yields a feature vector that incorporates both spatial location and temporal dynamics information. The deformation parameter set consists of the position offset, rotation offset, and scale offset of the Gaussian elements from the three lightweight MLP decoder outputs. As a decoding feature, the deformed parameter set is linearly combined with the second Gaussian element to obtain the third Gaussian element. This forms the set of the third Gaussian elements. .

[0047] The shared mesh initialization layer serves as the basic feature storage unit for the module. It initializes two independent learnable tensor meshes, with a spatially shared mesh. Shared grid with time .in, A 3D tensor of N_x×N_y×D (N_x and N_y are spatial resolutions, and D is the feature dimension) is used to store spatial information in the xy plane. A 3D tensor of N_z×N_t×D (N_z is the spatial resolution along the z-axis, and N_t is the temporal resolution) is used to store spatiotemporal information in the zt dimension. Both meshes are initialized in the normalized domain [-1,1].

[0048] The bilinear interpolation layer acts as a bridge between the grid features and the Gaussian elements, receiving the spatiotemporal center coordinates of the Gaussian elements. Bilinear interpolation is performed on the two shared grids respectively. ,according to Calculate interpolation weights And extract spatial features; targeting ,according to Calculate interpolation weights It also extracts time features and outputs feature vectors with two dimensions, both of which are D.

[0049] The feature fusion layer uses element-wise multiplication to fuse the two D-dimensional feature vectors output by the bilinear interpolation layer, resulting in a feature vector that integrates spatiotemporal information. This layer design achieves tight coupling between spatial and temporal information through multiplication interactions, while avoiding complex calculations and reducing computational overhead.

[0050] The MLP decoding layer consists of three identical lightweight MLPs (miniature fully connected networks). Each MLP contains two fully connected layers (input dimension D, hidden dimension D / 2, and output dimension corresponding to the deformation parameter dimension), and the activation function is ReLU. The three MLPs respectively receive fused features. The position offset of the decoded output high-order primitives Rotational offset and scale offset , constitute the deformation parameter set .

[0051] The interlayer connection relationship is a shared mesh initialization layer - bilinear interpolation layer (processing two meshes in parallel) - feature fusion layer - MLP decoding layer (three parallel outputs), forming a linear process of "initialization-interpolation extraction-fusion-decoding". The output of each layer is directly used as the input of the next layer, with no feedback connection, ensuring computational efficiency.

[0052] By employing a biplane shared mesh design, 4D spatiotemporal information is decomposed into interpolation fusion of two 2D meshes. Compared to the traditional HexPlane method, this significantly reduces computational cost. Down to ( The number of HexPlane planes is reduced to significantly lower the computational burden while ensuring the capture of the spatiotemporal coupling relationship of Gaussian elements, thus meeting the needs of real-time reconstruction of endoscopic videos; the lightweight MLP decoding layer achieves efficient mapping from features to deformation parameters.

[0053] For step S500, the following steps are performed through the model optimization module: the third Gaussian primitive is projected onto a two-dimensional plane through the camera pose, the rendering color of the image pixels on the two-dimensional plane is calculated to obtain the rendered image, the loss function is calculated based on the difference between the rendered image and the endoscope video frame, and the parameters of the scene reconstruction model are adjusted based on the loss function to obtain the target scene reconstruction model.

[0054] The input data for the model optimization module are: the set of third Gaussian elements, the set of endoscopic video frames, and the set of camera poses.

[0055] In the model optimization module, based on the rendering principle of 3D Gaussian splatting, the third Gaussian splatter element is projected onto a 2D plane through the camera pose (projection covariance is...). (Using point cloud volume rendering formula) Calculate the rendered color for each pixel; construct a loss function using the difference between the rendered image and the original endoscopic video frame as the loss. (L is the rendering loss function), the learnable parameters of the entire network are iteratively optimized through backpropagation. The learnable parameters include Φ of the periodic modulation module, the shared mesh of the dual-plane module, and the MLP parameters.

[0056] The input data for the model optimization module is the trained target scene reconstruction model.

[0057] The model optimization module takes pixel-level rendering loss as the optimization target, directly correlates the difference between the model output and the original data, and ensures the accuracy of the model reconstruction results; the loss averaging of batch processing H-frame data enhances the model's adaptability to different frame scenes and avoids overfitting.

[0058] For step S600, an endoscopic image is captured by a camera. , ; K). The endoscopic image is input into the trained target scene reconstruction model. The endoscopic scene is reconstructed using the target scene reconstruction model to obtain the target endoscopic scene image, which is represented as: .

[0059] Each module of the scene reconstruction model uses Gaussian elements as the core data carrier, forming a hierarchical and progressive collaborative mechanism of "initialization - local deformation modulation - global perspective adjustment - iterative optimization" to accurately match the core requirements of endoscopic scene reconstruction.

[0060] In the foundational stage, the Gaussian primitive initialization module uses the MAPF scheme to generate densely distributed initial primitives, providing a spatial basis for subsequent deformation modeling. Its output is directly used as the input of the periodic modulation module, ensuring the continuity of data flow.

[0061] During the local-global joint adjustment phase, the periodic modulation module first targets the initial Gaussian element. Local deformation modeling is performed to address the nonlinear qualitative change problem in local tissues, and Gaussian elements are output. The dual-plane module is based on Gaussian elements. The spatiotemporal information is used to make global perspective adjustments to compensate for the perspective shift caused by the rapid movement of the camera. The two form a complementary adjustment from local to global, ensuring that the Gaussian element can match the local organizational form and adapt to global perspective changes.

[0062] During the closed-loop optimization phase, the model optimization module will use the Gaussian elements output by the biplane module. The image is converted into a rendered image. By comparing the loss with that of the original frame, the learnable parameters of all preceding modules are updated through backpropagation. This makes the deformation modeling of the periodic modulation module more consistent with the actual tissue movement and the perspective adjustment of the dual-plane module more accurate, forming a closed loop of "data processing - error feedback - parameter optimization". Finally, a model that can accurately reconstruct the endoscope scene is obtained.

[0063] Through the above collaboration, each module solved four core problems in endoscopic reconstruction: initial primitive quality, local deformation modeling, global view adaptation, and accuracy optimization, and together achieved efficient and accurate endoscopic scene reconstruction.

[0064] Embodiments of this application provide an electronic device. The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the endoscopic scene reconstruction method as described above.

[0065] This electronic device can be any smart terminal, including computers.

[0066] In general, for the hardware structure of electronic devices, the processor can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, to execute relevant programs and implement the technical solutions provided in the embodiments of this application.

[0067] The memory can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory and is called and executed by the processor.

[0068] Input / output interfaces are used to implement information input and output.

[0069] The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0070] The bus transmits information between various components of a device, such as the processor, memory, input / output interfaces, and communication interfaces. The processor, memory, input / output interfaces, and communication interfaces communicate with each other within the device via the bus.

[0071] Embodiments of this application provide a computer storage medium. The computer storage medium stores computer-executable instructions for performing the endoscopic scene reconstruction method described above.

[0072] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0073] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0074] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0075] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms. Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0078] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for reconstructing an endoscopic scene, characterized in that, include: Acquire endoscope video frames and camera poses, and input the endoscope video frames and camera poses into the scene reconstruction model; A first Gaussian element is generated based on the endoscope video frame and the camera pose. The second Gaussian element is obtained by locally deforming and modulating the first Gaussian element. Extract the spatial and temporal features of the second Gaussian element stored in the shared grid, fuse the spatial and temporal features to obtain the fused features, decode the fused features to obtain the decoded features, and linearly combine the decoded features with the second Gaussian element to obtain the third Gaussian element; The third Gaussian primitive is projected onto a two-dimensional plane through the camera pose, and the rendering color of the image pixels on the two-dimensional plane is calculated to obtain a rendered image. A loss function is calculated based on the difference between the rendered image and the endoscope video frame. The parameters of the scene reconstruction model are adjusted based on the loss function to obtain the target scene reconstruction model. The target endoscope scene image is obtained by performing an endoscope scene reconstruction operation using the target scene reconstruction model.

2. The endoscopic scene reconstruction method according to claim 1, characterized in that, The first Gaussian element includes several attributes: spatial center, rotation parameter, scale, opacity, and spherical harmonic coefficient.

3. The endoscopic scene reconstruction method according to claim 1, characterized in that, The second Gaussian element is obtained by locally deforming and modulating the first Gaussian element, including: The deformation parameters are calculated based on the first Gaussian element by periodically modulating the Gaussian function; The deformation parameters and the first Gaussian element are linearly combined to obtain the second Gaussian element.

4. The endoscopic scene reconstruction method according to claim 3, characterized in that, The deformation parameters are calculated based on the first Gaussian element by periodically modulating the Gaussian function, including: The time decay trend of the first Gaussian element is captured by the basic Gaussian function. According to the time decay trend, the first Gaussian element is injected with periodic dynamic characteristics by the cosine function based on the introduced frequency and amplitude to obtain the deformation parameters.

5. The endoscopic scene reconstruction method according to claim 1, characterized in that, Extract the spatial and temporal features of the second Gaussian element stored in the shared grid, including: The spatial features of the second Gaussian element are stored through a shared spatial grid, and the temporal features of the second Gaussian element are stored through a shared temporal grid. Determine the target second high-level element from among multiple second high-level elements; Spatial features of the target second Gaussian element are extracted from the spatially shared grid using bilinear interpolation, and temporal features of the target second Gaussian element are extracted from the temporally shared grid.

6. The endoscopic scene reconstruction method according to claim 1, characterized in that, The fusion of spatial features and temporal features to obtain fused features includes: multiplying the spatial features and temporal features by element-wise multiplication to obtain fused features.

7. The endoscopic scene reconstruction method according to claim 1, characterized in that, The periodic modulation Gaussian function module of the scene reconstruction model performs the following steps: the first Gaussian element is locally deformed and modulated to obtain the second Gaussian element; The periodically modulated Gaussian function module is composed of a basic Gaussian function layer, a periodically modulated layer, and a first linear combination layer connected in series, and multiple periodically modulated layers are superimposed to form a composite function module.

8. The endoscopic scene reconstruction method according to claim 1, characterized in that, The following steps are performed by the dual-plane module of the scene reconstruction model: extract the spatial and temporal features of the second Gaussian element stored in the shared mesh, fuse the spatial and temporal features to obtain the fused features, decode the fused features to obtain the decoded features, and linearly combine the decoded features and the second Gaussian element to obtain the third Gaussian element; The dual-plane module consists of a shared mesh initialization layer, a bilinear interpolation layer, a feature fusion layer, an MLP decoding layer, and a second linear combination layer.

9. An electronic device, characterized in that, include: The endoscopic scene reconstruction method as described in any one of claims 1 to 8 is provided in the memory, the processor, and the computer program stored in the memory and executable on the processor.

10. A computer storage medium, characterized in that, The device stores computer-executable instructions for performing the endoscopic scene reconstruction method as described in any one of claims 1 to 8.

Images