Single endoscope under soft tissue dynamic three-dimensional reconstruction method, computer equipment and storage medium

Abstract

The application provides a monocular endoscopic soft tissue dynamic three-dimensional reconstruction method, a computer device and a storage medium, and belongs to the technical field of image processing. In the method, the confidence map calculated from the depth prior is used to comprehensively evaluate and correct the depth, thereby solving the problem of geometric radiation blur; by using the concept of surface rendering, a density field represented by a signed distance function and a color field represented by a spherical harmonic function are adopted, precise depth prediction and efficient image rendering are realized by optimizing a loss function, more detailed scene details are restored, and accurate dynamic 3D reconstruction of soft tissue is obtained.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and particularly relates to a method, computer equipment and storage medium for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy. Background Technology

[0002] Endoscopic minimally invasive surgery has become a major surgical procedure for the treatment of soft tissue organs, such as partial resection of nephrotic tumors, partial resection of liver tumors, and pancreaticoduodenectomy. However, the limited environmental perception space and surgical time constraints of endoscopic minimally invasive surgery place high demands on the experience and skills of surgeons. Augmented reality surgical navigation can effectively alleviate the problem of limited intraoperative environmental perception for surgeons, providing surgical guidance and becoming a research hotspot in recent years.

[0003] In existing technologies, video overlay-based augmented reality surgical navigation is the main form of augmented reality navigation in endoscopic minimally invasive surgery. The implementation primarily involves reconstructing the anatomical structure model of the target tissue or organ from the patient's CT / MRI medical images, registering the anatomical structure model with the intraoperative endoscopic images using image registration technology, and finally enhancing the intraoperative scene using augmented reality display technology to achieve image guidance. Registration can be broadly categorized into two methods: point-based registration and area-based registration.

[0004] However, current problems include the fact that point-based registration surgical navigation systems often require manual intervention, which disrupts the normal surgical procedure, distracts the surgeon, and increases the risk of complications. Furthermore, registration using implanted markers is invasive, difficult to apply clinically, and often requires additional tracking equipment. Area-based registration surgical navigation requires intraoperative tissue morphology measurements to obtain surface information of the target tissue, but the complex intraoperative endoscopic imaging environment (smoke, reflections, etc.) makes it difficult to obtain dense reconstruction results of the target tissue. Simultaneously, due to the limited endoscopic field of view during surgery, the reconstructed area of ​​the target tissue surface is small, making automatic registration difficult. Most currently available endoscopic augmented reality navigation systems rely on manual or semi-automatic registration, with semi-automatic systems requiring manual intervention for initial registration. In addition, due to the unique characteristics of the abdominal cavity, soft tissues and organs deform during surgery, and many existing surgical navigation systems ignore this deformation, failing to compensate for it and resulting in poor navigation accuracy. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a method, computer equipment, and storage medium for dynamic three-dimensional reconstruction of soft tissue under a monocular endoscope.

[0006] In a first aspect, the present invention provides a method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy, comprising:

[0007] The neural radiation field is trained using endoscopic RGB images to obtain the neural depth prior of the target scene during surgery;

[0008] The RGB images from the endoscope are input into the trained depth estimation network, and the output is a depth prediction map for each RGB image.

[0009] Based on the neural depth prior of the target scene and the depth prediction map of the target RGB image, the confidence level of each pixel in the target RGB image is determined, and a depth confidence map is obtained.

[0010] A neural radiation field space is constructed, and light rays are emitted into the target RGB image to reconstruct the dynamic surgical scene;

[0011] Dynamic Gaussian sampling is used based on the depth confidence map, and point sampling is performed on each ray; concentrated sampling is performed in areas with high confidence, and uniform sampling is performed in areas with low confidence.

[0012] The coordinates of the sampling points on the light rays are combined with the timestamps, and then input into the deformable network after Fourier encoding to predict the displacement of the sampling points of each light ray based on the light ray sampling points in the static standard frame. The coordinates of all light ray sampling points in the dynamic frame are obtained by combining the displacement predicted by the deformable network with the coordinates of the sampling points in the standard frame.

[0013] Input the coordinates of all light sampling points into the SDF network to complete the mapping between the sampling point coordinates and the volume density, and obtain the distance from the sampling point to the target surface and the gradient of the SDF network;

[0014] The distance from the sampling point to the target surface and the gradient of the SDF network are input into the spherical harmonic function to obtain the RGB value of each sampling point;

[0015] The sampling points are rendered as pixel colors along the light rays, and the deformation network, SDF network, and spherical harmonic function are optimized based on the luminance difference between the pixel colors and the real RGB colors. The surface gradient and divergence of the sampling points are calculated, and the SDF network is converged through Eikonal loss and DivLoss to render the correct scene depth information.

[0016] Optionally, the step of determining the confidence level of each pixel in the target RGB image based on the neural depth prior of the target scene and the depth prediction map of the target RGB image to obtain a depth confidence map includes:

[0017] Construct the confidence weight function for pixel (u,v) in the i-th frame of the RGB image:

[0018]

[0019] Where σ(·) is the confidence weight function; The neural depth prior for pixel (u,v); is the depth prediction map for pixel (u,v); μ is the first truncation threshold; δ is the second truncation threshold.

[0020] Optionally, the construction of the neural radiation field space and the emission of light rays onto the target RGB image to reconstruct the dynamic surgical scene includes:

[0021] Construct an importance sampling expression based on binary device mask:

[0022]

[0023] Among them, J i Γ represents mask importance; M represents scaling factor; Γ represents the importance of the mask. i The binary ray filtering mask for the i-th frame of the RGB image; Represents pixel-wise multiplication; T is the total number of frames in the RGB image sequence; |·| F M represents the Frobenius norm; j The binary ray filtering mask for the j-th frame of the RGB image.

[0024] Optionally, the depth confidence map-based dynamic Gaussian sampling, which performs point sampling on each ray, includes:

[0025] The sampling points of each ray are represented as Gaussian pulses g(u,v,s,i) according to the confidence weighting function:

[0026]

[0027] Where e is the natural constant; τ is the exponent; and s represents the distance of the light rays.

[0028] Optionally, the step of combining the coordinates of the sampling points on the light rays with the timestamps, inputting them into the deformable network after Fourier encoding, to predict the displacement of the sampling points of each light ray based on the light ray sampling points in the static standard frame, and obtaining the coordinates of all light ray sampling points in the dynamic frame by combining the displacement predicted by the deformable network with the coordinates of the sampling points in the standard frame, includes:

[0029] Given a spatial point and time point t, Representing three dimensions; training the deformable network Ψ t The displacement Δx of each sampling point is output, and the position and time information are mapped using position encoding. The position and time information are then input into the deformable network to obtain:

[0030] (x+Δx)=x+Ψ t ;

[0031] (x+Δx) represents the coordinates of the sampled point after deformation.

[0032] Optionally, the step of inputting the coordinates of all light sampling points into the SDF network to complete the mapping between the sampling point coordinates and the volume density, and obtaining the distance from the sampling point to the target surface and the gradient of the SDF network, includes:

[0033] Construct the expression for the zero level set S of the signal distance function of the sampling points:

[0034]

[0035] in, Represents the signal distance function;

[0036] Map the deformed coordinates of the sampling points to s, f, and n:

[0037]

[0038] in, Represents a dimension; ω represents the number of dimensions; f represents the features of the object's surface; n is the gradient of the SDF network ξ(·).

[0039] Optionally, the step of inputting the distance from the sampling point to the target surface and the gradient of the SDF network into the spherical harmonic function to obtain the RGB value of each sampling point includes:

[0040] Construct the expression for the spherical harmonic function:

[0041]

[0042] Let represent a spherical harmonic function; θ is the polar angle in spherical coordinates; φ is the azimuth angle in spherical coordinates; L is a natural number; m is 0 or 1; π is pi. Represents Legendre polynomials;

[0043] Construct the color expression for the sampling point:

[0044]

[0045] Where c(·) represents the color of the sampling point; Sig represents the sigmoid function; Represents the spherical harmonic coefficients;

[0046] Calculate the RGB value of each pixel using the following formula:

[0047]

[0048] C(r) is the RGB value of each pixel; near is the nearest point of the ray; far is the farthest point of the ray; w(s) is the weight of each sampling point; c(D,k) represents the color of each sampling point; ρ(·) represents the opacity function corresponding to the volume density; 0≤u≤s; r(s) represents the ray; γ is the probability density function.

[0049] Optionally, the step of rendering the sampling points as pixel colors along the light rays, optimizing the deformation network, SDF network, and spherical harmonic function based on the luminance difference between the pixel colors and the true RGB colors, calculating the surface gradient and divergence of the sampling points, and converging the SDF network through Eikonal loss and Div Loss to render the correct scene depth information includes:

[0050] Construct the color loss expression:

[0051]

[0052] Among them, L color Indicates color loss; Indicates the rendered color;

[0053] Construct the Eikonal loss expression:

[0054]

[0055] Among them, L eikonal For Eikonal's loss; This represents the gradient at point x; ||||2 is the 2-norm;

[0056] Construct the Div loss expression:

[0057]

[0058] Among them, L div For Div loss; T represents; H(·) represents the Hessian matrix;

[0059] The loss of the entire model is calculated using the following formula:

[0060] L total =L color +λ1L eikonal +λ2L div ;

[0061] Where λ1 is the first equilibrium hyperparameter and λ2 is the second equilibrium hyperparameter.

[0062] In a second aspect, the present invention provides a computer device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the dynamic three-dimensional reconstruction method of soft tissue under monocular endoscopy as described in the first aspect.

[0063] Thirdly, the present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the dynamic three-dimensional reconstruction method for soft tissue under monocular endoscopy described in the first aspect.

[0064] This invention provides a method, computer device, and storage medium for dynamic 3D reconstruction of soft tissue under a monocular endoscope. The method utilizes a confidence map calculated from depth priors to comprehensively evaluate and correct depth, thereby solving the problem of geometrical radiometric blur. By employing the concept of surface rendering, a density field characterized by a signed distance function and a color field represented by a spherical harmonic function are used. Through optimization of the loss function, accurate depth prediction and efficient image rendering can be achieved, restoring more detailed scene details and obtaining accurate dynamic 3D reconstruction of soft tissue. Attached Figure Description

[0065] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 A flowchart illustrating a method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy provided in an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of a light spot sampling method based on Gaussian pulses provided in an embodiment of the present invention;

[0068] Figure 3 A quantitative data comparison chart of the image rendering quality of the embodiments of the present invention compared with the current state-of-the-art (SOTA) methods, provided for the purposes of this invention;

[0069] Figure 4 A qualitative comparison chart of the quality of tissue 3D reconstruction between the embodiments of the present invention and the current SOTA method, provided for the purposes of this invention;

[0070] Figure 5 A qualitative comparison diagram of the embodiments of the present invention with the current surface reconstruction method EndoSurf in terms of image rendering, surface restoration, and depth restoration, provided for the purposes of this invention;

[0071] Figure 6Another qualitative comparison diagram of the embodiments of the present invention with the current surface reconstruction method EndoSurf in terms of image rendering, surface restoration, and depth restoration, provided for the purposes of this embodiment of the invention;

[0072] Figure 7 The ablation experiment results of the loss function provided in the embodiments of the present invention are shown in the figure.

[0073] Figure 8 A qualitative comparison diagram of the quality of restoring occluded tissue texture in the embodiments of the present invention compared with the current SOTA method, provided for the purposes of this invention;

[0074] Figure 9 A schematic diagram of a monocular laparoscopic full-domain dynamic scene reconstruction system for intraoperative navigation and postoperative medical technique training provided in an embodiment of the present invention;

[0075] Figure 10 This is a schematic diagram of the concealed device provided in an embodiment of the present invention;

[0076] Figure 11 This is a schematic diagram of a dynamic three-dimensional reconstruction system for soft tissue under a monocular endoscope, provided as an embodiment of the present invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] Given a video clip of a surgical scene, the goal is to achieve instrument invisibility and reconstruct the texture and geometry of the tissue surface, thereby restoring a clear and dynamic surgical view based solely on RGB images. This process is represented as... Where i represents the frame index in the image sequence, and T represents the total number of frames; I i M represents the RGB image of the i-th frame; i It is a binary ray filtering mask corresponding to each RGB frame, designed to make smoke, bloodstains, and equipment invisible in video. Pose matrix P i Provides the transformation matrix from the world coordinate system to the image plane coordinate system. In the case of a single viewpoint, the pose matrix remains consistent across each frame. Variable t i It represents the timestamp of each frame, and the time is normalized using i / T.

[0079] Example 1

[0080] like Figure 1As shown, this embodiment provides a method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy, including:

[0081] Step 101: Train the neural radiation field using endoscopic RGB images to obtain the neural depth prior of the target scene during surgery.

[0082] During the training phase, the key supervision signal comes from deformation-based view synthesis. Once the depth value of each pixel in the target frame is estimated, the pixel coordinates are back-projected into the 3D camera space using known camera intrinsics. Utilizing the estimated self-motion trajectory, the 3D point cloud can be projected onto another image plane. Given the source and target images, view synthesis can be represented as:

[0083]

[0084] H(I s ) and H(I t ) represent the homogeneous expressions for the coordinates of the source and target view images, respectively; K represents the calibrated camera intrinsic parameter matrix; C t→s D represents the estimated self-motion from the target view to the source view; t The predicted depth map represents the target frame. Therefore, the photometric appearance loss can be smoothly derived from the deformation from the source view frame to the target view frame:

[0085] L s→t (I)=T(I s )-T(I t );

[0086] T(·) represents a differentiable inverse transformer through which gradients can be efficiently backpropagated to supervise the entire self-supervised framework.

[0087] Step 102: Input the endoscopic RGB image into the trained depth estimation network and output the depth prediction map for each RGB image.

[0088] Step 103: Based on the neural depth prior of the target scene and the depth prediction map of the target RGB image, determine the confidence level of each pixel in the target RGB image to obtain the depth confidence map.

[0089] In this embodiment, as Figure 2 As shown, a depth prior is combined with a biased depth map to guide the sampling of rays in a neural radiation field by determining an adaptive depth confidence level. Specifically, the depth prior is evaluated and scored based on the biased depth map, assigning higher confidence levels to rays showing greater similarity and lower confidence levels to rays showing less similarity.

[0090] Construct the confidence weight function for pixel (u,v) in the i-th frame of the RGB image:

[0091]

[0092] Where σ(·) is the confidence weight function; The neural depth prior for pixel (u,v); is the depth prediction map for pixel (u,v); μ is the first truncation threshold; δ is the second truncation threshold.

[0093] Concentrating sampling points near the depth prior on rays with high confidence, while discretely sampling depth on rays with low confidence, helps with depth correction.

[0094] Step 104: Construct a neural radiation field space and emit light rays into the target RGB image to restore the dynamic surgical scene.

[0095] In this step, a ray is projected for each pixel, with the model being r(s) = o + sd, where o is the ray origin (i.e., the camera center), d is the ray direction, and s represents the ray distance.

[0096] The purpose of this embodiment is to perform full-area tissue reconstruction under laparoscopy and restore tissue in areas obscured by surgical instruments. Therefore, it is important to avoid projecting light rays onto the entire image to eliminate surgical instruments. Specifically, a binary segmentation mask for surgical instruments is used, initializing the pixel values ​​of the instrument portion to 1 and the pixel values ​​of the tissue portion to 0.

[0097] For example, construct an expression for the importance sampling of binary instrument masks:

[0098]

[0099] Among them, J i Γ represents mask importance; M represents scaling factor; Γ represents the importance of the mask. i The binary ray filtering mask for the i-th frame of the RGB image; Represents pixel-wise multiplication; T is the total number of frames in the RGB image sequence; |·| F M represents the Frobenius norm; j The binary ray filtering mask for the j-th frame of the RGB image.

[0100] (1-M i The first term ensures that the importance of light projection onto the instrument portion is always 0, achieving effective instrument light filtering. The scaling factor Γ uses the Frobenius norm to assign a scaling ratio of tissue pixel importance based on the occlusion frequency, and assigns a higher sampling coefficient to areas with higher instrument occlusion frequency to achieve realistic restoration of tissue in occluded areas.

[0101] Step 105: Based on the depth confidence map, dynamic Gaussian sampling is used to perform point sampling on each ray; concentrated sampling is performed in areas with high confidence, and uniform sampling is performed in areas with low confidence.

[0102] The sampling points of each ray are represented as Gaussian pulses according to the confidence weight allocation function.

[0103] For example, the sampling points of each ray are represented as Gaussian pulses g(u,v,s,i) according to the confidence weighting function:

[0104]

[0105] Where e is the natural constant; τ is the exponent; and s represents the distance of the light rays.

[0106] Step 106: Combine the coordinates of the sampling points on the light rays with the timestamps, and input them into the deformable network after Fourier encoding to predict the displacement of the sampling points of each light ray based on the light ray sampling points in the static standard frame. By combining the displacement predicted by the deformable network with the coordinates of the sampling points in the standard frame, the coordinates of all light ray sampling points in the dynamic frame are obtained.

[0107] In this step, a dynamic-static decoupling method is used to define the dynamic scene as a static standard frame and a deformation based on the static standard frame. By inputting the coordinates and timestamps of ray points into the Deformation Net via Fourier encoding, the displacement of each ray's sampling point based on the ray sampling point in the static standard frame is predicted. By combining the displacement predicted by the Deformation Net with the coordinates of the sampling points in the standard frame, the coordinates of all ray sampling points in the dynamic frame can be obtained.

[0108] In this embodiment, a deformable space and a normalized space are used to describe the dynamic surgical scene. The normalized space is represented by a signed distance function and a spherical harmonic function, describing a rigid neural radiation field. It can be viewed as a deformation of the temporal field for each frame. Specifically, a spatiotemporal network Ψ is optimized. t To estimate the deformation between the scene in the deformation space at a specific time point and the scene in the normal space.

[0109] For example, given a spatial point and time point t, Representing three dimensions; training the deformable network Ψ t The displacement Δx of each sampling point is output, and the position and time information are mapped using position encoding. The position and time information are then input into the deformable network to obtain:

[0110] (x+Δx)=x+Ψ t ;

[0111] (x+Δx) represents the coordinates of the sampled point after deformation.

[0112] Decoupling deformation through deformation space, the deformation of each frame is treated as the displacement of ray sampling points starting from the first frame; then, all deformation points are fed together into the normal space to encode color and volume density. The camera viewpoint is redundant, so individual input coordinates are sufficient to reasonably predict the displacement of ray sampling points.

[0113] Step 107: Input the coordinates of all light sampling points into the SDF network to complete the mapping between the sampling point coordinates and the volume density, and obtain the distance from the sampling point to the target surface and the gradient of the SDF network.

[0114] The signal distance field represents the distance from a point to the surface of an object, where the sign indicates whether the point is inside or outside the object.

[0115] For example, construct the expression for the zero level set S of the signal distance function of the sampling points:

[0116]

[0117] in, This represents the signal distance function.

[0118] Map the deformed coordinates of the sampling points to s, f, and n:

[0119]

[0120] in, Represents a dimension; ω represents the number of dimensions; f represents the features of the object's surface; n is the gradient of the SDF network ξ(·).

[0121] Step 108: Input the distance from the sampling point to the target surface and the gradient of the SDF network into the spherical harmonic function to obtain the RGB value of each sampling point.

[0122] Spherical harmonic functions constitute the set of orthogonal basis functions describing functions on a sphere. Spherical harmonic coefficients, used as coefficients in function expansions, can be understood as a low-dimensional representation of spherical functions.

[0123] For example, construct the expression for the spherical harmonic function:

[0124]

[0125] Let represent a spherical harmonic function; θ is the polar angle in spherical coordinates; φ is the azimuth angle in spherical coordinates; L is a natural number; m is 0 or 1; π is pi. Represents Legendre polynomials;

[0126] Construct the color expression for the sampling point:

[0127]

[0128] Where c(·) represents the color of the sampling point; Sig represents the sigmoid function; This represents the spherical harmonic coefficients.

[0129] Calculate the RGB value of each pixel using the following formula:

[0130]

[0131] C(r) is the RGB value of each pixel; near is the nearest point of the ray; far is the farthest point of the ray; w(s) is the weight of each sampling point; c(D,k) represents the color of each sampling point; ρ(·) represents the opacity function corresponding to the volume density; 0≤u≤s; r(s) represents the ray; γ is the probability density function.

[0132] Step 109: Render the sampling points into pixel colors along the light rays, and optimize the deformation network, SDF network, and spherical harmonic function based on the luminance difference between the pixel colors and the real RGB colors. Calculate the surface gradient and divergence of the sampling points, and use Eikonalloss and Div Loss to make the SDF network converge in order to render the correct scene depth information.

[0133] To train the entire network, the goal is to minimize the color difference between the rendered color and the true ground color without any 3D information supervision. Sufficient constraints are also imposed on the neural SDF field to optimally reconstruct the object surface information under specified conditions.

[0134] Using only the gradient constraint of the SDF can lead to unstable convergence of the loss function. This phenomenon often stems from adjusting the SDF gradient to be close to one in the Enochor loss, resulting in infinite solutions and loss oscillations during network training. To address this issue, the Laplacian operator of the SDF is introduced, which penalizes the second derivative along the normal direction of the 3D surgical scene while preserving more visual detail. Adding this term to the loss function helps the network converge to a sticky solution, thereby increasing the overall stability of the model.

[0135] For example, construct the color loss expression:

[0136]

[0137] Among them, L color Indicates color loss; This indicates the color to be rendered.

[0138] Construct the Eikonal loss expression:

[0139]

[0140] Among them, L eikonal For Eikonal's loss; Let x represent the gradient at point x; ||||2 is the 2-norm.

[0141] Construct the Div loss expression:

[0142]

[0143] Among them, L div For Div loss; T represents; H(·) represents the Hessian matrix.

[0144] The loss of the entire model is calculated using the following formula:

[0145] L total =L color +λ1L eikonal +λ2L div ;

[0146] Where λ1 is the first equilibrium hyperparameter and λ2 is the second equilibrium hyperparameter.

[0147] This embodiment uses four medical datasets to evaluate the performance of Endo-4SRF and compare it with existing technologies. The medical datasets include two public datasets (EndoNeRF and EndoVis) and two internal datasets (cardiac atrial fibrillation ablation and abdominal lymph node dissection). All video sequences were normalized to a resolution of 512×640 pixels. This embodiment uses four evaluation metrics for quantitative comparison: PSNR, SSIM, LPIPS, and FLIP to evaluate the quality of reconstructed images within tissue regions not obscured by the instrument. Detailed results are as follows... Figure 3 As shown, typical 3D results are displayed Figure 4 The quantitative results show that the dynamic three-dimensional reconstruction method for soft tissue under monocular endoscopy provided in this embodiment achieves good performance under most evaluation metrics.

[0148] Figure 5 and Figure 6 The quantitative results of the global rendering are displayed, showing RGB images with normal maps and depth maps. The dynamic 3D reconstruction method for soft tissue under monocular endoscopy provided in this embodiment exhibits realistic effects at the image rendering level. Furthermore, by combining a dynamic Gaussian sampling module based on depth confidence, the scheme in this embodiment performs well in terms of 3D perception of tissues and can render accurate depth maps without depth supervision.

[0149] Furthermore, the experimental results show the normal map of the tissue surface. Figure 7This embodiment demonstrates that the solution can reconstruct a relatively smooth tissue surface by utilizing SDF functions and additional second-order constraints.

[0150] The solution in this embodiment effectively perceives the surface texture information of exposed tissue and demonstrates a stronger ability to recover tissue texture information obscured by the instrument. For example... Figure 8 As shown, the texture details of the rendered images are compared, demonstrating the superior rendering capabilities of the solution in this embodiment.

[0151] like Figure 9 and Figure 10 As shown, this demonstrates the powerful performance of the Endo-4SRF in clinical settings, enabling the reconstruction of dynamic tissue images using monocular endoscopic video sequences without depth supervision, achieving instrument stealth. This demonstrates the potential of the Endo-4SRF in enhancing intraoperative image-guided robotic navigation and postoperative surgical training intelligence.

[0152] The dynamic 3D reconstruction method for soft tissue under monocular endoscopy provided in this embodiment uses a confidence map calculated from depth priors to comprehensively evaluate and correct the depth, thereby solving the problem of geometric radiation blur. By utilizing the concept of surface rendering, a density field characterized by a signed distance function and a color field represented by a spherical harmonic function are used. By optimizing the loss function, accurate depth prediction and efficient image rendering can be achieved, restoring more detailed scene details and obtaining accurate dynamic 3D reconstruction of soft tissue.

[0153] Example 2

[0154] Based on the same inventive concept as Embodiment 1, this embodiment provides a dynamic three-dimensional reconstruction system for soft tissue under monocular endoscopy. Since the principle of this system in solving the problem is similar to the aforementioned dynamic three-dimensional reconstruction method for soft tissue under monocular endoscopy, the implementation of this system can refer to the implementation of the dynamic three-dimensional reconstruction method for soft tissue under monocular endoscopy.

[0155] like Figure 11 As shown, a dynamic three-dimensional reconstruction system for soft tissue under monocular endoscopy includes:

[0156] Training module 10 is used to train the neural radiation field using endoscopic RGB images to obtain the neural depth prior of the target scene during surgery.

[0157] The depth prediction map generation module 20 is used to input the endoscopic RGB image into the trained depth estimation network and output the depth prediction map of each RGB image.

[0158] The confidence determination module 30 is used to determine the confidence of each pixel in the target RGB image based on the neural depth prior of the target scene and the depth prediction map of the target RGB image, and obtain a depth confidence map.

[0159] Module 40 is used to construct a neural radiation field space and emit light rays to the target RGB image to restore the dynamic surgical scene.

[0160] The sampling module 50 is used to perform dynamic Gaussian sampling based on the depth confidence map, and to perform point sampling on each ray; wherein concentrated sampling is performed in areas with high confidence, and uniform sampling is performed in areas with low confidence.

[0161] The coordinate generation module 60 is used to combine the coordinates of the sampling points on the light rays with the timestamps, and input them into the deformable network after Fourier encoding to predict the displacement of the sampling points of each light ray based on the light ray sampling points in the static standard frame. The coordinates of all light ray sampling points in the dynamic frame are obtained by combining the displacement predicted by the deformable network with the coordinates of the sampling points in the standard frame.

[0162] The mapping module 70 is used to input the coordinates of all light sampling points into the SDF network to complete the mapping between the sampling point coordinates and the volume density, and to obtain the distance from the sampling point to the target surface and the gradient of the SDF network.

[0163] The color generation module 80 is used to input the distance from the sampling point to the target surface and the gradient of the SDF network into the spherical harmonic function to obtain the RGB value of each sampling point.

[0164] The scene generation module 90 is used to render the sampling points into pixel colors along the light rays, and optimize the deformation network, SDF network and spherical harmonic function according to the luminance difference between the pixel colors and the real RGB colors. It calculates the surface gradient and divergence of the sampling points, and makes the SDF network converge through Eikonal loss and Div Loss to render the correct scene depth information.

[0165] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0166] Example 3

[0167] This embodiment provides a computer device, including a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the steps of the dynamic three-dimensional reconstruction method of soft tissue under monocular endoscopy described in Embodiment 1.

[0168] For a more detailed explanation of the above method, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0169] Example 4

[0170] This embodiment provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the dynamic three-dimensional reconstruction method of soft tissue under monocular endoscopy described in Embodiment 1.

[0171] For a more detailed explanation of the above method, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0172] Example 5

[0173] This embodiment provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the steps of the dynamic three-dimensional reconstruction method of soft tissue under monocular endoscopy described in Embodiment 1.

[0174] For a more detailed explanation of the above method, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0175] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems, devices, storage media, and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0176] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0177] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0178] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0179] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0180] The present invention has been described in detail above with reference to specific embodiments and exemplary examples; however, these descriptions should not be construed as limiting the present invention. Those skilled in the art will understand that various equivalent substitutions, modifications, or improvements can be made to the technical solutions and embodiments of the present invention without departing from the spirit and scope of the invention, and all such modifications and improvements fall within the scope of the present invention. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy, characterized in that, include: The neural radiation field was trained using endoscopic RGB images to obtain the neural depth prior of the target scene during surgery; The RGB images from the endoscope are input into the trained depth estimation network, and the output is a depth prediction map for each RGB image. Based on the neural depth prior of the target scene and the depth prediction map of the target RGB image, the confidence level of each pixel in the target RGB image is determined, and a depth confidence map is obtained. A neural radiation field space is constructed, and light rays are emitted into the target RGB image to reconstruct the dynamic surgical scene; Dynamic Gaussian sampling is used based on the depth confidence map to perform point sampling on each ray; In this method, concentrated sampling is performed in areas with high confidence, while uniform sampling is performed in areas with low confidence. The coordinates of the sampling points on the light rays are combined with the timestamps, and then input into the deformable network after Fourier encoding to predict the displacement of the sampling points of each light ray based on the light ray sampling points in the static standard frame. The coordinates of all light ray sampling points in the dynamic frame are obtained by combining the displacement predicted by the deformable network with the coordinates of the sampling points in the standard frame. Input the coordinates of all light sampling points into the SDF network to complete the mapping between the sampling point coordinates and the volume density, and obtain the distance from the sampling point to the target surface and the gradient of the SDF network; The distance from the sampling point to the target surface and the gradient of the SDF network are input into the spherical harmonic function to obtain the RGB value of each sampling point; The sampling points are rendered as pixel colors along the light rays, and the deformation network, SDF network, and spherical harmonic function are optimized based on the luminance difference between the pixel colors and the real RGB colors. The surface gradient and divergence of the sampling points are calculated, and the SDF network is converged through Eikonal loss and DivLoss to render the correct scene depth information.

2. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 1, characterized in that, The step of determining the confidence level of each pixel in the target RGB image based on the neural depth prior of the target scene and the depth prediction map of the target RGB image to obtain a depth confidence map includes: Construct the confidence weight function for pixel (u,v) in the i-th frame of the RGB image: Where σ(·) is the confidence weight function; The neural depth prior for pixel (u,v); is the depth prediction map for pixel (u,v); μ is the first truncation threshold; δ is the second truncation threshold.

3. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 1, characterized in that, The process of constructing a neural radiation field space and emitting light rays onto the target RGB image to recreate the dynamic surgical scene includes: Construct an importance sampling expression based on binary device mask: Among them, J i Γ represents mask importance; M represents scaling factor; Γ represents the importance of the mask. i The binary ray filtering mask for the i-th frame of the RGB image; Represents pixel-wise multiplication; T is the total number of frames in the RGB image sequence; |·| F M represents the Frobenius norm; j This is the binary ray filtering mask for the j-th frame of the RGB image.

4. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 2, characterized in that, The depth confidence map-based method employs dynamic Gaussian sampling, performing point sampling on each ray, including: The sampling points of each ray are represented as Gaussian pulses g(u,v,s,i) according to the confidence weighting function: Where e is the natural constant; τ is the exponent; and s represents the distance of the light rays.

5. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 4, characterized in that, The process involves combining the coordinates of the sampling points on the light rays with timestamps, inputting them into a deformable network after Fourier encoding, to predict the displacement of each light ray's sampling point based on the light ray sampling points in the static standard frame. The displacement predicted by the deformable network is then combined with the coordinates of the sampling points in the standard frame to obtain the coordinates of all light ray sampling points in the dynamic frame. This includes: Given a spatial point and time point t, Representing three dimensions; training the deformable network Ψ t The displacement Δx of each sampling point is output, and the position and time information are mapped using position encoding. The position and time information are then input into the deformable network to obtain: (x+Δx)=x+Ψ t ; (x+Δx) represents the coordinates of the sampled point after deformation.

6. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 5, characterized in that, The process of inputting the coordinates of all light sampling points into the SDF network to map the sampling point coordinates to the volume density, and obtaining the distance from the sampling point to the target surface and the gradient of the SDF network, includes: Construct the expression for the zero level set S of the signal distance function of the sampling points: in, Represents the signal distance function; Map the deformed coordinates of the sampling points to s, f, and n: in, Represents a dimension; ω represents the number of dimensions; f represents the features of the object's surface; n is the gradient of the SDF network ξ(·).

7. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 6, characterized in that, The step of inputting the distance from the sampling point to the target surface and the gradient of the SDF network into the spherical harmonic function to obtain the RGB value of each sampling point includes: Construct the expression for the spherical harmonic function: Let represent a spherical harmonic function; θ is the polar angle in spherical coordinates; φ is the azimuth angle in spherical coordinates; L is a natural number; m is 0 or 1; π is pi. Represents Legendre polynomials; Construct the color expression for the sampling point: Where c(·) represents the color of the sampling point; Sig represents the sigmoid function, used to obtain the normalized color; Represents the spherical harmonic coefficients; Calculate the RGB value of each pixel using the following formula: C(r) is the RGB value of each pixel; near is the nearest point of the ray; far is the farthest point of the ray; w(s) is the weight of each sampling point; c(D,k) represents the color of each sampling point; ρ(·) represents the opacity function corresponding to the volume density; 0≤u≤s; r(s) represents the ray; γ is the probability density function.

8. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 7, characterized in that, The process involves rendering sampling points into pixel colors along the light rays, optimizing the deformation network, SDF network, and spherical harmonic function based on the luminance difference between the pixel colors and the true RGB colors, calculating the surface gradient and divergence of the sampling points, and using Eikonal loss and Div Loss to converge the SDF network to render the correct scene depth information. This includes: Construct the color loss expression: Among them, L color Indicates color loss; Indicates the rendered color; Construct the Eikonal loss expression: Among them, L eikonal For Eikonal's loss; This represents the gradient at point x; ||||2 is the 2-norm; Construct the Div loss expression: Among them, L div For Div loss; T represents; H(·) represents the Hessian matrix; The loss of the entire model is calculated using the following formula: L total =L color +λ1L eikonal +λ2L div ; Where λ1 is the first equilibrium hyperparameter and λ2 is the second equilibrium hyperparameter.

9. A computer device, characterized in that, It includes a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; when executed by a processor, the computer programs implement the steps of the dynamic three-dimensional reconstruction method for soft tissue under monocular endoscopy as described in any one of claims 1-8.

Images