Dynamic three-dimensional reconstruction method for soft tissue under monocular endoscope, computer equipment and storage medium

Abstract

The invention provides a dynamic three-dimensional reconstruction method for soft tissue under a monocular endoscope, computer equipment and a storage medium, and belongs to the technical field of image processing. According to the method, the depth is comprehensively evaluated and corrected by using a confidence map calculated from depth prior, so that the problem of geometric radiation blur is solved; according to the method, the concept of surface rendering is utilized, a density field represented by a signed distance function and a color field represented by a spherical harmonic function are adopted, and a loss function is optimized, so that accurate depth prediction and efficient image rendering can be realized, finer scene details can be recovered, and accurate dynamic 3D reconstruction of soft tissues can be obtained.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a dynamic three-dimensional reconstruction method of soft tissue under a monocular endoscope, a computer device and a storage medium. Background Art

[0002] Endoscopic minimally invasive surgery has become the main surgical procedure for soft tissue organ treatment, such as partial resection of kidney tumors, partial resection of liver tumors, pancreaticoduodenectomy, etc. However, the limited environmental perception space and operation time limit of endoscopic minimally invasive surgery make the surgery very demanding on the doctor's experience and skills. Augmented reality surgical navigation can effectively alleviate the problem of limited environmental perception of doctors during surgery and provide surgical guidance for doctors, which has become a research hotspot in recent years.

[0003] In the existing technology, augmented reality surgical navigation technology based on video superposition is the main form of augmented reality navigation for endoscopic minimally invasive surgery. The main implementation method is to reconstruct the anatomical structure model of the target tissue and organ from the patient's CT / MRI medical images, realize the registration of the anatomical structure model with the intraoperative endoscopic image through image registration technology, and finally complete the enhanced display of the intraoperative scene through augmented reality display technology to achieve image guidance. Among them, registration can be roughly divided into two methods: point-based registration and surface-based registration.

[0004] However, the current problem is that the surgical navigation system based on point registration often requires manual intervention, which interrupts the normal surgical process, distracts the doctor's attention and increases the risk of complications. The registration method of implanted markers is invasive to the human body, difficult to apply clinically, and the system often requires additional tracking equipment; the surgical navigation method based on surface registration requires intraoperative tissue morphology measurement to obtain the surface information of the target tissue during surgery, and the complex intraoperative endoscopic imaging environment (smoke, reflections, etc.) makes it difficult to obtain dense reconstruction results of the target tissue organs during surgery. At the same time, due to the limited intraoperative endoscopic field of view, the target tissue surface reconstruction area is small, and automatic registration is difficult to achieve. Most of the currently disclosed endoscopic augmented reality navigation systems are completed by manual registration or semi-automatic registration, among which the semi-automatic registration system refers to a navigation system that requires manual intervention to complete the initial registration. In addition, due to the particularity of the abdominal cavity, soft tissue organs are deformed during surgery, and the surgical navigation systems in the related art mostly ignore the deformation of soft tissues. The surgical navigation system provided cannot compensate for the deformation of soft tissues, resulting in poor navigation accuracy. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method for dynamic three-dimensional reconstruction of soft tissue under a monocular endoscope, a computer device and a storage medium.

[0006] In a first aspect, the present invention provides a method for dynamic three-dimensional reconstruction of soft tissue under a monocular endoscope, comprising:

[0007] Using endoscopic RGB images to train neural radiation fields, we obtain neural depth priors for target scenes during surgery.

[0008] Input the endoscope RGB image into the trained depth estimation network, and output the depth prediction map of each RGB image;

[0009] According to the neural depth prior of the target scene and the depth prediction map of the target RGB image, the confidence of each pixel in the target RGB image is determined to obtain a depth confidence map;

[0010] Construct the neural radiation field space and emit rays to the target RGB image to restore the dynamic surgical scene;

[0011] Based on the depth confidence map, dynamic Gaussian sampling is used to perform point sampling on each ray. Concentrated sampling is performed in areas with high confidence, and uniform sampling is performed in areas with low confidence.

[0012] The coordinates of the sampling points on the light are combined with the timestamp and input into the deformation network after Fourier encoding to predict the displacement of the sampling points of each light based on the light sampling points in the static standard frame. The displacement predicted by the deformation network is combined with the coordinates of the sampling points in the standard frame to obtain the coordinates of all light sampling points in the dynamic frame.

[0013] The coordinates of all light sampling points are input into the SDF network to complete the mapping between the sampling point coordinates and the volume density, and the distance from the sampling point to the target surface and the gradient of the SDF network are obtained;

[0014] The distance from the sampling point to the target surface and the gradient of the SDF network are input into the spherical harmonic function to obtain the RGB value of each sampling point;

[0015] The sampling points are rendered into pixel colors along the light rays, and the deformation network, SDF network and spherical harmonics are optimized according to the photometric difference between the pixel color and the true RGB color. The surface gradient and divergence of the sampling points are calculated, and the SDF network is converged through Eikonal loss and DivLoss to render the correct scene depth information.

[0016] Optionally, determining the confidence of each pixel in the target RGB image according to the neural depth prior of the target scene and the depth prediction map of the target RGB image to obtain the depth confidence map includes:

[0017] Construct the confidence weight function of the pixel (u, v) in the i-th frame RGB image:

[0018]

[0019] Among them, σ(·) is the confidence weight function; is the neural depth prior for pixel (u,v); is the depth prediction map of pixel (u, v); μ is the first truncation threshold; δ is the second truncation threshold.

[0020] Optionally, constructing a neural radiation field space and emitting light to a target RGB image to restore a dynamic surgical scene includes:

[0021] Construct an importance sampling expression based on a binary instrumentation mask:

[0022]

[0023] Among them, J i is the importance of the mask; Γ is the scaling factor; M i is the binary light filtering mask of the i-th frame RGB image; represents pixel-by-pixel multiplication; T is the total number of frames in the RGB image sequence; |·| F represents the Frobenius norm; M j is the binary ray filtering mask of the j-th RGB image.

[0024] Optionally, the step of adopting dynamic Gaussian sampling based on the depth confidence map and performing point sampling on each ray includes:

[0025] The sampling point of each ray is represented as a Gaussian pulse g(u,v,s,i) according to the confidence weight function:

[0026]

[0027] Among them, e is a natural constant; τ is an exponent; and s represents the distance of the light.

[0028] Optionally, the coordinates of the sampling points on the light are combined with the timestamp, and are input into the deformation network through Fourier encoding to predict the displacement of the sampling points of each light based on the light sampling points in the static standard frame, and the displacement predicted by the deformation network is combined with the coordinates of the sampling points in the standard frame to obtain the coordinates of all light sampling points in the dynamic frame, including:

[0029] Given a point in space and time point t, Represent three dimensions; train the deformation network Ψ t , output the displacement Δx of each sampling point, use the position encoding to map the position and time information, and input the position and time information into the deformation network to obtain:

[0030] (x+Δx)=x+Ψ t ;

[0031] (x+Δx) is the coordinate of the sampling point after deformation.

[0032] Optionally, the coordinates of all light sampling points are input into the SDF network to complete the mapping of the sampling point coordinates and the volume density, and obtain the distance from the sampling point to the target surface and the gradient of the SDF network, including:

[0033] Construct the zero level set S expression of the signal distance function of the sampling point:

[0034]

[0035] in, represents the signal distance function;

[0036] Map the deformed coordinates of the sampling points to s, f, and n:

[0037]

[0038] in, Represents a dimension; represents ω dimensions; f represents the characteristics of the object surface; n is the gradient of the SDF network ξ(·).

[0039] Optionally, the distance from the sampling point to the target surface and the gradient of the SDF network are input into the spherical harmonic function to obtain the RGB value of each sampling point, including:

[0040] Construct the spherical harmonics expression:

[0041]

[0042] represents spherical harmonic function; θ is the polar angle in the spherical coordinate system; φ is the azimuth angle in the spherical coordinate system; L is a natural number; m is 0 or 1; π is pi; represents Legendre polynomials;

[0043] Construct the sampling point color expression:

[0044]

[0045] Where c(·) represents the color of the sampling point; Sig represents the sigmoid function; represents the spherical harmonic coefficients;

[0046] The RGB value of each pixel is calculated according to the following formula:

[0047]

[0048] C(r) is the RGB value of each pixel; near is the nearest point of the light; far is the farthest point of the light; w(s) is the weight of each sampling point; c(D,k) represents the color of each sampling point; ρ(·) represents the opacity function corresponding to the volume density; 0≤u≤s; r(s) represents the light; γ is the probability density function.

[0049] Optionally, the sampling points are rendered into pixel colors along the light rays, and the deformation network, SDF network and spherical harmonics are optimized according to the photometric difference between the pixel colors and the real RGB colors, the surface gradient and divergence of the sampling points are calculated, and the SDF network is converged by Eikonalloss and Div Loss to render the correct scene depth information, including:

[0050] Construct the color loss expression:

[0051]

[0052] Among them, L color Indicates color loss; Indicates the rendering color;

[0053] Construct the Eikonal loss expression:

[0054]

[0055] Among them, L eikonal for Eikonal losses; Represents the gradient at point x; ||||2 is the 2-norm;

[0056] Construct the Div loss expression:

[0057]

[0058] Among them, L div is Div loss; T represents; H(·) represents Hessian matrix;

[0059] The loss of the entire model is calculated according to the following formula:

[0060] L total =L color +λ1L eikonal +λ2L div ;

[0061] Among them, λ1 is the first balancing hyperparameter; λ2 is the second balancing hyperparameter.

[0062] In a second aspect, the present invention provides a computer device comprising a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the steps of the method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy described in the first aspect are implemented.

[0063] In a third aspect, the present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy described in the first aspect are implemented.

[0064] The present invention provides a method, computer device and storage medium for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy. In the method, a confidence map calculated from a depth prior is used to comprehensively evaluate and correct the depth, thereby solving the problem of geometric radiation blur; by using the concept of surface rendering, a density field represented by a signed distance function and a color field represented by a spherical harmonic function are adopted, and by optimizing the loss function, accurate depth prediction and efficient image rendering can be achieved, more detailed scene details can be restored, and accurate dynamic 3D reconstruction of soft tissue can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A schematic diagram of a flow chart of a method for dynamic three-dimensional reconstruction of soft tissue under a monocular endoscope provided by an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of a light point sampling method based on Gaussian pulses provided in an embodiment of the present invention;

[0068] Figure 3 A quantitative data comparison chart of the image rendering quality of the embodiment of the present invention compared with the current SOTA method provided for the embodiment of the present invention;

[0069] Figure 4 A qualitative comparison diagram of the quality of three-dimensional reconstruction of tissues provided by the embodiment of the present invention compared with the current SOTA method;

[0070] Figure 5 A qualitative comparison diagram of the present invention in terms of image rendering, surface restoration and depth restoration compared to the current surface reconstruction method EndoSurf provided for the present invention;

[0071] Figure 6Another qualitative comparison diagram of the present invention provided in the embodiment of the present invention compared with the current surface reconstruction method EndoSurf in image rendering, surface restoration and depth restoration;

[0072] Figure 7 A graph showing ablation experiment results of a loss function provided by an embodiment of the present invention;

[0073] Figure 8 A qualitative comparison diagram of the quality of restoring occluded tissue textures provided by the embodiment of the present invention compared to the current SOTA method;

[0074] Fig. 9 A schematic diagram of a monocular laparoscope global dynamic scene reconstruction system for intraoperative navigation and postoperative medical skills training provided by an embodiment of the present invention;

[0075] Fig.10 A schematic diagram of an invisible device provided by an embodiment of the present invention;

[0076] Fig.11 A structural schematic diagram of a dynamic three-dimensional reconstruction system of soft tissue under a monocular endoscope provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] Given a video clip of a surgical scene, the goal is to make the instrument invisible and reconstruct the texture and geometry of the tissue surface, thereby restoring a clear and dynamic surgical field based only on the RGB image. This process is represented as Where i represents the frame index in the image sequence, T represents the total number of frames; I i Represents the RGB image of the i-th frame; M i It is a binary light filtering mask corresponding to each RGB frame, which aims to achieve the invisibility of smoke, blood stains and instruments in the video. i Provides the transformation matrix from the world coordinate system to the image plane coordinate system. In the case of a single view, the pose matrix remains consistent in each frame. Variable t i Represents the timestamp of each frame, and the time is normalized by i / T.

[0079] Example 1

[0080] like Figure 1As shown, this embodiment provides a method for dynamic three-dimensional reconstruction of soft tissue under a monocular endoscope, comprising:

[0081] Step 101, using the endoscope RGB image to train the neural radiation field, to obtain the neural depth prior of the target scene during surgery.

[0082] During the training phase, the key supervisory signal comes from deformation-based view synthesis. Once the depth value of each pixel in the target frame is estimated, the pixel coordinates are back-projected into the 3D camera space using the known camera intrinsic parameters. Using the estimated self-motion trajectory, the 3D point cloud can be projected onto another image plane. Given the source image and the target image, view synthesis can be expressed as:

[0083]

[0084] H(I s ) and H(I t ) represent the homogeneous expressions of the image coordinates of the source view and the target view respectively; K represents the camera intrinsic parameter matrix after calibration; C t→s represents the estimated ego-motion from the target view to the source view; D t Represents the predicted depth map of the target frame. Therefore, the photometric appearance loss can be smoothly derived from the deformation of the source view frame to the target view frame:

[0085] L s→t (I) = T(I s )-T(I t );

[0086] T(·) represents a differentiable inverse transformer through which gradients can be efficiently back-propagated to supervise the entire self-supervised framework.

[0087] Step 102: Input the endoscope RGB image into the trained depth estimation network, and output a depth prediction map of each RGB image.

[0088] Step 103, according to the neural depth prior of the target scene and the depth prediction map of the target RGB image, the confidence of each pixel in the target RGB image is determined to obtain a depth confidence map.

[0089] In this embodiment, Figure 2 As shown, the depth prior is combined with the biased depth to guide the sampling of rays in the neural radiation field by determining a depth-adaptive confidence. Specifically, the depth prior is evaluated and scored based on the biased depth map, and higher confidence is assigned to rays that show greater similarity, while lower confidence is assigned to rays that show less similarity.

[0090] Construct the confidence weight function of the pixel (u, v) in the i-th frame RGB image:

[0091]

[0092] Among them, σ(·) is the confidence weight function; is the neural depth prior for pixel (u,v); is the depth prediction map of pixel (u, v); μ is the first truncation threshold; δ is the second truncation threshold.

[0093] The sampling points are concentrated near the depth prior on rays with higher confidence, while the depth is sampled discretely on rays with lower confidence, which helps with depth correction.

[0094] Step 104, constructing a neural radiation field space and emitting light toward the target RGB image to restore the dynamic surgical scene.

[0095] In this step, a ray is projected for each pixel, and the model is r(s)=o+sd, where o is the ray starting point (ie, the camera center), d is the ray direction, and s represents the ray distance.

[0096] The purpose of this embodiment is to reconstruct the whole tissue under laparoscopy and restore the tissue in the area blocked by the instrument. Therefore, it is necessary to avoid projecting light to the pixels of the entire image to eliminate the surgical instrument. Specifically, the surgical instrument binary segmentation mask initializes the pixel value of the instrument part to 1 and the pixel value of the tissue part to 0.

[0097] Exemplarily, construct an importance sampling expression based on a binary instrument mask:

[0098]

[0099] Among them, J i is the importance of the mask; Γ is the scaling factor; M i is the binary light filtering mask of the i-th frame RGB image; represents pixel-by-pixel multiplication; T is the total number of frames in the RGB image sequence; |·| F represents the Frobenius norm; M j is the binary ray filtering mask of the j-th RGB image.

[0100] (1-M i ) ensures that the importance of the instrument part of the light projection is always 0, and realizes effective instrument light filtering. The sampling scaling factor Γ uses the Frobenius norm to assign a scaling ratio of the importance of tissue pixels according to the occlusion frequency, and assigns a higher sampling coefficient to the area with a higher instrument occlusion frequency to achieve realistic restoration of the occluded area tissue.

[0101] Step 105 , using dynamic Gaussian sampling based on the depth confidence map, performs point sampling on each ray; concentrated sampling is performed in areas with high confidence, and uniform sampling is performed in areas with low confidence.

[0102] The sampling points of each ray are expressed as Gaussian pulses according to a confidence weight assignment function.

[0103] Exemplarily, the sampling point of each ray is represented as a Gaussian pulse g(u,v,s,i) according to the confidence weight function:

[0104]

[0105] Among them, e is a natural constant; τ is an exponent; and s represents the distance of the light.

[0106] Step 106, the coordinates of the sampling points on the light are combined with the timestamp, and input into the deformation network after Fourier encoding to predict the displacement of the sampling points of each light ray based on the light ray sampling points in the static standard frame. The displacement predicted by the deformation network is combined with the coordinates of the sampling points in the standard frame to obtain the coordinates of all light ray sampling points in the dynamic frame.

[0107] In this step, the dynamic scene is defined as a static standard frame and a deformation based on the static standard frame by using the method of dynamic and static decoupling. The coordinates of the light points are combined with the timestamp through Fourier encoding and input into the deformation network (Deformation Net), and the displacement of each light sampling point based on the light sampling point in the static standard frame is predicted. The displacement predicted by the deformation network is combined with the coordinates of the sampling points in the standard frame to obtain the coordinates of all light sampling points in the dynamic frame.

[0108] In this embodiment, a deformation space and a standard space are used to describe the dynamic surgical scene, where the standard space is represented by a signed distance function and spherical harmonics, describing a rigid neural radiation field. It can be regarded as the deformation of the time field in each frame. Specifically, a spatiotemporal network Ψ is optimized. t To estimate the deformation between the scene at a specific time point in the deformation space and the scene in the canonical space.

[0109] For example, given a spatial point and time point t, Represent three dimensions; train the deformation network Ψ t , output the displacement Δx of each sampling point, use the position encoding to map the position and time information, and input the position and time information into the deformation network to obtain:

[0110] (x+Δx)=x+Ψ t ;

[0111] (x+Δx) is the coordinate of the sampling point after deformation.

[0112] I decouple the deformations through the deformation space, treating the deformation of each frame as the displacement of the ray sampling points from the first frame; I then feed all the deformed points together into the canonical space to encode the color and volume density. The camera viewpoint is redundant, so the input coordinates alone are sufficient to reasonably predict the displacement of the ray sampling points.

[0113] Step 107, input the coordinates of all light sampling points into the SDF network to complete the mapping between the sampling point coordinates and the volume density, and obtain the distance from the sampling point to the target surface and the gradient of the SDF network.

[0114] The signal distance field represents the distance from a point to the surface of an object, where the sign indicates whether the point is inside or outside the object.

[0115] Exemplarily, the zero level set S expression of the signal distance function of the sampling point is constructed:

[0116]

[0117] in, Represents the signal distance function.

[0118] Map the deformed coordinates of the sampling points to s, f, and n:

[0119]

[0120] in, Represents a dimension; represents ω dimensions; f represents the characteristics of the object surface; n is the gradient of the SDF network ξ(·).

[0121] Step 108: Input the distance from the sampling point to the target surface and the gradient of the SDF network into the spherical harmonic function to obtain the RGB value of each sampling point.

[0122] Spherical harmonics constitute a set of orthogonal basis functions that describe functions on the sphere. Spherical harmonic coefficients are used as coefficients of function expansion and can be understood as a low-dimensional representation of spherical functions.

[0123] For example, construct the spherical harmonic function expression:

[0124]

[0125] represents spherical harmonic function; θ is the polar angle in the spherical coordinate system; φ is the azimuth angle in the spherical coordinate system; L is a natural number; m is 0 or 1; π is pi; represents Legendre polynomials;

[0126] Construct the sampling point color expression:

[0127]

[0128] Where c(·) represents the color of the sampling point; Sig represents the sigmoid function; represents the spherical harmonic coefficients.

[0129] The RGB value of each pixel is calculated according to the following formula:

[0130]

[0131] C(r) is the RGB value of each pixel; near is the nearest point of the light; far is the farthest point of the light; w(s) is the weight of each sampling point; c(D,k) represents the color of each sampling point; ρ(·) represents the opacity function corresponding to the volume density; 0≤u≤s; r(s) represents the light; γ is the probability density function.

[0132] Step 109, render the sampling points into pixel colors along the light rays, and optimize the deformation network, SDF network and spherical harmonics according to the photometric difference between the pixel colors and the real RGB colors, calculate the surface gradient and divergence of the sampling points, and make the SDF network converge through Eikonalloss and Div Loss to render and obtain the correct scene depth information.

[0133] To train the entire network, the goal is to minimize the color difference between the rendered color and the ground truth color without any 3D information supervision, and sufficient constraints are imposed on the neural SDF field to optimally reconstruct the object surface information under the specified conditions.

[0134] Using only the gradient constraint of the SDF may lead to unstable convergence of the loss function. This phenomenon usually comes from the EconoL loss that adjusts the SDF gradient close to one, resulting in infinite solutions and loss oscillation when training the network. To solve this problem, the Laplacian operator of the SDF is introduced, which penalizes the second-order derivatives in the normal direction of the 3D surgical scene and maintains more visual details. Adding this term to the loss function helps the network converge to a sticky solution, thereby increasing the stability of the entire model.

[0135] Exemplarily, the color loss expression is constructed:

[0136]

[0137] Among them, L color Indicates color loss; Indicates the rendering color.

[0138] Construct the Eikonal loss expression:

[0139]

[0140] Among them, L eikonal for Eikonal losses; Represents the gradient at point x; ||||2 is the 2-norm.

[0141] Construct the Div loss expression:

[0142]

[0143] Among them, L div is the Div loss; T represents; H(·) represents the Hessian matrix.

[0144] The loss of the entire model is calculated according to the following formula:

[0145] L total =L color +λ1L eikonal +λ2L div ;

[0146] Among them, λ1 is the first balancing hyperparameter; λ2 is the second balancing hyperparameter.

[0147] This embodiment uses four medical datasets to evaluate the performance of Endo-4SRF and conduct comparative experiments with existing technologies. The medical datasets include two public datasets (EndoNeRF and EndoVis) and two internal datasets (atrial fibrillation ablation and abdominal lymph node dissection). All video sequences are standardized to a resolution of 512×640 pixels. This embodiment uses four evaluation indicators for quantitative comparison, namely PSNR, SSIM, LPIPS, and FLIP to evaluate the quality of reconstructed images in tissue areas that are not occluded by the instrument. Detailed results are shown in Figure 2. Figure 3 As shown, typical 3D results are shown in Figure 4 From the quantitative results, it can be noted that the dynamic three-dimensional reconstruction method of soft tissue under monocular endoscopy provided in this embodiment achieves good performance under most evaluation indicators.

[0148] Figure 5 and Figure 6 The quantitative results of global rendering are shown, and an RGB image with a normal map and a depth map is displayed. The dynamic three-dimensional reconstruction method of soft tissue under monocular endoscopy provided in this embodiment shows realistic effects at the image rendering level. In addition, through the combination of a dynamic Gaussian sampling module based on depth confidence, the solution of this embodiment performs well in 3D perception of tissues and can render accurate depth maps without depth supervision.

[0149] In addition, the experimental results show the normal map of the tissue surface. Figure 7It is demonstrated that the scheme of this embodiment can reconstruct a relatively smooth tissue surface by utilizing the SDF function and additional second-order constraints.

[0150] The solution of this embodiment can better perceive the surface texture information of the exposed tissue and demonstrate a stronger ability to restore the texture information of the tissue blocked by the instrument. Figure 8 As shown, the texture details of the rendered images are compared, demonstrating the superior rendering capability of the solution of this embodiment.

[0151] like Fig. 9 and Fig.10 As shown in the figure, Endo-4SRF demonstrates its powerful performance in clinical situations, and is able to reconstruct dynamic tissue images using monocular endoscopic video sequences without deep supervision, making surgical instruments invisible. This shows the potential of Endo-4SRF in enhancing intraoperative image-guided robot navigation and postoperative surgical training intelligence.

[0152] The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy provided in this embodiment uses a confidence map calculated from a depth prior to comprehensively evaluate and correct the depth, thereby solving the problem of geometric radiation blur; using the concept of surface rendering, a density field represented by a signed distance function and a color field represented by a spherical harmonic function are adopted, and by optimizing the loss function, accurate depth prediction and efficient image rendering can be achieved, more detailed scene details can be restored, and accurate dynamic 3D reconstruction of soft tissue can be obtained.

[0153] Example 2

[0154] Based on the same inventive concept as Example 1, this example provides a dynamic three-dimensional reconstruction system for soft tissue under a monocular endoscope. Since the principle of solving the problem by this system is similar to the aforementioned dynamic three-dimensional reconstruction method for soft tissue under a monocular endoscope, the implementation of this system can refer to the implementation of the dynamic three-dimensional reconstruction method for soft tissue under a monocular endoscope.

[0155] like Fig.11 As shown, the dynamic three-dimensional reconstruction system of soft tissue under monocular endoscopy includes:

[0156] The training module 10 is used to train the neural radiation field using the endoscopic RGB image to obtain the neural depth prior of the target scene during surgery.

[0157] The depth prediction map generation module 20 is used to input the endoscope RGB image into the trained depth estimation network and output a depth prediction map of each RGB image.

[0158] The confidence determination module 30 is used to determine the confidence of each pixel in the target RGB image according to the neural depth prior of the target scene and the depth prediction map of the target RGB image to obtain a depth confidence map.

[0159] The construction module 40 is used to construct the neural radiation field space and emit light to the target RGB image to restore the dynamic surgical scene.

[0160] The sampling module 50 is used to perform point sampling on each ray by using dynamic Gaussian sampling based on the depth confidence map; concentrated sampling is performed in areas with high confidence, and uniform sampling is performed in areas with low confidence.

[0161] The coordinate generation module 60 is used to combine the coordinates of the sampling points on the light with the timestamp, and input them into the deformation network after Fourier encoding to predict the displacement of the sampling points of each light based on the light sampling points in the static standard frame. The displacement predicted by the deformation network is combined with the coordinates of the sampling points in the standard frame to obtain the coordinates of all light sampling points in the dynamic frame.

[0162] The mapping module 70 is used to input the coordinates of all light sampling points into the SDF network to complete the mapping between the sampling point coordinates and the volume density, and obtain the distance from the sampling point to the target surface and the gradient of the SDF network.

[0163] The color generation module 80 is used to input the distance from the sampling point to the target surface and the gradient of the SDF network into the spherical harmonic function to obtain the RGB value of each sampling point.

[0164] The scene generation module 90 is used to render the sampling points into pixel colors along the light rays, optimize the deformation network, SDF network and spherical harmonics according to the photometric difference between the pixel colors and the real RGB colors, calculate the surface gradient and divergence of the sampling points, and make the SDF network converge through Eikonal loss and Div Loss to render the correct scene depth information.

[0165] For more specific working processes of the above modules, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0166] Example 3

[0167] This embodiment provides a computer device, including a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the steps of the method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy described in Example 1 are implemented.

[0168] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0169] Example 4

[0170] This embodiment provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the method for dynamic three-dimensional reconstruction of soft tissue under a monocular endoscope described in Embodiment 1 are implemented.

[0171] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0172] Example 5

[0173] This embodiment provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the steps of the method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy described in Example 1 are implemented.

[0174] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0175] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the systems, devices, storage media, and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0176] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.

[0177] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0178] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0179] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0180] The present invention has been described in detail above in conjunction with specific implementations and exemplary examples, but these descriptions cannot be understood as limiting the present invention. Those skilled in the art understand that, without departing from the spirit and scope of the present invention, a variety of equivalent substitutions, modifications or improvements may be made to the technical solution of the present invention and its implementation methods, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy, characterized in that: include: Using endoscopic RGB images to train neural radiation fields, we obtain neural depth priors for target scenes during surgery. Input the endoscope RGB image into the trained depth estimation network, and output the depth prediction map of each RGB image; According to the neural depth prior of the target scene and the depth prediction map of the target RGB image, the confidence of each pixel in the target RGB image is determined to obtain a depth confidence map; Construct the neural radiation field space and emit rays to the target RGB image to restore the dynamic surgical scene; Based on the depth confidence map, dynamic Gaussian sampling is used to perform point sampling on each ray; The sampling is concentrated in the area with high confidence, and the sampling is uniform in the area with low confidence; The coordinates of the sampling points on the light are combined with the timestamp and input into the deformation network after Fourier encoding to predict the displacement of the sampling points of each light based on the light sampling points in the static standard frame. The displacement predicted by the deformation network is combined with the coordinates of the sampling points in the standard frame to obtain the coordinates of all light sampling points in the dynamic frame. The coordinates of all light sampling points are input into the SDF network to complete the mapping between the sampling point coordinates and the volume density, and the distance from the sampling point to the target surface and the gradient of the SDF network are obtained; The distance from the sampling point to the target surface and the gradient of the SDF network are input into the spherical harmonic function to obtain the RGB value of each sampling point; The sampling points are rendered into pixel colors along the light rays, and the deformation network, SDF network and spherical harmonics are optimized according to the photometric difference between the pixel color and the true RGB color. The surface gradient and divergence of the sampling points are calculated, and the SDF network is converged through Eikonal loss and DivLoss to render the correct scene depth information.

2. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 1, characterized in that: The method determines the confidence of each pixel in the target RGB image according to the neural depth prior of the target scene and the depth prediction map of the target RGB image to obtain the depth confidence map, including: Construct the confidence weight function of the pixel (u, v) in the i-th frame RGB image: Among them, σ(·) is the confidence weight function; is the neural depth prior for pixel (u,v); is the depth prediction map of pixel (u, v); μ is the first truncation threshold; δ is the second truncation threshold.

3. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 1, characterized in that: The method of constructing a neural radiation field space and emitting light to a target RGB image to restore a dynamic surgical scene includes: Construct an importance sampling expression based on a binary instrumentation mask: Among them, J i is the importance of the mask; Γ is the scaling factor; M i is the binary light filtering mask of the i-th frame RGB image; represents pixel-by-pixel multiplication; T is the total number of frames in the RGB image sequence; |·| F represents the Frobenius norm; M j is the binary ray filtering mask of the j-th RGB image.

4. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 2, characterized in that: The method adopts dynamic Gaussian sampling based on the depth confidence map and performs point sampling on each ray, including: The sampling point of each ray is represented as a Gaussian pulse g(u,v,s,i) according to the confidence weight function: Among them, e is a natural constant; τ is an exponent; and s represents the distance of the light.

5. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 4, characterized in that: The coordinates of the sampling points on the light are combined with the timestamp, and input into the deformation network through Fourier encoding to predict the displacement of the sampling points of each light based on the light sampling points in the static standard frame. The displacement predicted by the deformation network is combined with the coordinates of the sampling points in the standard frame to obtain the coordinates of all light sampling points in the dynamic frame, including: Given a point in space and time point t, Represent three dimensions; train the deformation network Ψ t , output the displacement Δx of each sampling point, use the position encoding to map the position and time information, and input the position and time information into the deformation network to obtain: (x+Δx)=x+Ψ t ; (x+Δx) is the coordinate of the sampling point after deformation.

6. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 5, characterized in that: The coordinates of all light sampling points are input into the SDF network to complete the mapping of the sampling point coordinates and the volume density, and obtain the distance from the sampling point to the target surface and the gradient of the SDF network, including: Construct the zero level set S expression of the signal distance function of the sampling point: in, represents the signal distance function; Map the deformed coordinates of the sampling points to s, f, and n: in, Represents a dimension; represents ω dimensions; f represents the characteristics of the object surface; n is the gradient of the SDF network ξ(·).

7. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 6, characterized in that: The distance from the sampling point to the target surface and the gradient of the SDF network are input into the spherical harmonic function to obtain the RGB value of each sampling point, including: Construct the spherical harmonics expression: represents spherical harmonic function; θ is the polar angle in the spherical coordinate system; φ is the azimuth angle in the spherical coordinate system; L is a natural number; m is 0 or 1; π is pi; represents Legendre polynomials; Construct the sampling point color expression: Where c(·) represents the color of the sampling point; Sig represents the sigmoid function, which is used to obtain the normalized color; represents the spherical harmonic coefficients; The RGB value of each pixel is calculated according to the following formula: C(r) is the RGB value of each pixel; near is the nearest point of the light; far is the farthest point of the light; w(s) is the weight of each sampling point; c(D,k) represents the color of each sampling point; ρ(·) represents the opacity function corresponding to the volume density; 0≤u≤s; r(s) represents the light; γ is the probability density function.

8. The method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy according to claim 7, characterized in that: The sampling points are rendered into pixel colors along the light rays, and the deformation network, SDF network and spherical harmonics are optimized according to the photometric difference between the pixel colors and the real RGB colors, the surface gradient and divergence of the sampling points are calculated, and the SDF network is converged through Eikonal loss and Div Loss to render the correct scene depth information, including: Construct the color loss expression: Among them, L color Indicates color loss; Indicates the rendering color; Construct the Eikonal loss expression: Among them, L eikonal for Eikonal losses; Represents the gradient at point x; ||||2 is the 2-norm; Construct the Div loss expression: Among them, L div is Div loss; T represents; H(·) represents Hessian matrix; The loss of the entire model is calculated according to the following formula: L total =L color +λ1L eikonal +λ2L div ; Among them, λ1 is the first balancing hyperparameter; λ2 is the second balancing hyperparameter.

9. A computer device, characterized in that: It comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the steps of the method for dynamic three-dimensional reconstruction of soft tissue under a monocular endoscope as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; when the computer programs are executed by the processor, the steps of the method for dynamic three-dimensional reconstruction of soft tissue under monocular endoscopy as described in any one of claims 1-8 are implemented.

Images