A method and system for 3D reconstruction of laparoscopic surgical scenes based on 3D Gaussian sputtering

Abstract

The present invention provides a method and system for three-dimensional reconstruction of laparoscopic surgical scenes based on three-dimensional Gaussian sputtering, and relates to the field of three-dimensional reconstruction of scenes. In the present invention, it is determined whether the color image frame at the current moment is a key frame, a dynamic or static scene optimizer is started, and then a posture optimizer is started, and the two optimizers are alternately iterated and run until both converge, so as to achieve simultaneous estimation of accurate camera posture and tissue deformation information. The designed three-dimensional reconstruction framework can establish a three-dimensional model of the overall environment when only the local in vivo environment and tissue deformation are captured, thereby overcoming the disadvantage of limited laparoscope field of view. In addition, depth map information that provides geometric surface information is introduced and used to supervise the center point coordinates of 3D Gaussians so that they are bound around the surface, greatly improving the three-dimensional reconstruction accuracy of the tissue surface, and the addition of the depth map provides a geometric prior for all 3D Gaussians, so that there is no need to adjust the 3D Gaussian density.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional scene reconstruction, and in particular to a three-dimensional reconstruction method and system for a laparoscopic surgery scene based on three-dimensional Gaussian sputtering. Background Art

[0002] 3D reconstruction refers to the technology of restoring the 3D shape of a scene from a 2D image, and is an important research direction in computer vision and graphics. For example, 3D reconstruction of laparoscopic surgery can be used to construct a 3D model of in vivo tissue (including the geometric structure of the tissue surface and the texture and color of the tissue), helping doctors or robots to observe the tissue environment in the body from a 3D perspective, and can be used for surgical training, surgical simulation, postoperative evaluation, etc.

[0003] In the related technology, the existing three-dimensional reconstruction methods of laparoscopic surgery scenes include Simultaneous Localization and Mapping (SLAM) methods and deep learning methods. Among them, the SLAM method estimates the position of the laparoscope when each laparoscopic image is taken, and then unifies the entire scene into the same coordinate system based on the position of the laparoscope, and realizes the overall three-dimensional reconstruction based on the depth map of all laparoscopic images. The deep learning method hopes that the network can learn the geometric structure of the tissue through pictures. The mainstream Neural Radiance Fields (NeRF) method and 3D Gaussian method mainly realize the back propagation of the network by rendering the colored geometric body into the picture, and compare the difference between the rendered picture and the real picture to modify the properties of the geometric body.

[0004] However, related technologies all assume that the reconstructed objects in the scene are static. For SLAM methods, pose estimation is generally performed through multi-view Figure 3 The accuracy of the solution depends on the accurate matching of feature points on images at different times, which is difficult to achieve in a dynamic environment. For deep learning methods, dynamic scenes cause the binding relationship between spatial position and geometric attributes to change over time, making the learning of geometric attributes of spatial position more difficult. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides a method and system for three-dimensional reconstruction of laparoscopic surgical scenes based on three-dimensional Gaussian sputtering, which solves the technical problem of not supporting three-dimensional reconstruction of dynamic scenes.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A three-dimensional reconstruction method for a laparoscopic surgery scene based on three-dimensional Gaussian sputtering, comprising:

[0010] Acquire a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence;

[0011] Based on the deformation-aware keyframe pool, the initial camera pose of the color image frame at the current moment is estimated, and whether it is a keyframe to be stored in the keyframe pool; wherein the keyframe pool maintains a relatively static regular space 3D Gaussian 3D model;

[0012] If the current color image frame is not a key frame, then

[0013] Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, a dynamic scene optimizer is started to estimate a deformation field for converting the regular space 3D Gaussian 3D model into the real space 3D Gaussian 3D model;

[0014] The color image frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian model are used as input to start the pose optimizer to update the current camera pose;

[0015] The dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least a real-time spatial 3D Gaussian 3D model at the current moment.

[0016] Preferably, if the color image frame at the current moment is a key frame, then

[0017] Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, the static scene optimizer is started to update the regular space 3D Gaussian 3D model;

[0018] The color image frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model are used as input to start the pose optimizer to update the current camera pose;

[0019] The dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least a regular space 3D Gaussian 3D model at the current moment.

[0020] Preferably, the deformation-aware key frame pool estimates the initial camera pose of the color image frame at the current moment, and determines whether it is a key frame to be stored in the key frame pool; including:

[0021] Extract the SIFT feature descriptor of the color image frame, traverse all the key frames in the matching key frame pool, and take the key frame with the most matching key point pairs as the best matching frame;

[0022] Calculate the pose transformation between the color image frame and the best matching frame to obtain the initial camera pose;

[0023] Based on the FPFH feature descriptor of the matching key point pair between the color image frame and the best matching frame, the deformation degree of the color image frame is estimated, and the estimated deformation measurement value is stored in the buffer pool;

[0024] Based on the best matching frame, judging the quality of the color image frame; wherein the judging index includes the number of matching key points on the color image frame and / or the area of ​​the two-dimensional convex hull formed by the matching key points;

[0025] If the deformation metric value of the color image frame at the current moment is less than the corresponding threshold and the quality is greater than the corresponding threshold, it is judged as a key frame and stored in the key frame pool;

[0026] If no new key frame is stored in the key frame pool after the preset time, the color image frame with the smallest deformation metric value is found in the current buffer pool and stored in the key frame pool, and the data in the buffer pool is cleared.

[0027] Preferably, the optimizable parameter of the pose optimizer is the current camera pose, and a corresponding loss function is designed:

[0028]

[0029] in, L pose represents the total loss function of the pose optimizer; L c 、L d They represent the color image loss function and depth image loss function of the pose optimizer respectively;

[0030] Respectively represent the color image frame and its depth map at the current moment;

[0031] Represents the F norm of a matrix or vector;

[0032] express and SSIM loss;

[0033] A differentiable renderer representing a 3D Gaussian splatter, used to render 3D Gaussian 3D models With the current camera pose Pt Rendering into predicted color image frames and the predicted depth map ;

[0034] They respectively represent the current regular space 3D Gaussian 3D model and the current real-time space 3D Gaussian 3D model.

[0035] Preferably, based on the fact that the optimizable parameter of the dynamic scene optimizer is the deformation field, a corresponding loss function is designed:

[0036]

[0037] in, L dynamic represents the total loss function of the dynamic scene optimizer; They represent the color map loss function and depth map loss function of the dynamic scene optimizer respectively;

[0038] Deformation Field W t The 3D Gaussian model of the regular space at the previous moment The deformation effect;

[0039] Represents a 3D Gaussian i The three-dimensional position of the center point in canonical space; KNN ( i ) represents a 3D Gaussian i The K nearest neighbors of Represents a neighboring 3D Gaussian j The three-dimensional position of the center point in canonical space; Represents 3D Gaussian i and neighboring 3D Gaussian j The three-dimensional position of the center point in real space;

[0040] w ij Represents a 3D Gaussian i and neighboring 3D Gaussian j The connection weight between them; exp represents an exponential function with the natural constant e as the base; λ is the weight hyperparameter used to adjust w ij The value of Represents the bi-norm of a vector.

[0041] Preferably, based on the fact that the optimizable parameters of the static scene optimizer are the properties of the regular space 3D Gaussian 3D model at the previous moment, a corresponding loss function is designed:

[0042]

[0043] in, L static represents the total loss function of the static scene optimizer; They represent the color map loss and depth map loss of the static scene optimizer respectively;

[0044] is the property of the regular space 3D Gaussian three-dimensional model at the previous moment; the color of each 3D Gaussian adopts the spherical harmonic function coefficient c lm Indicates that the shape is S Matrix and R The matrix represents R Matrix as Quaternion Q , transparency is adopted α Indicates that the center point position is x express.

[0045] Preferably, during the operation of the static scene optimizer, a densification process is also performed to increase the number of 3D Gaussians in the regular space 3D Gaussian three-dimensional model.

[0046] A laparoscopic surgery scene three-dimensional reconstruction system based on three-dimensional Gaussian sputtering, comprising:

[0047] An acquisition module, used for acquiring a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence;

[0048] A judgment module is used to estimate the initial camera pose of the color image frame at the current moment based on the deformation-aware key frame pool, and to judge whether it is a key frame to be stored in the key frame pool; wherein the key frame pool maintains a relatively static regular space 3D Gaussian 3D model;

[0049] Building blocks for

[0050] If the current color image frame is not a key frame, then

[0051] Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, a dynamic scene optimizer is started to estimate a deformation field for converting the regular space 3D Gaussian 3D model into the real space 3D Gaussian 3D model;

[0052] The color image frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian model are used as input to start the pose optimizer to update the current camera pose;

[0053] The dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least a real-time spatial 3D Gaussian 3D model at the current moment.

[0054] Preferably, the building block is also used for

[0055] If the current color image frame is a key frame, then

[0056] Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, the static scene optimizer is started to update the regular space 3D Gaussian 3D model;

[0057] The color image frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model are used as input to start the pose optimizer to update the current camera pose;

[0058] The dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least a regular space 3D Gaussian 3D model at the current moment.

[0059] A storage medium stores a computer program for three-dimensional reconstruction of a laparoscopic surgery scene based on three-dimensional Gaussian sputtering, wherein the computer program enables a computer to execute the above-mentioned three-dimensional reconstruction method of a laparoscopic surgery scene.

[0060] An electronic device, comprising:

[0061] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the three-dimensional reconstruction method of the laparoscopic surgical scene as described above.

[0062] (III) Beneficial effects

[0063] The present invention provides a method and system for three-dimensional reconstruction of laparoscopic surgery scenes based on three-dimensional Gaussian sputtering. Compared with the prior art, it has the following beneficial effects:

[0064] In the present invention, when the color image frame at the current moment is not a key frame, the dynamic scene optimizer is started to estimate the deformation field used to convert the regular space 3D Gaussian three-dimensional model into the real-time space 3D Gaussian three-dimensional model; then the pose optimizer is started to update the current camera pose; and the two optimizers are run alternately and iteratively until both converge, and the real-time space 3D Gaussian three-dimensional model at the current moment is output. That is, it is possible to generate a high-fidelity dynamic in vivo three-dimensional scene, which is convenient for doctors to observe the patient's internal environment. In addition, depth map information that can provide geometric surface information is introduced and used to supervise the center point coordinates of the 3D Gaussian so that it is bound around the surface, greatly improving the three-dimensional reconstruction accuracy of the tissue surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A block diagram of a method for three-dimensional reconstruction of a laparoscopic surgery scene based on three-dimensional Gaussian sputtering provided in an embodiment of the present invention;

[0067] Figure 2 A block diagram of another method for three-dimensional reconstruction of a laparoscopic surgery scene based on three-dimensional Gaussian sputtering provided in an embodiment of the present invention;

[0068] Figure 3 A complete technical roadmap for a method for three-dimensional reconstruction of a laparoscopic surgical scene provided by an embodiment of the present invention;

[0069] Figure 4 This is a reconstruction effect diagram of a real in vivo environment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] The embodiments of the present application provide a method and system for three-dimensional reconstruction of laparoscopic surgical scenes based on three-dimensional Gaussian sputtering, thereby solving the technical problem of not supporting three-dimensional reconstruction of dynamic scenes.

[0072] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0073] The embodiment of the present invention aims to restore a complete in-vivo 3D model through a sequence of image frames of in-vivo tissues captured by laparoscope. The 3D model has the following characteristics:

[0074] (1) High completeness: The 3D model integrates the information of all image sequences and contains the complete laparoscopic in vivo scene.

[0075] (2) Dynamicity: The three-dimensional model will be dynamically updated according to the actual deformation of the tissue.

[0076] (3) Incremental: As the image sequence continues to be input, the 3D model will gradually become complete.

[0077] (4) High resolution: The three-dimensional model uses 3D Gaussian as the basic geometric expression and is rendered using the latest 3D Gaussian sputtering technology, which can achieve high-resolution three-dimensional model construction.

[0078] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0079] Embodiment 1:

[0080] like Figure 1 As shown, a 3D reconstruction method of a laparoscopic surgery scene based on 3D Gaussian sputtering comprises:

[0081] S1, obtaining a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence;

[0082] S2, based on the deformation-aware key frame pool, estimating the initial camera pose of the color image frame at the current moment, and determining whether it is a key frame to be stored in the key frame pool; wherein the key frame pool maintains a relatively static regular space 3D Gaussian 3D model;

[0083] If the current color image frame is not a key frame, then

[0084] S311, taking the color image frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, starting the dynamic scene optimizer to estimate the deformation field used to convert the regular space 3D Gaussian 3D model into the real-time space 3D Gaussian 3D model;

[0085] S312, taking the color image frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian three-dimensional model as input, and starting the pose optimizer to update the current camera pose;

[0086] S313, alternately iteratively running the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least the real-time spatial 3D Gaussian three-dimensional model at the current moment.

[0087] The embodiment of the present invention can generate a high-fidelity dynamic three-dimensional in-vivo scene, which is convenient for doctors to observe the patient's internal environment. In addition, the depth map information that can provide geometric surface information is introduced and used to supervise the center point coordinates of the 3D Gaussian so that it is bound around the surface, which greatly improves the three-dimensional reconstruction accuracy of the tissue surface.

[0088] In an optional embodiment, if Figure 2 As shown, the laparoscopic surgery scene three-dimensional reconstruction method also includes:

[0089] If the current color image frame is a key frame, then

[0090] S321, taking the color image frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, starting the static scene optimizer to update the regular space 3D Gaussian 3D model;

[0091] S322, taking the color image frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model as input, and starting the pose optimizer to update the current camera pose;

[0092] S323. Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least the canonical space 3D Gaussian 3D model at the current moment.

[0093] When the color image frame at the current moment is a key frame, an embodiment of the present invention updates the key frame pool, starts the static scene optimizer, and updates the regular space 3D Gaussian 3D model; then starts the pose optimizer to update the current camera pose; and alternately iterates and runs the two optimizers until both converge, and outputs the regular space 3D Gaussian 3D model at the current moment, so as to ensure that a relatively static regular space 3D Gaussian 3D model can be maintained in the key frame pool, thereby restoring the complete in-vivo 3D model.

[0094] In the implementation of the present invention, a three-dimensional model of a scene (any symbol containing " " represents the three-dimensional model of the scene) consists of multiple 3D Gaussian G ( x ), the mathematical formula of each 3D Gaussian is as follows. It should be noted that it is necessary to supplement the basic mathematical expressions in the embodiments of the present invention:

[0095] (1) Mathematical expression of three-dimensional model:

[0096] In the embodiment of the present invention, a three-dimensional model of a scene (any symbol containing " " represents the three-dimensional model of the scene) consists of multiple 3D Gaussian G ( x ), the mathematical formula of each 3D Gaussian is as follows:

[0097]

[0098] Among them, 3D Gaussians are geometrically represented as ellipsoids in space, and the geometry of a three-dimensional model of a scene is "built" by these ellipsoids. x Represents the center point position of the three-dimensional Gaussian, that is, the mean of the three-dimensional Gaussian distribution, which is also the coordinate of the center point of the ellipsoid. R Represents the rotation of a three-dimensional Gaussian in space, using quaternions during optimization Q =( q 0 ,q x ,q y ,q z )express. S It represents the scale factor of the three-dimensional Gaussian on the X, Y, and Z axes, that is, the radius of the ellipsoid on its X, Y, and Z axes.

[0099] Considering that the above formula only describes the shape of the 3D Gaussian, and each 3D Gaussian also carries color information and transparency information related to the viewing angle, the embodiment of the present invention further makes the following settings:

[0100] Each 3D Gaussian transparency uses α express, α ∈[0,1], color information uses three sets of spherical harmonic coefficients c lm Represents the color values ​​of the three channels of RGB respectively. l is the order of the spherical harmonic coefficients, m is an integer, - l≤m≤l .

[0101] Among them, the spherical harmonic coefficients are the spherical harmonic functions The form and number of spherical harmonics are determined after the maximum order L is determined. Therefore, the values ​​of the three color channels RGB can be represented by three sets of spherical harmonic coefficients. For example, the value of the R channel can be expressed as: ,in θ represents the zenith angle, φ Represents the azimuth, and these two angles together represent the direction of the viewing angle.

[0102] (2) Mathematical expression of deformation field:

[0103] The deformation field acts on the center point of each three-dimensional Gaussian ellipsoid, causing it to shift in space. The embodiment of the present invention uses three sets of Fourier series , Fit this deformation, each set of Fourier series fits the deformation of one dimension. For example, the displacement in the X dimension It can be calculated by the following formula:

[0104]

[0105] Similarly, the displacement in the Y dimension can be calculated using formula (a): v Y And the displacement in the Z dimension v z , and define the deformation field Wt ( x ):

[0106]

[0107] Formula (b) represents the effect on the center point x The displacement vector of 3D Gaussian and deformation field is given in the following specific form:

[0108]

[0109] in, Represents the three-dimensional coordinates of the center point after displacement, Represents the deformed 3D Gaussian.

[0110] (3) Mathematical expression of camera pose:

[0111] The embodiment of the present invention uses quaternions and translation vectors to express the camera pose, that is, ,in P t Represents the camera pose, which consists of a quaternion that expresses the camera rotation q t and a translation vector expressing the camera translation t t express.

[0112] like Figure 3 As shown, Figure 3 A complete technical roadmap for the three-dimensional reconstruction method of the laparoscopic surgical scene provided by an embodiment of the present invention is given.

[0113] Next, we will combine Figure 3 The following are the steps of the above solution:

[0114] In step S1, a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence are obtained.

[0115] In this step, a color image frame sequence of laparoscopic surgery is obtained. And its corresponding depth map sequence , where t=0,1,2,... represents a time series. Exemplarily, the depth map here can be obtained using a depth estimation algorithm such as MIOL.

[0116] In step S2, based on the deformation-aware key frame pool, the initial camera pose of the color image frame at the current moment is estimated, and it is determined whether it is a key frame that needs to be stored in the key frame pool; wherein the key frame pool maintains a relatively static regular space 3D Gaussian three-dimensional model.

[0117] First of all, it should be clarified that the embodiment of the present invention defines the above deformation-aware key frame pool to have the following functions:

[0118] (1) Identify and store color image frames with relatively small deformation and high quality As keyframe I k , the set of all key frames is is the key frame index. The information in the key frame pool maintains a static 3D model information, called the canonical space. In contrast to the canonical space, the space where the dynamic 3D model is located is called the real-time space.

[0119] (2) Calculate the initial camera pose corresponding to the color image frame at the current moment .

[0120] On this basis, this step specifically includes:

[0121] S21. Extract the SIFT feature descriptor of the color image frame, traverse all key frames in the matching key frame pool, and take the key frame with the most matching key point pairs as the best matching frame.

[0122] Specifically: First, use the key point recognition algorithm to obtain the key points (i.e. points with obvious features, such as edge points of objects) on any pair of matching images; then, extract the SIFT feature descriptors of these key points, and use the SIFT feature descriptors to find matching point pairs among these key points (i.e., if the vector angle of the SIFT feature descriptors between two key points is less than a certain threshold, then this is considered to be a pair of matching key points); finally, take the key frame with the most matching key point pairs as the best matching frame.

[0123] S22, calculating the pose transformation between the color image frame and the best matching frame to obtain the initial camera pose.

[0124] For example, 3D-2D PNP (Perspective N Points) is used to calculate the pose transformation between the color image frame and the best matching frame, thereby obtaining the initial camera pose. , as the initial value for subsequent pose optimization.

[0125] S23. Based on the FPFH feature descriptor of the matching key point pair between the color image frame and the best matching frame, estimate the deformation degree of the color image frame, and store the estimated deformation measurement value in the buffer pool.

[0126] Specifically, after obtaining the best matching frame in S21, based on the pixel coordinates of the matching key point pairs between the color image frame and the best matching frame, the three-dimensional coordinates of these matching key point pairs are restored using the depth map data, and then the FPFH feature descriptor of the matching key points can be obtained using the traditional FPFH algorithm to estimate the deformation degree of the color image frame; the deformation estimation function is as follows:

[0127]

[0128] in, F nr represents the deformation metric value of the color image frame; and It is a pair of FPFH feature descriptors matching key points; N s Represents the number of matching key point pairs; Represents the F-norm of a matrix or vector.

[0129] S24, judging the quality of the color image frame; wherein the judging index includes the number of matching key points on the color image frame and / or the area of ​​the two-dimensional convex hull formed by the matching key points.

[0130] S25. If the deformation measurement value of the color image frame at the current moment is less than the corresponding threshold and the quality is greater than the corresponding threshold, it is judged as a key frame and stored in the key frame pool.

[0131] If the color image frame at the current moment is a key frame, the key frame information will be stored in the key frame pool. The stored information includes but is not limited to the frame ID, initial camera pose, color image frame, depth map, key point coordinates, and SIFT feature descriptors of the key points.

[0132] S26: If no new key frame is stored in the key frame pool after the preset time, a color image frame with the smallest deformation metric value is searched in the current buffer pool and stored in the key frame pool, and the data in the buffer pool is cleared.

[0133] like Figure 3As shown, for whether the color image frame at the current moment is a key frame or not, the embodiment of the present invention processes it according to different processing ideas to output the regular space 3D Gaussian 3D model or the real space 3D Gaussian 3D model at the current moment, as shown below:

[0134] First, if the current color image frame is not a key frame, then

[0135] In step S311, the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment are taken as input, and the dynamic scene optimizer is started to estimate the deformation field used to convert the regular space 3D Gaussian 3D model into the real-time space 3D Gaussian 3D model.

[0136] The scene optimizer in the embodiment of the present invention has two types: dynamic scene optimizer OP dynamic and Static Scene Optimizer OP static , since the color image frame at the current moment is not a key frame, only the dynamic scene optimizer is started here. The dynamic scene optimizer has the following functions:

[0137] Compared to the static scene optimizer, the dynamic scene optimizer only estimates a deformation field W t , W t The regular space 3D Gaussian 3D model will be converted into a real space 3D Gaussian 3D model , which will be converted during the optimization process; the formula is as follows:

[0138]

[0139] In step S312, the color image frame and its depth map at the current moment, the current camera pose, and the current real-time space 3D Gaussian three-dimensional model are used as input to start the pose optimizer to update the current camera pose.

[0140] Pose Optimizer OP pose It will be run during the scene optimizer iteration, for example, it will be run when the scene optimizer iterates T, 2T, 3T...nT times, so the input of the pose optimizer is the initial pose , current picture and And the 3D Gaussian 3D model output by the scene optimizer after iterating for a period of time. If the started scene optimizer is a static scene optimizer, the input 3D Gaussian 3D model is a regular space 3D model If the scene optimizer started is a dynamic scene optimizer, the input 3D Gaussian 3D model is a 3D model of real-time space. , the output of the pose optimizer is the pose of the current frame P t , the pose optimizer will output the best pose after optimization. The scene optimizer started here is a dynamic scene optimizer, and the formula is expressed as follows:

[0141]

[0142] In step S313, the dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least the real-time spatial 3D Gaussian 3D model at the current moment.

[0143] Exemplarily, the optimization solution methods of the two optimizers both use the Adam algorithm.

[0144] (1) Loss function and optimization parameters of the dynamic scene optimizer

[0145] Based on the fact that the optimizable parameter of the dynamic scene optimizer is the deformation field, the corresponding loss function is designed:

[0146]

[0147] in, L dynamic represents the total loss function of the dynamic scene optimizer; They represent the color map loss function and depth map loss function of the dynamic scene optimizer respectively;

[0148] Deformation Field W t The 3D Gaussian model of the regular space at the previous moment The deformation effect;

[0149] Represents a 3D Gaussian i The three-dimensional position of the center point in canonical space; KNN ( i ) represents a 3D Gaussian i The K nearest neighbors of Represents a neighboring 3D Gaussian j The three-dimensional position of the center point in canonical space; Represents 3D Gaussian i and neighboring 3D Gaussian j The three-dimensional position of the center point in real space;

[0150] w ij Represents a 3D Gaussiani and neighboring 3D Gaussian j The connection weight between them; exp represents an exponential function with the natural constant e as the base; λ is the weight hyperparameter used to adjust w ij The value of Represents the bi-norm of a vector.

[0151] (2) About the loss function and optimization parameters of the pose optimizer

[0152] Based on the optimizable parameters of the pose optimizer as the current camera pose, the corresponding loss function is designed:

[0153]

[0154] in, L pose represents the total loss function of the pose optimizer; L c 、L d They represent the color image loss function and depth image loss function of the pose optimizer respectively;

[0155] Respectively represent the color image frame and its depth map at the current moment;

[0156] Represents the F norm of a matrix or vector;

[0157] express and SSIM loss;

[0158] A differentiable renderer representing a 3D Gaussian splatter, used to render 3D Gaussian 3D models With the current camera pose P t Rendering into predicted color image frames and the predicted depth map ;

[0159] They respectively represent the current regular space 3D Gaussian 3D model and the current real-time space 3D Gaussian 3D model.

[0160] Second, if the current color image frame is a key frame, then

[0161] In step S321, the color image frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment are taken as input to start the static scene optimizer to update the regular space 3D Gaussian 3D model.

[0162] The scene optimizer in the embodiment of the present invention has two types: dynamic scene optimizer OP dynamic and Static Scene Optimizer OP static ,Since the color image frame at the current moment is a key frame, only the static scene optimizer is started here.

[0163] For the static scene optimizer, on the one hand, the color image frame at the current moment is used to optimize and generate incremental information (Incremental information refers to the new geometric and texture information carried by the latest frame compared to the previous image sequence); on the other hand, the previous regular space 3D Gaussian 3D model is used , And the camera pose output by the pose optimizer P t , modeling the latest regular space 3D Gaussian 3D model , the formula is as follows:

[0164]

[0165] It should be noted that the incremental information will be directly added to the Gaussian 3D model during the optimization process. The color, shape and other properties of the 3D Gaussian will also be optimized to make it consistent with the observations of the latest frame image.

[0166] In step S322, the color image frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian three-dimensional model are used as input to start the pose optimizer to update the current camera pose.

[0167] Similarly, since the scene optimizer started here is a static scene optimizer, the formula is expressed as follows:

[0168]

[0169] In step S323, the dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least the regular space 3D Gaussian 3D model at the current moment.

[0170] Exemplarily, the optimization solution methods of the two optimizers both use the Adam algorithm.

[0171] (1) Loss function and optimization parameters of static scene optimizer

[0172] Based on the fact that the optimizable parameters of the static scene optimizer are the properties of the regular space 3D Gaussian 3D model at the previous moment, the corresponding loss function is designed:

[0173]

[0174] in, L static represents the total loss function of the static scene optimizer; They represent the color map loss and depth map loss of the static scene optimizer respectively;

[0175] is the property of the regular space 3D Gaussian three-dimensional model at the previous moment; the color of each 3D Gaussian adopts the spherical harmonic function coefficient c lm Indicates that the shape is S Matrix and R The matrix represents R Matrix as Quaternion Q , transparency is adopted α Indicates that the center point position is x express.

[0176] (2) About the loss function and optimization parameters of the pose optimizer

[0177] It is understandable that the content of this part can be completely referred to the content of formulas (1) to (4), and will not be repeated here.

[0178] In particular, during the operation of the static scene optimizer, the embodiment of the present invention also performs a densification process to increase the number of 3D Gaussians in the regular space 3D Gaussian three-dimensional model to obtain incremental information. .

[0179] Specifically, there are two densification methods: segmentation and cloning. The Adam algorithm calculates the gradient for the 3D Gaussian center point visible from the perspective of each color image frame at the current moment. After a period of iteration, if the average gradient obtained by a 3D Gaussian is greater than a certain threshold, it is considered to need to be densified. Among them:

[0180] If the Gaussian is too large, that is, the maximum value in the Gaussian S matrix is ​​greater than or equal to the preset threshold, segmentation densification is performed. The segmentation operation splits the Gaussian into two. For example, the S of these two 3D Gaussians will be multiplied by 0.8, and the center point coordinates of the new 3D Gaussian are the center point coordinates of the original 3D Gaussian plus a bias randomly sampled from a three-dimensional Gaussian distribution with a mean of 0, and the remaining attributes are copied. If the Gaussian is too small, that is, the maximum value in the Gaussian S matrix is ​​less than the preset threshold, cloning densification is performed, that is, copying an identical 3D Gaussian at the original position. The new 3D Gaussians generated by these cloning and segmentation operations are incremental information .

[0181] Since the incremental information is generated in the scene optimizer process, the incremental information, that is, the newly generated 3D Gaussian, will be further optimized, and its properties such as shape, size and color will be optimized to be consistent with the "length" of the color image frame.

[0182] Thus, the embodiment of the present invention designs a three-dimensional reconstruction framework for complex in vivo environments, which can establish a three-dimensional model of the overall environment when the color image frame sequence only captures the local in vivo environment and tissue deformation, thus overcoming the disadvantage of limited laparoscope field of view.

[0183] In addition, in an embodiment of the present invention, depth map information that provides geometric surface information is introduced and used to supervise the center point coordinates of the 3D Gaussians so that they are bound around the surface, greatly improving the three-dimensional reconstruction accuracy of the tissue surface. The addition of the depth map provides a geometric prior for all 3D Gaussians, making it unnecessary to adjust the 3D Gaussian density.

[0184] It is worth pointing out that if Figure 3 As shown, in addition to outputting the 3D Gaussian model at the current moment, the embodiment of the present invention can also simultaneously output the optimal camera pose of the color image frame at the current moment. In other words, the embodiment of the present invention can simultaneously estimate the accurate camera pose and tissue deformation information, providing support information for motion planning for robotic surgery.

[0185] The solution provided in the embodiment of the present invention can be used for simulating real surgical scenes, and combined with VR display technology can help novice doctors perform surgical training. Figure 4 Demonstrated the effect of reconstructing the real in vivo environment.

[0186] Embodiment 2:

[0187] The embodiment of the present invention provides a laparoscopic surgery scene three-dimensional reconstruction system based on three-dimensional Gaussian sputtering, comprising:

[0188] An acquisition module, used for acquiring a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence;

[0189] A judgment module is used to estimate the initial camera pose of the color image frame at the current moment based on the deformation-aware key frame pool, and to judge whether it is a key frame to be stored in the key frame pool; wherein the key frame pool maintains a relatively static regular space 3D Gaussian 3D model;

[0190] Building blocks for

[0191] If the current color image frame is not a key frame, then

[0192] Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, a dynamic scene optimizer is started to estimate a deformation field for converting the regular space 3D Gaussian 3D model into the real space 3D Gaussian 3D model;

[0193] The color image frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian model are used as input to start the pose optimizer to update the current camera pose;

[0194] The dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least a real-time spatial 3D Gaussian 3D model at the current moment.

[0195] In an optional embodiment, the construction module in the laparoscopic surgery scene three-dimensional reconstruction system is also used

[0196] If the current color image frame is a key frame, then

[0197] Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, the static scene optimizer is started to update the regular space 3D Gaussian 3D model;

[0198] The color image frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model are used as input to start the pose optimizer to update the current camera pose;

[0199] The dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least a regular space 3D Gaussian 3D model at the current moment.

[0200] Embodiment 3:

[0201] An embodiment of the present invention provides a storage medium storing a computer program for three-dimensional reconstruction of a laparoscopic surgery scene based on three-dimensional Gaussian sputtering, wherein the computer program enables a computer to execute the three-dimensional reconstruction method of a laparoscopic surgery scene as described in Example 1.

[0202] Embodiment 4:

[0203] An embodiment of the present invention provides an electronic device, including:

[0204] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing a three-dimensional reconstruction method of a laparoscopic surgical scene as described in Example 1.

[0205] It can be understood that the three-dimensional reconstruction system, storage medium and electronic device for laparoscopic surgery scenes based on three-dimensional Gaussian sputtering provided in the embodiments of the present invention correspond to the three-dimensional reconstruction method for laparoscopic surgery scenes based on three-dimensional Gaussian sputtering provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the three-dimensional reconstruction method for laparoscopic surgery scenes, and will not be repeated here.

[0206] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0207] 1. The embodiment of the present invention designs a three-dimensional reconstruction framework for complex in vivo environments, which can establish a three-dimensional model of the overall environment when the color image frame sequence only captures the local in vivo environment and tissue deformation, thereby overcoming the disadvantage of limited laparoscope field of view.

[0208] 2. In the embodiment of the present invention, depth map information that provides geometric surface information is introduced, and it is used to supervise the center point coordinates of the 3D Gaussian so that it is bound around the surface, which greatly improves the three-dimensional reconstruction accuracy of the tissue surface. The addition of the depth map provides a geometric prior for all 3D Gaussians, making it unnecessary to adjust the 3D Gaussian density.

[0209] 3. In addition to outputting the 3D Gaussian model at the current moment, the embodiment of the present invention can also simultaneously output the optimal camera pose of the color image frame at the current moment. In other words, the embodiment of the present invention can simultaneously estimate the accurate camera pose and tissue deformation information, providing support information for motion planning for robotic surgery.

[0210] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0211] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D reconstruction method for laparoscopic surgery scenes based on 3D Gaussian sputtering, characterized in that: include: Acquire a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence; Based on the deformation-aware keyframe pool, the initial camera pose of the color image frame at the current moment is estimated, and whether it is a keyframe to be stored in the keyframe pool; wherein the keyframe pool maintains a relatively static regular space 3D Gaussian 3D model; If the current color image frame is not a key frame, then Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, a dynamic scene optimizer is started to estimate a deformation field for converting the regular space 3D Gaussian 3D model into the real space 3D Gaussian 3D model; The color image frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian model are used as input to start the pose optimizer to update the current camera pose; Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both are converged, so as to output at least a real-time spatial 3D Gaussian 3D model at the current moment; If the current color image frame is a key frame, then Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, the static scene optimizer is started to update the regular space 3D Gaussian 3D model; The color image frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model are used as input to start the pose optimizer to update the current camera pose; Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least a regular space 3D Gaussian 3D model at the current moment; The deformation-aware key frame pool estimates the initial camera pose of the color image frame at the current moment, and determines whether it is a key frame to be stored in the key frame pool; including: Extract the SIFT feature descriptor of the color image frame, traverse all the key frames in the matching key frame pool, and take the key frame with the most matching key point pairs as the best matching frame; Calculate the pose transformation between the color image frame and the best matching frame to obtain the initial camera pose; Based on the FPFH feature descriptor of the matching key point pair between the color image frame and the best matching frame, the deformation degree of the color image frame is estimated, and the estimated deformation measurement value is stored in the buffer pool; Based on the best matching frame, judging the quality of the color image frame; wherein the judging index includes the number of matching key points on the color image frame and / or the area of ​​the two-dimensional convex hull formed by the matching key points; If the deformation metric value of the color image frame at the current moment is less than the corresponding threshold and the quality is greater than the corresponding threshold, it is judged as a key frame and stored in the key frame pool; If no new key frame is stored in the key frame pool after the preset time, the color image frame with the smallest deformation metric value is found in the current buffer pool and stored in the key frame pool, and the data in the buffer pool is cleared.

2. The method for three-dimensional reconstruction of a laparoscopic surgery scene according to claim 1, characterized in that: Based on the optimizable parameters of the pose optimizer as the current camera pose, the corresponding loss function is designed: in, L pose represents the total loss function of the pose optimizer; L c 、L d They represent the color image loss function and depth image loss function of the pose optimizer respectively; Respectively represent the color image frame and its depth map at the current moment; Represents the F norm of a matrix or vector; express and SSIM loss; A differentiable renderer representing a 3D Gaussian splatter, used to render 3D Gaussian 3D models With the current camera pose P t Rendering into predicted color image frames and the predicted depth map ; They respectively represent the current regular space 3D Gaussian 3D model and the current real-time space 3D Gaussian 3D model.

3. The method for three-dimensional reconstruction of a laparoscopic surgery scene according to claim 2, characterized in that: Based on the fact that the optimizable parameter of the dynamic scene optimizer is the deformation field, the corresponding loss function is designed: in, L dynamic represents the total loss function of the dynamic scene optimizer; They represent the color map loss function and depth map loss function of the dynamic scene optimizer respectively; Deformation Field W t The 3D Gaussian model of the regular space at the previous moment The deformation effect; Represents a 3D Gaussian i The three-dimensional position of the center point in canonical space; KNN ( i ) represents a 3D Gaussian i The K nearest neighbors of Represents a neighboring 3D Gaussian j The three-dimensional position of the center point in canonical space; Represents 3D Gaussian i and neighboring 3D Gaussian j The three-dimensional position of the center point in real space; w ij Represents a 3D Gaussian i and neighboring 3D Gaussian j The connection weight between them; exp represents an exponential function with the natural constant e as the base; λ is the weight hyperparameter used to adjust w ij The value of represents the two-norm of a vector; and / or Based on the fact that the optimizable parameters of the static scene optimizer are the properties of the regular space 3D Gaussian 3D model at the previous moment, the corresponding loss function is designed: in, L static represents the total loss function of the static scene optimizer; They represent the color map loss and depth map loss of the static scene optimizer respectively; is the property of the regular space 3D Gaussian three-dimensional model at the previous moment; the color of each 3D Gaussian adopts the spherical harmonic function coefficient c lm Indicates that the shape is S Matrix and R The matrix represents R Matrix as Quaternion Q , transparency is adopted α Indicates that the center point position is x express.

4. The method for three-dimensional reconstruction of a laparoscopic surgery scene according to claim 1, characterized in that: During the operation of the static scene optimizer, a densification process is also performed to increase the number of 3D Gaussians in the regular space 3D Gaussian three-dimensional model.

5. A laparoscopic surgery scene 3D reconstruction system based on 3D Gaussian sputtering, characterized in that: include: An acquisition module, used for acquiring a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence; A judgment module is used to estimate the initial camera pose of the color image frame at the current moment based on the deformation-aware key frame pool, and to judge whether it is a key frame to be stored in the key frame pool; wherein the key frame pool maintains a relatively static regular space 3D Gaussian 3D model; Building blocks for If the current color image frame is not a key frame, then Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, a dynamic scene optimizer is started to estimate a deformation field for converting the regular space 3D Gaussian 3D model into the real space 3D Gaussian 3D model; The color image frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian model are used as input to start the pose optimizer to update the current camera pose; Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both are converged, so as to output at least a real-time spatial 3D Gaussian 3D model at the current moment; The building blocks are also used If the current color image frame is a key frame, then Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, the static scene optimizer is started to update the regular space 3D Gaussian 3D model; The color image frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model are used as input to start the pose optimizer to update the current camera pose; Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least a regular space 3D Gaussian 3D model at the current moment; The judgment module is used to estimate the initial camera pose of the color image frame at the current moment based on the deformation-aware key frame pool, and to judge whether it is a key frame to be stored in the key frame pool; including: Extract the SIFT feature descriptor of the color image frame, traverse all the key frames in the matching key frame pool, and take the key frame with the most matching key point pairs as the best matching frame; Calculate the pose transformation between the color image frame and the best matching frame to obtain the initial camera pose; Based on the FPFH feature descriptor of the matching key point pair between the color image frame and the best matching frame, the deformation degree of the color image frame is estimated, and the estimated deformation measurement value is stored in the buffer pool; Based on the best matching frame, judging the quality of the color image frame; wherein the judging index includes the number of matching key points on the color image frame and / or the area of ​​the two-dimensional convex hull formed by the matching key points; If the deformation metric value of the color image frame at the current moment is less than the corresponding threshold and the quality is greater than the corresponding threshold, it is judged as a key frame and stored in the key frame pool; If no new key frame is stored in the key frame pool after the preset time, the color image frame with the smallest deformation metric value is found in the current buffer pool and stored in the key frame pool, and the data in the buffer pool is cleared.

6. A storage medium, characterized in that: It stores a computer program for three-dimensional reconstruction of a laparoscopic surgery scene based on three-dimensional Gaussian sputtering, wherein the computer program enables a computer to execute the three-dimensional reconstruction method of a laparoscopic surgery scene as described in any one of claims 1 to 4.

7. An electronic device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the three-dimensional reconstruction method of the laparoscopic surgical scene as described in any one of claims 1 to 4.

Images