Interactive surgical case guidance method and system based on manual annotation guidance

Abstract

The present invention provides an interactive surgical case guidance method, system, storage medium and electronic device based on manual annotation guidance, which relates to the field of model registration. In the present invention, feature marker points are selected on the preoperative CT model, and corresponding intraoperative marker points are selected on the color image frame; then, non-rigid registration is performed based on the 3D Gaussian three-dimensional model rendered by three-dimensional Gaussian sputtering to obtain the intraoperative CT model; then, the intraoperative CT model is Gaussian initialized and optimized; then, the ROI area marked by the doctor on the color image frame at the intraoperative moment is received, and 3D Gaussian and CT Gaussian sampling are completed respectively; finally, two Gaussian models are separately rendered based on GPU sorting and block rendering strategies to obtain the fusion image of the original image and CT image for interactive surgical case guidance. The present invention can present more accurate tissue anatomical information and achieve accurate decision-making, which is especially helpful for novice doctors to observe the surgical characteristics of different surgical cases at different surgical moments and improve surgical experience cognition.

Description

Technical Field

[0001] The present invention relates to the field of model registration, and in particular to an interactive surgical case guidance method, system, storage medium and electronic device based on manual annotation guidance. Background Art

[0002] In laparoscopic surgery, the intraoperative field of view is limited and the operation is restricted by space. In this context, how to align the preoperative model with the actual intraoperative situation is one of the current research hotspots.

[0003] In the related art, the literature (Koo B, Robu MR, Allam M, et al. Automatic, global registration in laparoscopic liver surgery [J]. International Journal of Computer Assisted Radiology and Surgery, 2022: 1-10.) proposed a 3D-2D registration solution designed specifically for laparoscopic liver surgery. Specifically, the solution uses a neural network to extract image features in intraoperative images, such as discrete feature points, feature lines formed by continuous feature points, and feature surfaces formed by continuous feature points, and then projects some or all of the feature points to the target coordinate system to update the preoperative three-dimensional model.

[0004] However, the above scheme lacks sufficient registration accuracy and can only display the general outline and topological structure of the three-dimensional model, but cannot observe the anatomical information of the deep internal structure tissue of organs such as the liver, and thus cannot provide surgeons with more sufficient preoperative information. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides an interactive surgical case guidance method, system, storage medium and electronic device based on manual annotation guidance, which solves the technical problems of lack of sufficient registration accuracy and inability to observe anatomical information of internal deep structure tissues.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] An interactive surgical case guidance method based on manual annotation guidance, comprising:

[0010] Based on the preoperative nii volume data, a preoperative CT model and a first grayscale value information set thereof are obtained, and a point mark on the preoperative CT model by a doctor is received to obtain a three-dimensional coordinate set of a 3D mark point;

[0011] Acquire a color image frame sequence of laparoscopic surgery, receive the point markings made by the doctor on the color image frame at the current moment, obtain a 2D marker point uv coordinate set corresponding to the 3D marker point 3D coordinate set, use the Ransac-PnP algorithm to solve the rigid transformation matrix from the preoperative to the intraoperative coordinate system, and obtain a 3D point cloud set after rigid registration;

[0012] A laparoscopic surgery scene 3D reconstruction method based on 3D Gaussian sputtering is used to obtain a 3D Gaussian 3D model at the current moment; and a V2snet method is used to non-rigidly align the 3D point cloud set with the 3D Gaussian 3D model at the current moment to obtain an intraoperative CT model;

[0013] Performing CT Gaussian initialization on the intraoperative CT model to obtain a CT Gaussian model;

[0014] The intraoperative CT model is interpolated, and the interpolated point cloud model is used to render the corresponding CT grayscale image and depth map based on the preset camera pose; and the CT grayscale image and depth map are used as supervision signals to optimize the CT Gaussian model.

[0015] Preferably, the interactive surgical case guidance method further includes: receiving the ROI area marked by the doctor on the color image frame at the current moment, restoring the pixel curve of the edge of the area to the three-dimensional space to obtain the section point information set of the intraoperative CT model; and based on the first gray value information set, obtaining a second gray value information set of the section point information set;

[0016] Based on GPU sorting and block rendering strategy, the current 3D Gaussian model and the optimized CT Gaussian model are rendered separately to obtain a fusion image of the original image and the CT image for interactive surgical case guidance; wherein:

[0017] The volume rendering method is used to render the 3D Gaussian outside the ROI area;

[0018] The coordinate z component of each sampling point in the section point information set is used as the depth value, and combined with the corresponding second gray value information, the CT Gaussian at the corresponding depth value of the ROI area is rendered.

[0019] Preferably, the three-dimensional reconstruction method of the laparoscopic surgery scene based on three-dimensional Gaussian sputtering obtains the 3D Gaussian three-dimensional model at the current moment; comprising:

[0020] Acquire a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence;

[0021] Based on the deformation-aware keyframe pool, the initial camera pose of the color image frame at the current moment is estimated, and whether it is a keyframe to be stored in the keyframe pool; wherein the keyframe pool maintains a relatively static regular space 3D Gaussian 3D model;

[0022] If the current color image frame is not a key frame, then

[0023] Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, a dynamic scene optimizer is started to estimate a deformation field for converting the regular space 3D Gaussian 3D model into the real space 3D Gaussian 3D model;

[0024] The color image frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian model are used as input to start the pose optimizer to update the current camera pose;

[0025] Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both are converged, so as to output at least a real-time spatial 3D Gaussian 3D model at the current moment;

[0026] If the current color image frame is a key frame, then

[0027] Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, the static scene optimizer is started to update the regular space 3D Gaussian 3D model;

[0028] The color image frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model are used as input to start the pose optimizer to update the current camera pose;

[0029] Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least a regular space 3D Gaussian 3D model at the current moment;

[0030] The real-time space 3D Gaussian 3D model corresponding to the last non-key frame up to the current moment is selected as the 3D Gaussian 3D model at the current moment.

[0031] Preferably, the deformation-aware key frame pool estimates the initial camera pose of the color image frame at the current moment, and determines whether it is a key frame to be stored in the key frame pool; including:

[0032] Extract the SIFT feature descriptor of the color image frame, traverse all the key frames in the matching key frame pool, and take the key frame with the most matching key point pairs as the best matching frame;

[0033] Calculate the pose transformation between the color image frame and the best matching frame to obtain the initial camera pose;

[0034] Based on the FPFH feature descriptor of the matching key point pair between the color image frame and the best matching frame, the deformation degree of the color image frame is estimated, and the estimated deformation measurement value is stored in the buffer pool;

[0035] Based on the best matching frame, judging the quality of the color image frame; wherein the judging index includes the number of matching key points on the color image frame and / or the area of ​​the two-dimensional convex hull formed by the matching key points;

[0036] If the deformation metric value of the color image frame at the current moment is less than the corresponding threshold and the quality is greater than the corresponding threshold, it is judged as a key frame and stored in the key frame pool;

[0037] If no new key frame is stored in the key frame pool after the preset time, the color image frame with the smallest deformation metric value is found in the current buffer pool and stored in the key frame pool, and the data in the buffer pool is cleared.

[0038] Preferably, based on the optimizable parameter of the pose optimizer being the current camera pose, a corresponding loss function is designed:

[0039]

[0040] in, L pose represents the total loss function of the pose optimizer; L c 、L d They represent the color image loss function and depth image loss function of the pose optimizer respectively;

[0041] , Respectively represent the color image frame and its depth map at the current moment;

[0042] Represents the F norm of a matrix or vector; express and SSIM loss;

[0043] A differentiable renderer representing a 3D Gaussian splatter, used to render 3D Gaussian 3D models With the current camera pose P t Rendering into predicted color video frames and the predicted depth map ;

[0044] , They respectively represent the current regular space 3D Gaussian 3D model and the current real-time space 3D Gaussian 3D model.

[0045] Preferably, based on the fact that the optimizable parameter of the dynamic scene optimizer is the deformation field, a corresponding loss function is designed:

[0046]

[0047] in, L dynamic represents the total loss function of the dynamic scene optimizer; They represent the color map loss function and depth map loss function of the dynamic scene optimizer respectively;

[0048] Deformation Field W t The 3D Gaussian model of the regular space at the previous moment The deformation effect;

[0049] Represents a 3D Gaussian i The three-dimensional position of the center point in canonical space; KNN ( i ) represents a 3D Gaussian i The K nearest neighbors of Represents a neighboring 3D Gaussian j The three-dimensional position of the center point in canonical space; Represents 3D Gaussian i and neighboring 3D Gaussian j The three-dimensional position of the center point in real space;

[0050] w ij Represents a 3D Gaussian i and neighboring 3D Gaussian j The connection weight between them; exp represents an exponential function with the natural constant e as the base; λ is the weight hyperparameter used to adjust w ij The value of Represents the bi-norm of a vector.

[0051] Preferably, based on the fact that the optimizable parameters of the static scene optimizer are the properties of the regular space 3D Gaussian 3D model at the previous moment, a corresponding loss function is designed:

[0052]

[0053] in, L static represents the total loss function of the static scene optimizer; They represent the color map loss and depth map loss of the static scene optimizer respectively;

[0054] is the property of the regular space 3D Gaussian three-dimensional model at the previous moment; the color of each 3D Gaussian adopts the spherical harmonic function coefficient c lm Indicates that the shape is S Matrix and R The matrix represents R Matrix as Quaternion , transparency is adopted α Indicates that the center point position is x express.

[0055] Preferably, during the operation of the static scene optimizer, a densification process is also performed to increase the number of 3D Gaussians in the regular space 3D Gaussian three-dimensional model.

[0056] Preferably, the method of using the V2snet method to non-rigidly register the three-dimensional point cloud set with the 3D Gaussian three-dimensional model at the current moment to obtain the intraoperative CT model comprises:

[0057] Based on 3D point cloud collection 3D Gaussian model at the current moment The center point coordinate information collection , the V2snet method is used to predict the flexible deformation displacement field of the three-dimensional point cloud set V t ;

[0058] Based on the flexible deformation displacement field V t , obtain the intraoperative CT model in the form of point cloud .

[0059] Preferably, the CT Gaussian model is expressed as , whose attributes include the center point position, opacity, color information and shape information of each CT Gaussian; the CT Gaussian initialization of the intraoperative CT model to obtain the CT Gaussian model; including:

[0060] Intraoperative CT model The three-dimensional coordinates of each point in are taken as the center point coordinates of the initial CT Gaussian;

[0061] The opacity of each initial CT Gaussian is used α express, α∈[0,1];

[0062] For the color information of each initial CT Gaussian, a set of spherical harmonic coefficients is used c lm express The single channel grayscale value corresponding to each point in ;

[0063] For the shape information of each initial CT Gaussian, R Represents the rotation of the initial CT Gaussian in space, using quaternion during optimization Q =( q 0 ,q x ,q y ,q z )express; S i Indicates i The scale coefficients of the initial CT Gaussian on the X, Y, and Z axes are used in the optimization process. KNN The algorithm finds the nearest Gaussian center points to each point on the model and passes Max The function finds the maximum value among the distances to obtain the scaling coefficient for each initial CT Gaussian.

[0064] Preferably, the intraoperative CT model is interpolated, and the interpolated point cloud model is used to render the corresponding CT grayscale image and depth map based on a preset camera pose; and the CT grayscale image and depth map are used as supervisory signals to optimize the CT Gaussian model; including:

[0065] Combined with the first gray value information set, the intraoperative CT model is transformed into Modeled as a colored mesh , and interpolate in each grid to obtain the interpolated point cloud model PC ; wherein the interpolated point cloud model PC Including 3D point cloud coordinate set and 3D point cloud color set;

[0066] Set a spherical coordinate system, set the radius range of the spherical coordinate system to the distance between the farthest CT Gaussians, set the polar angle range according to the camera's field of view, and set the azimuth angle range to the front of the model. Sample M camera positions uniformly on the spherical coordinate system and convert them into Cartesian coordinates.

[0067] Align the camera's viewing angle to the center of the Gaussian 3D model and offset it up, down, left, and right in that viewing angle. a degrees, expanding each camera position to a The camera poses are finally sampled. aM camera poses, and the interpolated 3D point cloud PC exist aM The camera poses are projected and rendered to obtain the PC CT grayscale image and depth map;

[0068] The rendered CT grayscale image and depth map are used as pseudo-true values ​​to supervise the CT Gaussian model Optimize to obtain the optimized CT Gaussian model .

[0069] Preferably, the receiving doctor marks the ROI area on the color image frame at the current moment, restores the pixel curve of the edge of the area to the three-dimensional space to obtain the section point information set of the intraoperative CT model; and based on the first gray value information set, obtains the second gray value information set of the section point information set; including:

[0070] Receive the ROI area marked by the doctor on the color image frame at the current moment, restore the pixel curve of the edge of the area to the three-dimensional space based on the camera internal parameters and depth map information, and obtain the three-dimensional curve ;

[0071] In the three-dimensional curve Randomly sample three points , , , forming two vectors and , and calculate the normal vector of the corresponding section , using sampling points q a The three-dimensional coordinates of Th ;

[0072] Based on the tangent plane equation Th Intraoperative CT Model , filter the coordinates of the model points located at the section, and obtain the section point information set P tan ;

[0073] In the first gray value information set G pre , filter the section point information set P tan The gray value information corresponding to each point in the gray value information set is obtained Gray tan .

[0074] Preferably, the GPU sorting and block rendering strategy is based on which the 3D Gaussian model at the current moment and the optimized CT Gaussian model are separately rendered; including:

[0075] (1) Screen segmentation and block numbering: The screen displaying the color image frame is divided into m × n blocks, each containing block_x × block_y pixels, and numbered in sequence;

[0076] (2) Screening of blocks in the marked area: Based on the pixel value of the ROI area on the screen, obtain the corresponding block on the screen and its ID number;

[0077] (3) Gaussian model classification and encoding: Two Gaussian ID representations are introduced to distinguish 3D Gaussian and CT Gaussian respectively;

[0078] 3D Gaussian is represented as [3D_GAUSSIAN, ID, depth], and CT Gaussian is represented as [CT_GAUSSIAN, ID, depth], and they are represented by 16 bits, 16 bits, and 32 bits of binary code respectively; depth represents the depth value of Gaussian in the scene;

[0079] (4) GPU quick sorting and rendering: The 3D Gaussian and CT Gaussian lists are quickly sorted on the GPU, and the blocks are rendered separately based on their ID numbers and depth values ​​according to the Gaussian sorting results; wherein the separate rendering is as follows:

[0080] The volume rendering method is used to render the 3D Gaussian outside the ROI area;

[0081] The section point information set P tan The coordinate z component of each sampling point in is used as the depth value, and combined with the corresponding second gray value information, the CT Gaussian at the corresponding depth value of the ROI area is rendered.

[0082] Preferably, the volume rendering method is used to render the 3D Gaussian outside the ROI area; it is expressed as:

[0083]

[0084] in, c Indicates the RGB color value finally rendered by the light; c i Indicated in i RGB color value at each sampling point; N Indicates the number of sampling points; α i Indicates the light i The opacity at each sampling point; Indicates that the light reaches i The probability that a sampling point is not blocked by all previous sampling points;

[0085] Preferably, the coordinate z component of each sampling point in the section point information set is used as the depth value, combined with the corresponding second gray value information, to render the CT Gaussian at the depth value corresponding to the ROI area; expressed as:

[0086]

[0087] in, gray Indicates the grayscale value finally rendered by the light; gray i Indicated in i The gray value at the sampling point.

[0088] An interactive surgical case guidance system based on manual annotation guidance, comprising:

[0089] A reconstruction module, used to obtain a preoperative CT model and a first grayscale value information set thereof based on the preoperative nii volume data, and receive a point mark made by a doctor on the preoperative CT model to obtain a three-dimensional coordinate set of a 3D mark point;

[0090] A registration module is used to obtain a color image frame sequence of laparoscopic surgery, receive the point marks made by the doctor on the color image frame at the current moment, obtain a 2D marker point uv coordinate set corresponding to the 3D marker point 3D coordinate set, use the Ransac-PnP algorithm to solve the rigid transformation matrix from the preoperative to the intraoperative coordinate system, and obtain a 3D point cloud set after rigid registration;

[0091] A method for three-dimensional reconstruction of a laparoscopic surgery scene based on three-dimensional Gaussian sputtering is provided to obtain a 3D Gaussian three-dimensional model at the current moment; and a V2snet method is used to non-rigidly align the three-dimensional point cloud set with the 3D Gaussian three-dimensional model at the current moment to obtain an intraoperative CT model;

[0092] An optimization module, used for performing CT Gaussian initialization on the intraoperative CT model to obtain a CT Gaussian model;

[0093] And used to interpolate the intraoperative CT model, use the interpolated point cloud model, render based on the preset camera pose to obtain the corresponding CT grayscale image and depth map; and use the CT grayscale image and depth map as supervision signals to optimize the CT Gaussian model;

[0094] A sampling module is used to receive the ROI area marked by the doctor on the color image frame at the current moment, restore the pixel curve of the edge of the area to the three-dimensional space, so as to obtain the section point information set of the intraoperative CT model; and obtain the second gray value information set of the section point information set based on the first gray value information set;

[0095] The rendering module is used to separately render the current 3D Gaussian model and the optimized CT Gaussian model based on GPU sorting and block rendering strategy to obtain a fused image of the original image and the CT image for interactive surgical case guidance; wherein:

[0096] The volume rendering method is used to render the 3D Gaussian outside the ROI area;

[0097] The coordinate z component of each sampling point in the section point information set is used as the depth value, and combined with the corresponding second gray value information, the CT Gaussian at the corresponding depth value of the ROI area is rendered.

[0098] A storage medium stores a computer program for interactive surgical case guidance based on manual annotation guidance, wherein the computer program enables a computer to execute the interactive surgical case guidance method as described above.

[0099] An electronic device, comprising:

[0100] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the interactive surgical case guidance method as described above.

[0101] (III) Beneficial effects

[0102] The present invention provides an interactive surgical case guidance method, system, storage medium and electronic device based on manual annotation guidance. Compared with the prior art, it has the following beneficial effects:

[0103] In the present invention, feature markers are selected on the preoperative CT model, and corresponding intraoperative markers are selected on the color image frame to achieve initial rigid registration of the preoperative to intraoperative model; then, non-rigid registration is performed based on the 3D Gaussian three-dimensional model rendered by three-dimensional Gaussian sputtering to obtain the intraoperative CT model, so as to achieve comprehensive monitoring and recording of the intraoperative operation process; then, the intraoperative CT model is Gaussian initialized and optimized; then, the ROI area marked by the doctor on the color image frame at the intraoperative moment is received, and 3D Gaussian and CT Gaussian sampling are completed respectively; finally, two Gaussian models are rendered separately based on GPU sorting and block rendering strategies to obtain the fusion image of the original image and CT image for interactive surgical case guidance. The present invention can present more accurate tissue anatomical information, achieve accurate decision-making, and is particularly helpful for novice doctors to observe the surgical characteristics of different surgical cases at different surgical moments and improve surgical experience cognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0105] Figure 1 A block diagram of an interactive surgical case guidance method based on manual annotation guidance provided by an embodiment of the present invention;

[0106] Figure 2 A block diagram of a method for three-dimensional reconstruction of a laparoscopic surgery scene based on three-dimensional Gaussian sputtering provided in an embodiment of the present invention;

[0107] Figure 3 A complete technical roadmap for an interactive surgical case guidance method based on manual annotation guidance provided by an embodiment of the present invention;

[0108] Figure 4 A complete technical roadmap for a method for three-dimensional reconstruction of laparoscopic surgical scenes based on three-dimensional Gaussian sputtering provided in an embodiment of the present invention;

[0109] Figure 5 A reconstruction effect diagram of a real in vivo environment provided by an embodiment of the present invention;

[0110] Figure 6 This is a rigid registration effect diagram of a real in vivo environment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0111] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0112] The embodiments of the present application solve the technical problems of lack of sufficient registration accuracy and inability to observe anatomical information of internal deep structure tissues by providing an interactive surgical case guidance method, system, storage medium and electronic device based on manual annotation guidance.

[0113] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0114] The embodiments of the present invention are dedicated to providing customized training for novice doctors in addition to surgery, especially for different surgical cases, to enhance the novice doctors' surgical experience and knowledge, and to improve the accuracy of surgery.

[0115] Specifically, by accurately registering the preoperative CT model with the 3D Gaussian model constructed by the intraoperative tissue, it is ensured that the doctor can accurately locate the target area during the operation and reduce the risk of the operation. At the same time, after the registration is completed, the embodiment of the present invention can record every moment of the operation, including the overall 3D model of the tissue at each moment, the changes in the tissue structure and the corresponding registration results and other data information.

[0116] Therefore, in addition to surgery, the embodiments of the present invention can provide customized training for novice doctors based on different surgical cases. Novice doctors can observe and learn in detail the changes in tissues at different times, as well as important data information that is difficult to observe in time during actual surgery. This allows them to better understand the characteristics of different surgical cases, improve their surgical experience and cognition, assist in making accurate decisions during actual intraoperative operations, and improve the accuracy and safety of surgery.

[0117] It is worth mentioning that an important feature of the embodiment of the present invention is its interactive nature. Doctors can manually mark points on the display screen during surgery and participate in the registration process, thereby improving the accuracy of registration. In addition, the embodiment of the present invention is also real-time and recordable, and the surgical information recorded during surgery can be displayed in real time by the system outside the surgery, providing strong support for the subsequent training and practice of novice doctors. Through this system, doctors can obtain good surgical guidance and training, thereby improving the accuracy and success rate of operations in actual operations.

[0118] In particular, the three-dimensional reconstruction method of the laparoscopic surgical scene based on three-dimensional Gaussian sputtering proposed in the embodiment of the present invention starts the dynamic scene optimizer when the color video frame at the current moment is not a key frame, estimates the deformation field used to convert the regular space 3D Gaussian three-dimensional model into the real-time space 3D Gaussian three-dimensional model; then starts the pose optimizer to update the current camera pose; and alternately iterates and runs the two optimizers until both converge, and outputs the real-time space 3D Gaussian three-dimensional model at the current moment. That is, it is possible to generate a high-fidelity dynamic in vivo three-dimensional scene, which is convenient for doctors to observe the patient's internal environment. In addition, depth map information that can provide geometric surface information is introduced, and it is used to supervise the center point coordinates of the 3D Gaussian so that it is bound around the surface, greatly improving the three-dimensional reconstruction accuracy of the tissue surface.

[0119] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0120] Embodiment 1:

[0121] like Figure 1 As shown, an embodiment of the present invention provides an interactive surgical case guidance method based on manual annotation guidance, comprising:

[0122] S1. Based on the preoperative nii volume data, a preoperative CT model and a first grayscale value information set thereof are obtained, and a doctor's point marking on the preoperative CT model is received to obtain a three-dimensional coordinate set of 3D marking points;

[0123] S2, obtaining a color image frame sequence of laparoscopic surgery, receiving the point markings made by the doctor on the color image frame at the current moment, obtaining a 2D marker point uv coordinate set corresponding to the 3D marker point 3D coordinate set, using the Ransac-PnP algorithm to solve the rigid transformation matrix from the preoperative to the intraoperative coordinate system, and obtaining a 3D point cloud set after rigid registration;

[0124] S3, a laparoscopic surgery scene three-dimensional reconstruction method based on three-dimensional Gaussian sputtering, to obtain a 3D Gaussian three-dimensional model at the current moment; and a V2snet method is used to non-rigidly align the three-dimensional point cloud set with the 3D Gaussian three-dimensional model at the current moment to obtain an intraoperative CT model;

[0125] S4, performing CT Gaussian initialization on the intraoperative CT model to obtain a CT Gaussian model;

[0126] S5, interpolating the intraoperative CT model, using the interpolated point cloud model, rendering based on a preset camera pose to obtain a corresponding CT grayscale image and depth map; and using the CT grayscale image and depth map as supervisory signals to optimize the CT Gaussian model;

[0127] S6, receiving the ROI area marked by the doctor on the color image frame at the current moment, restoring the pixel curve of the edge of the area to the three-dimensional space to obtain the section point information set of the intraoperative CT model; and obtaining the second gray value information set of the section point information set based on the first gray value information set;

[0128] S7. Based on the GPU sorting and block rendering strategy, the 3D Gaussian model at the current moment and the optimized CT Gaussian model are separately rendered to obtain a fused image of the original image and the CT image for interactive surgical case guidance; wherein:

[0129] The volume rendering method is used to render the 3D Gaussian outside the ROI area;

[0130] The coordinate z component of each sampling point in the section point information set is used as the depth value, and combined with the corresponding second gray value information, the CT Gaussian at the corresponding depth value of the ROI area is rendered.

[0131] The embodiments of the present invention can present more accurate tissue anatomical information and enable precise decision-making, which is particularly helpful for novice doctors to observe the surgical characteristics of different surgical cases at different surgical moments and improve their surgical experience cognition.

[0132] In an optional embodiment, the laparoscopic surgery scene three-dimensional reconstruction method based on three-dimensional Gaussian sputtering obtains the 3D Gaussian three-dimensional model at the current moment; Figure 2 As shown, including:

[0133] S10, obtaining a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence;

[0134] S11, based on a deformation-aware key frame pool, estimating an initial camera pose of a color image frame at the current moment, and determining whether it is a key frame to be stored in a key frame pool; wherein the key frame pool maintains a relatively static regular space 3D Gaussian 3D model;

[0135] If the current color image frame is not a key frame, then

[0136] S1211, taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, starting a dynamic scene optimizer to estimate a deformation field for converting the regular space 3D Gaussian 3D model into the real-time space 3D Gaussian 3D model;

[0137] S1212, taking the color image frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian three-dimensional model as input, and starting the pose optimizer to update the current camera pose;

[0138] S1213, alternately iteratively running the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least the real-time spatial 3D Gaussian three-dimensional model at the current moment;

[0139] If the current color image frame is a key frame, then

[0140] S1221, taking the color image frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, starting the static scene optimizer to update the regular space 3D Gaussian 3D model;

[0141] S1222, taking the color image frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model as input, and starting the pose optimizer to update the current camera pose;

[0142] S1223, alternately iteratively running the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least a regular space 3D Gaussian 3D model at the current moment;

[0143] S13: Select the real-time space 3D Gaussian 3D model corresponding to the last non-key frame up to the current moment as the 3D Gaussian 3D model at the current moment.

[0144] The embodiment of the present invention can establish a three-dimensional model of the overall environment when the color video frame sequence only captures the local in vivo environment and tissue deformation, thereby overcoming the disadvantage of the limited field of view of the laparoscope.

[0145] like Figure 3 As shown, Figure 3 The technical roadmap of the interactive surgical case guidance method provided by the embodiment of the present invention is given. Figure 3 The following are the steps of the above solution:

[0146] In step S1, based on the preoperative nii volume data, a preoperative CT model and a first grayscale value information set thereof are obtained, and point markings made by a doctor on the preoperative CT model are received to obtain a three-dimensional coordinate set of 3D marking points.

[0147] This step performs the reconstruction of the preoperative CT model, including:

[0148] Read preoperative nii volume data and obtain a three-dimensional matrix , the value stored in the matrix is ​​the intensity value information of the corresponding voxel block. In addition, in order to obtain gray value information that can be visualized, gray value encoding is performed.

[0149] The encoding formula is as follows:

[0150]

[0151] in, w Indicates window width, window width w is the range of intensity values ​​in the volume data; o Indicates window level, window level o Indicates the center of the intensity value in the volume data. It should be noted that for CT images, the intensity value is usually expressed in Hounsfield units (HU), and these HU values ​​can be mapped to grayscale values ​​after adjusting the window width and window level. Representation Matrix B d The stored voxel intensity values ​​are encoded and converted into a three-dimensional matrix of pixel values. Gray .

[0152] Afterwards, the three-dimensional matrix of the preoperative volume data B dAccording to the actual voxel size, it is mapped to the three-dimensional world coordinate system, where the coordinates of each point in the three-dimensional point set are sorted according to the rows, columns and depth of the matrix, that is, from the three-dimensional matrix B d ∈ The preoperative CT model changes to a point set of M×N×L , the specific formula is as follows:

[0153]

[0154] in, V_size Represents each voxel in xyz The actual physical dimensions in three directions, V_size x , V_size y and V_ size z are the sizes of the voxels in each direction, m, n, l Then xyz The dimension values ​​in three directions are selected. During the conversion process, the intensity value of the point will be filtered, that is, if Gray [m,n,l]=0, then the point is moved from The corresponding point set is deleted.

[0155] Similarly, the three-dimensional matrix Gray in accordance with The composition method is converted into a gray value information set of the corresponding points ;in for p i The voxel gray value corresponding to the point.

[0156] On this basis, the doctor reconstructed the preoperative CT model For example, mark the main characteristic parts of the target organ, and record the three-dimensional coordinate set of the 3D marking points as A 1.

[0157] In step S2, a color image frame sequence of laparoscopic surgery is obtained, the point markings made by the doctor on the color image frame at the current moment are received, and a 2D marker point uv coordinate set corresponding to the 3D marker point three-dimensional coordinate set is obtained. The Ransac-PnP algorithm is used to solve the rigid transformation matrix from the preoperative to the intraoperative coordinate system, and a three-dimensional point cloud set after rigid registration is obtained.

[0158] In this step, the doctor marks the color image frame at the current moment. The position of the mark point corresponds to the mark point before the operation, and the 2D mark point uv coordinate set is recorded as A 2.

[0159] Corresponding 3D-2D marker points before and during surgery A 1 and A 2. Solve the rigid transformation matrix T (including the rotation matrix) from the preoperative to the intraoperative coordinate system by using the Ransac-PnP algorithm R and translation vectors t ) to obtain a set of three-dimensional point clouds after rigid registration , the formula is as follows:

[0160]

[0161] in, Indicates the use of Algorithm to calculate the set of marked points A 1 and A 2, thus obtaining the rigid transformation matrix R and translation vectors t ; After rigid transformation R and t The coordinates of the point after .

[0162] In step S3, a laparoscopic surgery scene three-dimensional reconstruction method based on three-dimensional Gaussian sputtering is used to obtain a 3D Gaussian three-dimensional model at the current moment; and the three-dimensional point cloud set is non-rigidly aligned with the 3D Gaussian three-dimensional model at the current moment using the V2snet method to obtain an intraoperative CT model.

[0163] Before formally introducing the specific steps of the 3D reconstruction method of laparoscopic surgery scenes based on 3D Gaussian sputtering, it is necessary to supplement the relevant basic mathematical expressions:

[0164] (1) Mathematical expression of three-dimensional model:

[0165] In the embodiment of the present invention, a three-dimensional model of a scene (any symbol containing " " represents the three-dimensional model of the scene) consists of multiple 3D Gaussian G ( x ), the mathematical formula of each 3D Gaussian is as follows:

[0166]

[0167] Among them, 3D Gaussians are geometrically represented as ellipsoids in space, and the geometry of a three-dimensional model of a scene is "built" by these ellipsoids. x Represents the center point position of the three-dimensional Gaussian, that is, the mean of the three-dimensional Gaussian distribution, which is also the coordinate of the center point of the ellipsoid. R Represents the rotation of a three-dimensional Gaussian in space, using quaternions during optimization Q =( q 0 ,qx ,q y ,q z )express. S It represents the scale factor of the three-dimensional Gaussian on the X, Y, and Z axes, that is, the radius of the ellipsoid on its X, Y, and Z axes.

[0168] Considering that the above formula only describes the shape of the 3D Gaussian, and each 3D Gaussian also carries color information and transparency information related to the viewing angle, the embodiment of the present invention further makes the following settings:

[0169] Each 3D Gaussian transparency uses α express, α ∈[0,1], color information uses three sets of spherical harmonic coefficients c lm Respectively represent the color values ​​of the three channels of RGB, l is the order of the spherical harmonic coefficients, m is an integer, - l≤m≤l .

[0170] Among them, the spherical harmonic coefficients are the spherical harmonic functions The form and number of spherical harmonics are determined after the maximum order L is determined. Therefore, the values ​​of the three color channels RGB can be represented by three sets of spherical harmonic coefficients. For example, the value of the R channel can be expressed as: ,in θ represents the zenith angle, φ Represents the azimuth, and these two angles together represent the direction of the viewing angle.

[0171] (2) Mathematical expression of deformation field:

[0172] The deformation field acts on the center point of each three-dimensional Gaussian ellipsoid, causing it to shift in space. The embodiment of the present invention uses three sets of Fourier series , Fit this deformation, each set of Fourier series fits the deformation of one dimension. For example, the displacement in the X dimension It can be calculated by the following formula:

[0173]

[0174] Similarly, the displacement in the Y dimension can be calculated using formula (2): v Y And the displacement in the Z dimension v z , and define the deformation field Wt ( x ):

[0175]

[0176] Formula (4) represents the effect on the center point x The displacement vector of 3D Gaussian and deformation field is given in the following specific form:

[0177]

[0178] in, Represents the three-dimensional coordinates of the center point after displacement, Represents the deformed 3D Gaussian.

[0179] (3) Mathematical expression of camera pose:

[0180] The embodiment of the present invention uses quaternions and translation vectors to express the camera pose, that is, ,in P t Represents the camera pose, which consists of a quaternion that expresses the camera rotation q t and a translation vector expressing the camera translation t t express.

[0181] like Figure 4 As shown, Figure 4 A complete technical roadmap of the laparoscopic surgery scene three-dimensional reconstruction method provided by the embodiment of the present invention is given. Figure 4 , a laparoscopic surgery scene 3D reconstruction method based on 3D Gaussian sputtering, and obtaining a corresponding real-time space 3D Gaussian 3D model; including:

[0182] In step S10, a color image frame sequence of laparoscopic surgery and a corresponding depth map sequence are obtained.

[0183] In this step, a color image frame sequence of laparoscopic surgery is obtained. and its corresponding depth map sequence, , where t=0,1,2,... represents a time series. Exemplarily, the depth map here can be obtained using a depth estimation algorithm such as MIOL.

[0184] In step S11, based on the deformation-aware key frame pool, the initial camera pose of the color image frame at the current moment is estimated, and it is determined whether it is a key frame that needs to be stored in the key frame pool; wherein the key frame pool maintains a relatively static regular space 3D Gaussian three-dimensional model.

[0185] First of all, it should be clarified that the embodiment of the present invention defines the above deformation-aware key frame pool to have the following functions:

[0186] (1) Identify and store color image frames with relatively small deformation and high quality As keyframe I k , the set of all key frames is is the key frame index. The information in the key frame pool maintains a static 3D model information, called the canonical space. In contrast to the canonical space, the space where the dynamic 3D model is located is called the real-time space.

[0187] (2) Calculate the initial camera pose corresponding to the color image frame at the current moment .

[0188] On this basis, this step specifically includes:

[0189] S111, extracting SIFT feature descriptors of the color image frame, traversing all key frames in the matching key frame pool, and taking the key frame with the most matching key point pairs as the best matching frame.

[0190] Specifically: First, use the key point recognition algorithm to obtain the key points (i.e. points with obvious features, such as edge points of objects) on any pair of matching images; then, extract the SIFT feature descriptors of these key points, and use the SIFT feature descriptors to find matching point pairs among these key points (i.e., if the vector angle of the SIFT feature descriptors between two key points is less than a certain threshold, then this is considered to be a pair of matching key points); finally, take the key frame with the most matching key point pairs as the best matching frame.

[0191] S112, calculating the pose transformation between the color image frame and the best matching frame to obtain the initial camera pose.

[0192] For example, 3D-2D PNP (Perspective N Points) is used to calculate the pose transformation between the color image frame and the best matching frame, thereby obtaining the initial camera pose. , as the initial value for subsequent pose optimization.

[0193] S113, estimating the deformation degree of the color image frame based on the FPFH feature descriptor of the matching key point pair between the color image frame and the best matching frame, and storing the estimated deformation measurement value in a buffer pool.

[0194] Specifically, after obtaining the best matching frame in S21, based on the pixel coordinates of the matching key point pairs between the color image frame and the best matching frame, the three-dimensional coordinates of these matching key point pairs are restored using the depth map data, and then the FPFH feature descriptor of the matching key points can be obtained using the traditional FPFH algorithm to estimate the deformation degree of the color image frame; the deformation estimation function is as follows:

[0195]

[0196] in, F nr represents the deformation metric value of the color image frame; and It is a pair of FPFH feature descriptors matching key points; N s Represents the number of matching key point pairs; Represents the F-norm of a matrix or vector.

[0197] S114, judging the quality of the color image frame; wherein the judging index includes the number of matching key points on the color image frame and / or the area of ​​the two-dimensional convex hull formed by the matching key points.

[0198] S115: If the deformation metric value of the color image frame at the current moment is less than the corresponding threshold and the quality is greater than the corresponding threshold, it is determined as a key frame and stored in a key frame pool.

[0199] If the color image frame at the current moment is a key frame, the key frame information will be stored in the key frame pool. The stored information includes but is not limited to the frame ID, initial camera pose, color image frame, depth map, key point coordinates, and SIFT feature descriptors of the key points.

[0200] S116: If no new key frame is stored in the key frame pool after the preset time, a color image frame with the smallest deformation metric value is searched in the current buffer pool and stored in the key frame pool, and the data in the buffer pool is cleared.

[0201] like Figure 4 As shown, for whether the color image frame at the current moment is a key frame or not, the embodiment of the present invention processes it according to different processing ideas to output the regular space 3D Gaussian 3D model or the real space 3D Gaussian 3D model at the current moment, as shown below:

[0202] If the current color image frame is not a key frame, then

[0203] In step S1211, the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment are taken as input, and the dynamic scene optimizer is started to estimate the deformation field used to convert the regular space 3D Gaussian 3D model into the real-time space 3D Gaussian 3D model.

[0204] The scene optimizer in the embodiment of the present invention has two types: dynamic scene optimizer OP dynamic and Static Scene Optimizer OP static , since the color video frame at the current moment is not a key frame, only the dynamic scene optimizer is started here. The dynamic scene optimizer has the following functions:

[0205] Compared to the static scene optimizer, the dynamic scene optimizer only estimates a deformation field W t , W t The regular space 3D Gaussian 3D model will be converted into a real space 3D Gaussian 3D model , which will be converted during the optimization process; the formula is as follows:

[0206]

[0207] In step S1212, the color image frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian model are used as input to start the pose optimizer to update the current camera pose.

[0208] Pose Optimizer OP pose It will be run during the scene optimizer iteration, for example, it will be run when the scene optimizer iterates T, 2T, 3T...nT times, so the input of the pose optimizer is the initial pose , current picture and And the 3D Gaussian 3D model output by the scene optimizer after iterating for a period of time. If the started scene optimizer is a static scene optimizer, the input 3D Gaussian 3D model is a regular space 3D model If the scene optimizer started is a dynamic scene optimizer, the input 3D Gaussian 3D model is a 3D model of real-time space. , the output of the pose optimizer is the pose of the current frame P t , the pose optimizer will output the best pose after optimization. The scene optimizer started here is a dynamic scene optimizer, and the formula is expressed as follows:

[0209]

[0210] In step S1213, the dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least the real-time spatial 3D Gaussian three-dimensional model at the current moment.

[0211] Exemplarily, the optimization solution methods of the two optimizers both use the Adam algorithm.

[0212] (1) Loss function and optimization parameters of the dynamic scene optimizer

[0213] Based on the fact that the optimizable parameter of the dynamic scene optimizer is the deformation field, the corresponding loss function is designed:

[0214]

[0215] in, L dynamic represents the total loss function of the dynamic scene optimizer; They represent the color map loss function and depth map loss function of the dynamic scene optimizer respectively;

[0216] Deformation Field W t The 3D Gaussian model of the regular space at the previous moment The deformation effect;

[0217] Represents a 3D Gaussian i The three-dimensional position of the center point in canonical space; KNN ( i ) represents a 3D Gaussian i The K nearest neighbors of Represents a neighboring 3D Gaussian j The three-dimensional position of the center point in canonical space; Represents 3D Gaussian i and neighboring 3D Gaussian j The three-dimensional position of the center point in real space;

[0218] w ij Represents a 3D Gaussian i and neighboring 3D Gaussian j The connection weight between them; exp represents an exponential function with the natural constant e as the base; λ is the weight hyperparameter used to adjust w ij The value of Represents the bi-norm of a vector.

[0219] About the loss function and optimization parameters of the pose optimizer

[0220] Based on the optimizable parameters of the pose optimizer as the current camera pose, the corresponding loss function is designed:

[0221]

[0222] in, L pose represents the total loss function of the pose optimizer; L c 、L d They represent the color image loss function and depth image loss function of the pose optimizer respectively;

[0223] , Respectively represent the color image frame and its depth map at the current moment;

[0224] Represents the F norm of a matrix or vector; express and SSIM loss;

[0225] A differentiable renderer representing a 3D Gaussian splatter, used to render 3D Gaussian 3D models With the current camera pose P t Rendering into predicted color video frames and the predicted depth map ;

[0226] , They respectively represent the current regular space 3D Gaussian 3D model and the current real-time space 3D Gaussian 3D model.

[0227] Second, if the current color image frame is a key frame, then

[0228] In step S1221, the color image frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment are taken as input to start the static scene optimizer to update the regular space 3D Gaussian 3D model.

[0229] The scene optimizer in the embodiment of the present invention has two types: dynamic scene optimizer OP dynamic and Static Scene Optimizer OP static ,Since the color image frame at the current moment is not a key frame, only the dynamic scene optimizer is started here.

[0230] For the static scene optimizer, on the one hand, the color image frame at the current moment is used to optimize and generate incremental information (Incremental information refers to the new geometric and texture information carried by the latest frame compared to the previous image sequence); on the other hand, the previous regular space 3D Gaussian 3D model is used , And the camera pose output by the pose optimizer P t , modeling the latest regular space 3D Gaussian 3D model , the formula is as follows:

[0231]

[0232] It should be noted that the incremental information will be directly added to the Gaussian 3D model during the optimization process. The color, shape and other properties of the 3D Gaussian will also be optimized to make it consistent with the observation of the latest frame video.

[0233] In step S1222, the color image frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian three-dimensional model are used as input to start the pose optimizer to update the current camera pose.

[0234] Similarly, since the scene optimizer started here is a static scene optimizer, the formula is expressed as follows:

[0235]

[0236] In step S1223, the dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least the regular space 3D Gaussian three-dimensional model at the current moment.

[0237] Exemplarily, the optimization solution methods of the two optimizers both use the Adam algorithm.

[0238] (1) Loss function and optimization parameters of static scene optimizer

[0239] Based on the fact that the optimizable parameters of the static scene optimizer are the properties of the regular space 3D Gaussian 3D model at the previous moment, the corresponding loss function is designed:

[0240]

[0241] in, L static represents the total loss function of the static scene optimizer; They represent the color map loss and depth map loss of the static scene optimizer respectively;

[0242] is the property of the regular space 3D Gaussian three-dimensional model at the previous moment; the color of each 3D Gaussian adopts the spherical harmonic function coefficient c lm Indicates that the shape is S Matrix and R The matrix represents R Matrix as Quaternion , transparency is adopted α Indicates that the center point position is x express.

[0243] (2) About the loss function and optimization parameters of the pose optimizer

[0244] It can be understood that the content of this part can be completely referred to the content of formulas (22) to (25), and will not be repeated here.

[0245] In step S13, the real-time space 3D Gaussian 3D model corresponding to the last non-key frame up to the current moment is selected as the 3D Gaussian 3D model at the current moment.

[0246] In particular, during the operation of the static scene optimizer, the embodiment of the present invention also performs a densification process to increase the number of 3D Gaussians in the regular space 3D Gaussian three-dimensional model to obtain incremental information. .

[0247] Specifically, there are two densification methods: segmentation and cloning. The Adam algorithm calculates the gradient for the 3D Gaussian center point visible from the perspective of each color video frame at the current moment. After a period of iteration, if the average gradient obtained by a 3D Gaussian is greater than a certain threshold, it is considered to need to be densified. Among them:

[0248] If the Gaussian is too large, that is, the maximum value in the Gaussian S matrix is ​​greater than or equal to the preset threshold, segmentation densification is performed. The segmentation operation splits the Gaussian into two. For example, the S of these two 3D Gaussians will be multiplied by 0.8, and the center point coordinates of the new 3D Gaussian are the center point coordinates of the original 3D Gaussian plus a bias randomly sampled from a three-dimensional Gaussian distribution with a mean of 0, and the remaining attributes are copied. If the Gaussian is too small, that is, the maximum value in the Gaussian S matrix is ​​less than the preset threshold, cloning densification is performed, that is, copying an identical 3D Gaussian at the original position. The new 3D Gaussians generated by these cloning and segmentation operations are incremental information .

[0249] Since the incremental information is generated in the scene optimizer process, the incremental information, that is, the newly generated 3D Gaussian, will be further optimized, and its properties such as shape, size and color will be optimized to be consistent with the "length" of the color video frame.

[0250] Thus, the embodiment of the present invention designs a three-dimensional reconstruction framework for complex in vivo environments, which can establish a three-dimensional model of the overall environment when the color video frame sequence only captures the local in vivo environment and tissue deformation, thus overcoming the disadvantage of limited laparoscope field of view.

[0251] In addition, in an embodiment of the present invention, depth map information that provides geometric surface information is introduced and used to supervise the center point coordinates of the 3D Gaussians so that they are bound around the surface, greatly improving the three-dimensional reconstruction accuracy of the tissue surface. The addition of the depth map provides a geometric prior for all 3D Gaussians, making it unnecessary to adjust the 3D Gaussian density.

[0252] It is worth pointing out that if Figure 4As shown, in addition to outputting the 3D Gaussian model at the current moment, the embodiment of the present invention can also simultaneously output the optimal camera pose of the color video frame at the current moment. In other words, the embodiment of the present invention can simultaneously estimate the accurate camera pose and tissue deformation information, providing support information for motion planning for robotic surgery.

[0253] The solution provided in the embodiment of the present invention can be used for simulating real surgical scenes, and combined with VR display technology can help novice doctors perform surgical training. Figure 5 Demonstrated the effect of reconstructing the real in vivo environment.

[0254] After completing the rigid registration, and combining the 3D Gaussian model of the current moment obtained above, this step adopts the volume-to-surface non-rigid registration method V2snet , using the 3D point cloud set after rigid registration and a 3D Gaussian model provided by a 3D reconstruction method for laparoscopic surgery scenes based on 3D Gaussian sputtering The center point coordinate information collection , predict the subsequent flexible deformation displacement field of the 3D model after rigid registration V t , Figure 6 The rigid registration effect of the real in vivo environment is demonstrated; the formula is as follows:

[0255]

[0256] in, , V i express Middle i The displacement vector of a point, N pre Indicates the number of points, Represents the set of real numbers.

[0257] After completing the non-rigid registration, the flexible deformation displacement field can be used V t To obtain an intraoperative CT model accurately registered to the point cloud form , the formula is as follows:

[0258]

[0259] in, express Middle i The three-dimensional coordinates of a point.

[0260] In addition, the relationship between each point of the intraoperative CT model and its corresponding preoperative voxel grayscale value is as follows:

[0261]

[0262] in, N pre is the number of 3D model points, and G pre That is, the coordinate set of the intraoperative CT model and the preoperative voxel grayscale value information set corresponding to each coordinate point. It can be understood that G pre There is no change during the preoperative and intraoperative phases, so and G pr The information also corresponds one to one, such as the coordinate point The corresponding grayscale value is .

[0263] Note that the intraoperative CT model and grayscale value information set G pre Contains the internal deep structure information of tissues and organs, including the three-dimensional position of each point in the internal structure of the organ and the corresponding grayscale value; and the 3D Gaussian model at the current time frame It represents the surface structure information of tissues and organs.

[0264] In step S4, the intraoperative CT model is initialized with a CT Gaussian to obtain a CT Gaussian model.

[0265] The CT Gaussian model of the embodiment of the present invention is represented as , whose properties include the center point location of each CT Gaussian , Opacity α , color information (spherical harmonic coefficients c lm ) and shape information ( S i Matrix and R matrix, R Matrix with Quaternions denoted); the optimizable parameters of Gauss in the scene are expressed as follows:

[0266]

[0267] Specifically, the step of performing CT Gaussian initialization on the intraoperative CT model to obtain a CT Gaussian model includes:

[0268] Intraoperative CT model The three-dimensional coordinates of each point in are used as the center point coordinates of the initial CT Gaussian.

[0269] The opacity of each initial CT Gaussian is used α express, α ∈[0,1].

[0270] For the color information of each initial CT Gaussian, a set of spherical harmonic coefficients is used c lm express The single-channel grayscale value corresponding to each point in the spherical harmonic function is as follows:

[0271]

[0272] in, l is the order of the spherical harmonic coefficients, m is an integer, - l ≤ m ≤ l Therefore, a set of spherical harmonic coefficients can be used to represent the color value corresponding to the point.

[0273] For each initial CT Gaussian shape information:

[0274] R Represents the rotation of the initial CT Gaussian in space, using quaternion during optimization Q =( q 0 ,q x ,q y ,q z )express.

[0275] S i Indicates i The scale coefficients of the initial CT Gaussian on the X, Y, and Z axes, that is, the radius of the three-dimensional Gaussian ellipsoid on its X, Y, and Z axes, are used in the optimization process. KNN The algorithm finds the nearest Gaussian center points to each point on the model and passes Max The function finds the maximum value among the distances to obtain the scaling coefficient for each initial CT Gaussian.

[0276] For example, the radius of each three-dimensional Gaussian is determined by the one with the largest distance among the three Gaussian kernels closest to it, and the formula is expressed as follows:

[0277]

[0278] in, KNN (*) indicates that this 3D model is KNN The algorithm finds the three Gaussian centers closest to each point on the model; indices i , distances i Respectively represent iThe index of the Gaussian point and its distance to the nearest three Gaussian points; find the maximum value of the distance through the function Max(*) to obtain the scale coefficient corresponding to each axis of the three-dimensional Gaussian S .

[0279] In step S5, the intraoperative CT model is interpolated, and the interpolated point cloud model is used to render the corresponding CT grayscale image and depth map based on the preset camera pose; and the CT grayscale image and depth map are used as supervision signals to optimize the CT Gaussian model.

[0280] Since there are two different Gaussian models in the scene, and the CT Gaussian model is obtained by initialization, but its properties such as color are not optimized, it will cause the artificially marked area to appear blurred and the color is not smooth after rendering. Therefore, in order to optimize the CT Gaussian model in the scene, the embodiment of the present invention optimizes the CT model in the scene. Interpolation is performed to improve the scene resolution, and the optimization of the CT Gaussian model is supervised based on the image rendered by the interpolated point cloud model based on the given camera pose.

[0281] Specifically, this step includes:

[0282] (1) Point cloud during interpolation

[0283] First, combined with the first gray value information set, the intraoperative CT model is transformed into Modeled as a colored mesh ; expressed as:

[0284]

[0285] in, Me i Indicates the first i The coordinate set of the four vertices of a tetrahedron, C i Indicates the first i The color sets of the four vertices of a tetrahedron. For the i The tetrahedron j The three-dimensional coordinates of the vertices, For the i The tetrahedron j The grayscale value of each vertex. That is the set of three-dimensional coordinates and color values ​​of each tetrahedron after triangulation.

[0286] Next, interpolate in each grid to obtain the interpolated point cloud model PC ; wherein the interpolated point cloud model PCIncludes a 3D point cloud coordinate set and a 3D point cloud color set:

[0287]

[0288] in, G i represents the three-dimensional coordinates of each interpolated point cloud, C i Represents the color value of each interpolated point cloud, For the i The tetrahedron j The interpolation weights for each vertex.

[0289] Finally, the interpolated point cloud coordinate sets and color sets are as follows:

[0290]

[0291] in, me final represents the coordinate set of all three-dimensional points after interpolation, c final Represents the color value set of all 3D points after interpolation. The two are combined to form the coordinate and color information set of the point cloud. The formula is as follows:

[0292]

[0293] in, x pc express me final The three-dimensional point coordinates in Represents the point cloud color set c final The color value in PC That is, it is a point cloud information set consisting of these three-dimensional point cloud coordinates and three-dimensional point cloud colors.

[0294] (2) Camera pose sampling and intraoperative CT Gaussian optimization

[0295] First, a spherical coordinate system is set. The radius range of the spherical coordinate system is set to the distance between the farthest CT Gaussians. The polar angle range is set according to the field of view of the camera. The azimuth angle range is set to the front of the model. M camera positions are uniformly sampled on the spherical coordinate system and converted into Cartesian coordinates. The conversion formula is as follows:

[0296]

[0297] in, is the transformed camera position, r is the radius, θ is the polar angle, ϕ is the azimuth.

[0298] Then, the camera's viewing angle is aligned with the center of the Gaussian 3D model, and the camera is offset up, down, left, and right in the viewing direction. a degrees, expanding each camera position to a The camera poses are finally sampled. aM Camera poses .

[0299] For example, here a You can take 5.

[0300] After sampling, the interpolated 3D point cloud PC exist aM Projection of camera poses:

[0301]

[0302] in, K is the camera internal parameter, Project (*) To project the point cloud from 3D space to 2D image according to the camera model formula to obtain CT grayscale image and depth map . Indicated in i The 3D model obtained under the sampled camera poses PC Rendering image of Indicated in i The 3D model obtained under the sampled camera poses PC Depth map of .

[0303] Finally, the rendered CT grayscale image and depth map are used as pseudo-true values ​​to supervise the CT Gaussian model. Optimize to obtain the optimized CT Gaussian model .

[0304] In step S6, the ROI area marked by the doctor on the color image frame at the current moment is received, and the pixel curve of the edge of the area is restored to the three-dimensional space to obtain the section point information set of the intraoperative CT model; and based on the first gray value information set, a second gray value information set of the section point information set is obtained.

[0305] This step performs data acquisition of the marked area during surgery.

[0306] For example, the doctor can manually draw the corresponding ROI area on the display screen using a digital pen. The pixel curve of the edge of the ROI area can be restored to the three-dimensional space by the camera internal parameters and the depth map information, and the formula is as follows:

[0307]

[0308] in, A set of spatial coordinates representing a three-dimensional curve; Indicates the pixel curve i The two-dimensional homogeneous coordinates of pixels; Restore the two-dimensional pixel curve to the three-dimensional coordinates of the three-dimensional space; T represents transposition; N label is the total number of pixels of the pixel curve; z i A two-dimensional point The corresponding depth value in three-dimensional space; K ∈ Represents the camera intrinsic parameter matrix.

[0309] Next, in the three-dimensional curve Randomly sample three points , , , forming two vectors and , and calculate the normal vector of the corresponding section , using sampling points q a The three-dimensional coordinates of Th ; The formula is as follows:

[0310]

[0311] in, Respectively represented as normal vector Component in the xyz direction.

[0312]

[0313] Among them, := means Th is defined as the equation expression that follows.

[0314] Finally, based on the tangent plane equation Th Intraoperative CT Model , filter the coordinates of the model points located at the section, and obtain the section point information set P tan ; The formula is as follows:

[0315]

[0316] in, x i It is an intraoperative CT model The point in Th ( xi )express x i Also for the section Th Therefore, P tan That is, it represents the coordinate set of sampling points on the section. Gray tan The coordinates of the tangent point P tan The corresponding gray value information set, N tan is the total number of sampling points on the slice. For example, the coordinate point x i The corresponding grayscale value is gray i .

[0317] It should be noted that the purpose of obtaining the information of the sampling points of the section is to screen the CT Gaussian at the position plane of the deep structure of the corresponding organ.

[0318] In step S7, based on the GPU sorting and block rendering strategy, the 3D Gaussian model at the current moment and the optimized CT Gaussian model are separately rendered to obtain a fused image of the original image and the CT image for interactive surgical case guidance; wherein:

[0319] The volume rendering method is used to render the 3D Gaussian outside the ROI area;

[0320] The coordinate z component of each sampling point in the section point information set is used as the depth value, and combined with the corresponding second gray value information, the CT Gaussian at the corresponding depth value of the ROI area is rendered.

[0321] This step aims at the two Gaussian models (including 3D Gaussian and CT Gaussian) in the three-dimensional surgical scene, and achieves accurate separation and unobstructed rendering of the two Gaussian models through efficient GPU sorting and tile-based rendering strategy. This method effectively solves the problem of mismatch between the rendered image and the required image caused by front and back occlusion between the two Gaussian models in traditional rendering methods.

[0322] Specifically, the GPU sorting and block rendering strategy is based on separately rendering the current 3D Gaussian model and the optimized CT Gaussian model; including:

[0323] (1) Screen segmentation and block numbering: The screen displaying the color image frame is divided into m × n blocks, each containing block_x × block_y pixels, and numbered in sequence;

[0324] (2) Screening of blocks in the marked area: Based on the pixel value of the ROI area on the screen, obtain the corresponding block on the screen and its ID number;

[0325] (3) Gaussian model classification and encoding: Two Gaussian ID representations are introduced to distinguish 3D Gaussian and CT Gaussian respectively;

[0326] 3D Gaussian is represented as [3D_GAUSSIAN, ID, depth], and CT Gaussian is represented as [CT_GAUSSIAN, ID, depth], and they are represented by 16 bits, 16 bits, and 32 bits of binary code respectively; depth represents the depth value of Gaussian in the scene;

[0327] (4) GPU quick sorting and rendering: The 3D Gaussian and CT Gaussian lists are quickly sorted on the GPU, and the tiles are rendered separately based on their ID numbers and depth values ​​according to the Gaussian sorting results.

[0328] This method can ensure that the CT Gaussian is accurately presented at the cross-section, while avoiding the image rendering blur and distortion caused by mutual occlusion with the 3D Gaussian model. This method also allocates the Gaussian to the thread memory of each tile. The parallel processing capability of the GPU greatly improves the sorting and rendering efficiency.

[0329] Specifically, the separate rendering is as follows:

[0330] First, the volume rendering method is used to render the 3D Gaussian outside the ROI area; it is expressed as:

[0331]

[0332] in, c Indicates the RGB color value finally rendered by the light; c i Indicated in i RGB color value at each sampling point; N Indicates the number of sampling points; α i Indicates the light i The opacity at each sampling point; Indicates that the light reaches i The probability that a sampling point has not been blocked by all previous sampling points.

[0333] Second, using the coordinate z component of each sampling point in the section point information set as the depth value, combined with the corresponding second gray value information, the CT Gaussian at the corresponding depth value of the ROI area is rendered; expressed as:

[0334]

[0335] in, gray Indicates the grayscale value finally rendered by the light; gray i Indicated in i The gray value at the sampling point.

[0336] Finally, the renderer in this method combines the two Gaussian models and Separate rendering. The pixels outside the manually marked ROI area are directly rendered by 3D Gaussian to obtain the original image part of the organ tissue; while the pixels inside the manually marked area are rendered by the screened CT Gaussian to obtain the corresponding CT image part of the deep tissue structure. The two parts are rendered simultaneously to obtain a fusion image of the original image and CT image required for observation, which can be used for interactive surgical case guidance.

[0337] So far, the embodiment of the present invention has completed all the steps of interactive surgical case guidance based on manual annotation guidance, and has achieved the training of novice doctors for different surgical cases outside of surgery, avoiding the possibility that doctors may not be able to directly observe deep tissue structures during actual minimally invasive surgery due to the limitations of instruments and the limited intraoperative field of view. It helps doctors to observe in detail the changes in tissues at different times and angles and important surgical data information, such as CT slice data, from the system, so that novice doctors can better understand the surgical characteristics of different surgical cases and improve their surgical experience cognition.

[0338] Embodiment 2:

[0339] The embodiment of the present invention provides an interactive surgical case guidance system based on manual annotation guidance, comprising:

[0340] A reconstruction module, used to obtain a preoperative CT model and a first grayscale value information set thereof based on the preoperative nii volume data, and receive a point mark made by a doctor on the preoperative CT model to obtain a three-dimensional coordinate set of a 3D mark point;

[0341] A registration module is used to obtain a color image frame sequence of laparoscopic surgery, receive the point marks made by the doctor on the color image frame at the current moment, obtain a 2D marker point uv coordinate set corresponding to the 3D marker point 3D coordinate set, use the Ransac-PnP algorithm to solve the rigid transformation matrix from the preoperative to the intraoperative coordinate system, and obtain a 3D point cloud set after rigid registration;

[0342] A method for three-dimensional reconstruction of a laparoscopic surgery scene based on three-dimensional Gaussian sputtering is provided to obtain a 3D Gaussian three-dimensional model at the current moment; and a V2snet method is used to non-rigidly align the three-dimensional point cloud set with the 3D Gaussian three-dimensional model at the current moment to obtain an intraoperative CT model;

[0343] An optimization module, used for performing CT Gaussian initialization on the intraoperative CT model to obtain a CT Gaussian model;

[0344] And used to interpolate the intraoperative CT model, use the interpolated point cloud model, render based on the preset camera pose to obtain the corresponding CT grayscale image and depth map; and use the CT grayscale image and depth map as supervision signals to optimize the CT Gaussian model;

[0345] A sampling module is used to receive the ROI area marked by the doctor on the color image frame at the current moment, restore the pixel curve of the edge of the area to the three-dimensional space, so as to obtain the section point information set of the intraoperative CT model; and obtain the second gray value information set of the section point information set based on the first gray value information set;

[0346] The rendering module is used to separately render the current 3D Gaussian model and the optimized CT Gaussian model based on GPU sorting and block rendering strategy to obtain a fused image of the original image and the CT image for interactive surgical case guidance; wherein:

[0347] The volume rendering method is used to render the 3D Gaussian outside the ROI area;

[0348] The coordinate z component of each sampling point in the section point information set is used as the depth value, and combined with the corresponding second gray value information, the CT Gaussian at the corresponding depth value of the ROI area is rendered.

[0349] Embodiment 3:

[0350] An embodiment of the present invention provides a storage medium storing a computer program for interactive surgical case guidance based on manual annotation guidance, wherein the computer program enables a computer to execute the interactive surgical case guidance method as described above.

[0351] Embodiment 4:

[0352] An embodiment of the present invention provides an electronic device, including:

[0353] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the interactive surgical case guidance method as described above.

[0354] It is understandable that the interactive surgical case guidance system, storage medium and electronic device based on manual annotation guidance provided in the embodiments of the present invention correspond to the interactive surgical case guidance method based on manual annotation guidance provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the interactive surgical case guidance method and will not be repeated here.

[0355] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0356] 1. In the embodiment of the present invention, feature marker points are selected on the preoperative CT model, and corresponding intraoperative marker points are selected on the color image frame to achieve the initial rigid registration of the preoperative to intraoperative model; then, non-rigid registration is performed based on the 3D Gaussian three-dimensional model rendered by three-dimensional Gaussian sputtering to obtain the intraoperative CT model, so as to achieve comprehensive monitoring and recording of the intraoperative operation process; then, the intraoperative CT model is Gaussian initialized and optimized; then, the ROI area marked by the doctor on the color image frame at the intraoperative moment is received, and 3D Gaussian and CT Gaussian sampling are completed respectively; finally, two Gaussian models are separately rendered based on the GPU sorting and block rendering strategy to obtain the fusion image of the original image and CT image for interactive surgical case guidance. The present invention can present more accurate tissue anatomical information and achieve accurate decision-making, which is especially helpful for novice doctors to observe the surgical characteristics of different surgical cases at different surgical moments and improve surgical experience cognition.

[0357] 2. The three-dimensional reconstruction method of the laparoscopic surgical scene based on three-dimensional Gaussian sputtering proposed in the embodiment of the present invention, when the color video frame at the current moment is not a key frame, starts the dynamic scene optimizer to estimate the deformation field used to convert the regular space 3D Gaussian three-dimensional model into the real-time space 3D Gaussian three-dimensional model; then starts the pose optimizer to update the current camera pose; and alternately iterates and runs the two optimizers until both converge, and outputs the real-time space 3D Gaussian three-dimensional model at the current moment. That is, it can generate high-fidelity dynamic in vivo three-dimensional scenes, which is convenient for doctors to observe the patient's internal environment. In addition, depth map information that can provide geometric surface information is introduced, and it is used to supervise the center point coordinates of the 3D Gaussian so that it is bound around the surface, which greatly improves the three-dimensional reconstruction accuracy of the tissue surface.

[0358] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0359] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An interactive surgical case guidance method based on manual annotation guidance, characterized in that: include: Based on the preoperative nii volume data, a preoperative CT model and a first grayscale value information set thereof are obtained, and a point mark on the preoperative CT model by a doctor is received to obtain a three-dimensional coordinate set of a 3D mark point; Acquire a color image frame sequence of laparoscopic surgery, receive the point markings made by the doctor on the color image frame at the current moment, obtain a 2D marker point uv coordinate set corresponding to the 3D marker point 3D coordinate set, use the Ransac-PnP algorithm to solve the rigid transformation matrix from the preoperative to the intraoperative coordinate system, and obtain a 3D point cloud set after rigid registration; A laparoscopic surgery scene 3D reconstruction method based on 3D Gaussian sputtering is used to obtain a 3D Gaussian 3D model at the current moment; and a V2snet method is used to non-rigidly align the 3D point cloud set with the 3D Gaussian 3D model at the current moment to obtain an intraoperative CT model; Performing CT Gaussian initialization on the intraoperative CT model to obtain a CT Gaussian model; Interpolate the intraoperative CT model, and use the interpolated point cloud model to render the corresponding CT grayscale image and depth map based on a preset camera pose; and use the CT grayscale image and depth map as supervisory signals to optimize the CT Gaussian model; The interactive surgical case guidance method further includes: Receive the ROI area marked by the doctor on the color image frame at the current moment, restore the pixel curve of the edge of the area to the three-dimensional space to obtain the section point information set of the intraoperative CT model; and obtain the second gray value information set of the section point information set based on the first gray value information set; Based on GPU sorting and block rendering strategy, the current 3D Gaussian model and the optimized CT Gaussian model are rendered separately to obtain a fusion image of the original image and the CT image for interactive surgical case guidance; wherein: The volume rendering method is used to render the 3D Gaussian outside the ROI area; Using the coordinate z component of each sampling point in the section point information set as a depth value, combined with the corresponding second gray value information, rendering the CT Gaussian at the depth value corresponding to the ROI area; The three-dimensional reconstruction method of the laparoscopic surgery scene based on three-dimensional Gaussian sputtering obtains the 3D Gaussian three-dimensional model at the current moment; comprising: Acquire a color image frame sequence of laparoscopic surgery and its corresponding depth map sequence; Based on the deformation-aware keyframe pool, the initial camera pose of the color image frame at the current moment is estimated, and whether it is a keyframe to be stored in the keyframe pool; wherein the keyframe pool maintains a relatively static regular space 3D Gaussian 3D model; If the current color image frame is not a key frame, then Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, a dynamic scene optimizer is started to estimate a deformation field for converting the regular space 3D Gaussian 3D model into the real space 3D Gaussian 3D model; The color image frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian model are used as input to start the pose optimizer to update the current camera pose; Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both are converged, so as to output at least a real-time spatial 3D Gaussian 3D model at the current moment; If the current color image frame is a key frame, then Taking the color image frame at the current moment and its depth map, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, the static scene optimizer is started to update the regular space 3D Gaussian 3D model; The color image frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model are used as input to start the pose optimizer to update the current camera pose; Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least a regular space 3D Gaussian 3D model at the current moment; The real-time space 3D Gaussian 3D model corresponding to the last non-key frame up to the current moment is selected as the 3D Gaussian 3D model at the current moment.

2. The interactive surgical case guidance method according to claim 1, characterized in that: The deformation-aware key frame pool estimates the initial camera pose of the color image frame at the current moment, and determines whether it is a key frame to be stored in the key frame pool; including: Extract the SIFT feature descriptor of the color image frame, traverse all the key frames in the matching key frame pool, and take the key frame with the most matching key point pairs as the best matching frame; Calculate the pose transformation between the color image frame and the best matching frame to obtain the initial camera pose; Based on the FPFH feature descriptor of the matching key point pair between the color image frame and the best matching frame, the deformation degree of the color image frame is estimated, and the estimated deformation measurement value is stored in the buffer pool; Based on the best matching frame, judging the quality of the color image frame; wherein the judging index includes the number of matching key points on the color image frame and / or the area of ​​the two-dimensional convex hull formed by the matching key points; If the deformation metric value of the color image frame at the current moment is less than the corresponding threshold and the quality is greater than the corresponding threshold, it is judged as a key frame and stored in the key frame pool; If no new key frame is stored in the key frame pool after the preset time, the color image frame with the smallest deformation metric value is found in the current buffer pool and stored in the key frame pool, and the data in the buffer pool is cleared; and / or Based on the optimizable parameters of the pose optimizer as the current camera pose, the corresponding loss function is designed: in, L pose represents the total loss function of the pose optimizer; L c 、L d They represent the color image loss function and depth image loss function of the pose optimizer respectively; , Respectively represent the color image frame and its depth map at the current moment; express and SSIM loss; A differentiable renderer representing a 3D Gaussian splatter, used to render 3D Gaussian 3D models With the current camera pose P t Rendering into predicted color video frames and the predicted depth map ; , Respectively represent the current regular space 3D Gaussian 3D model and the current real-time space 3D Gaussian 3D model; and / or Based on the fact that the optimizable parameter of the dynamic scene optimizer is the deformation field, the corresponding loss function is designed: in, L dynamic represents the total loss function of the dynamic scene optimizer; They represent the color map loss function and depth map loss function of the dynamic scene optimizer respectively; Deformation Field W t The 3D Gaussian model of the regular space at the previous moment The deformation effect; Represents a 3D Gaussian i The three-dimensional position of the center point in canonical space; KNN ( i ) represents a 3D Gaussian i The K nearest neighbors of Represents a neighboring 3D Gaussian j The three-dimensional position of the center point in canonical space; Represents 3D Gaussian i and neighboring 3D Gaussian j The three-dimensional position of the center point in real space; w ij Represents a 3D Gaussian i and neighboring 3D Gaussian j The connection weight between them; exp represents an exponential function with the natural constant e as the base; λ is the weight hyperparameter used to adjust w ij The value of represents the two-norm of a vector; and / or Based on the fact that the optimizable parameters of the static scene optimizer are the properties of the regular space 3D Gaussian 3D model at the previous moment, the corresponding loss function is designed: in, L static represents the total loss function of the static scene optimizer; They represent the color map loss and depth map loss of the static scene optimizer respectively; is the property of the regular space 3D Gaussian three-dimensional model at the previous moment; the color of each 3D Gaussian adopts the spherical harmonic function coefficient c lm Indicates that the shape is S Matrix and R The matrix represents R Matrix as Quaternion , transparency is adopted α Indicates that the center point position is x express; and / or During the operation of the static scene optimizer, a densification process is also performed to increase the number of 3D Gaussians in the regular space 3D Gaussian three-dimensional model.

3. The interactive surgical case guidance method according to claim 1, characterized in that: The V2snet method is used to non-rigidly register the three-dimensional point cloud set with the 3D Gaussian three-dimensional model at the current moment to obtain the intraoperative CT model; including: Based on 3D point cloud collection 3D Gaussian model at the current moment The center point coordinate information collection , the V2snet method is used to predict the flexible deformation displacement field of the three-dimensional point cloud set V t ; Based on the flexible deformation displacement field V t , obtain the intraoperative CT model in the form of point cloud ; and / or The CT Gaussian model is expressed as , whose attributes include the center point position, opacity, color information and shape information of each CT Gaussian; the CT Gaussian initialization of the intraoperative CT model to obtain the CT Gaussian model; including: Intraoperative CT model The three-dimensional coordinates of each point in are taken as the center point coordinates of the initial CT Gaussian; The opacity of each initial CT Gaussian is used α express, α ∈[0,1]; For the color information of each initial CT Gaussian, a set of spherical harmonic coefficients is used c lm express The single channel grayscale value corresponding to each point in ; For the shape information of each initial CT Gaussian, R Represents the rotation of the initial CT Gaussian in space, using quaternion during optimization Q =( q 0 ,q x ,q y ,q z )express; S i Indicates i The scale coefficients of the initial CT Gaussian on the X, Y, and Z axes are used in the optimization process. KNN The algorithm finds the nearest Gaussian center points to each point on the model and passes Max The function finds the maximum value among the distances to obtain the scale coefficient of each initial CT Gaussian; and / or The intraoperative CT model is interpolated, and the interpolated point cloud model is used to render the corresponding CT grayscale image and depth map based on a preset camera pose; and the CT grayscale image and depth map are used as supervisory signals to optimize the CT Gaussian model; including: Combined with the first gray value information set, the intraoperative CT model is transformed into Modeled as a colored mesh , and interpolate in each grid to obtain the interpolated point cloud model PC ; wherein the interpolated point cloud model PC Including 3D point cloud coordinate set and 3D point cloud color set; Set a spherical coordinate system, set the radius range of the spherical coordinate system to the distance between the farthest CT Gaussians, set the polar angle range according to the camera's field of view, and set the azimuth angle range to the front of the model. Sample M camera positions uniformly on the spherical coordinate system and convert them into Cartesian coordinates. Align the camera's viewing angle to the center of the Gaussian 3D model and offset it up, down, left, and right in that viewing angle. a degrees, expanding each camera position to a The camera poses are finally sampled. aM camera poses, and the interpolated 3D point cloud PC exist aM The camera poses are projected and rendered to obtain the PC CT grayscale image and depth map; The rendered CT grayscale image and depth map are used as pseudo-true values ​​to supervise the CT Gaussian model Optimize to obtain the optimized CT Gaussian model .

4. The interactive surgical case guidance method according to claim 3, characterized in that: The receiving doctor marks the ROI region on the color image frame at the current moment, and restores the pixel curve of the edge of the region to the three-dimensional space to obtain the section point information set of the intraoperative CT model; and based on the first gray value information set, obtaining a second gray value information set of the section point information set; include: Receive the ROI area marked by the doctor on the color image frame at the current moment, restore the pixel curve of the edge of the area to the three-dimensional space based on the camera internal parameters and depth map information, and obtain the three-dimensional curve ; In the three-dimensional curve Randomly sample three points , , , forming two vectors and , and calculate the normal vector of the corresponding section , using sampling points q a The three-dimensional coordinates of Th ; Based on the tangent plane equation Th Intraoperative CT model , filter the coordinates of the model points located at the section, and obtain the section point information set P tan ; In the first gray value information set G pre , filter the section point information set P tan The gray value information corresponding to each point in the gray value information set is obtained Gray tan .

5. The interactive surgical case guidance method according to claim 4, characterized in that: The GPU sorting and block rendering strategy is based on separately rendering the current 3D Gaussian model and the optimized CT Gaussian model; including: (1) Screen segmentation and block numbering: The screen displaying the color image frame is divided into m × n blocks, each containing block_x × block_y pixels, and numbered in sequence; (2) Screening of blocks in the marked area: Based on the pixel value of the ROI area on the screen, obtain the corresponding block on the screen and its ID number; (3) Gaussian model classification and encoding: Two Gaussian ID representations are introduced to distinguish 3D Gaussian and CT Gaussian respectively; 3D Gaussian is represented as [3D_GAUSSIAN, ID, depth], and CT Gaussian is represented as [CT_GAUSSIAN, ID, depth], and they are represented by 16 bits, 16 bits, and 32 bits of binary code respectively; depth represents the depth value of Gaussian in the scene; (4) GPU quick sorting and rendering: The 3D Gaussian and CT Gaussian lists are quickly sorted on the GPU, and the blocks are rendered separately based on their ID numbers and depth values ​​according to the Gaussian sorting results; wherein the separate rendering is as follows: The volume rendering method is used to render the 3D Gaussian outside the ROI area; The section point information set P tan The coordinate z component of each sampling point in is used as the depth value, and combined with the corresponding second gray value information, the CT Gaussian at the corresponding depth value of the ROI area is rendered.

6. The interactive surgical case guidance method according to claim 5, characterized in that: The volume rendering method is used to render the 3D Gaussian outside the ROI area; it is expressed as: in, c Indicates the RGB color value finally rendered by the light; c i Indicated in i RGB color value at each sampling point; N Indicates the number of sampling points; α i Indicates the light i The opacity at each sampling point; Indicates that the light reaches i The probability that a sampling point is not blocked by all previous sampling points; and / or The coordinate z component of each sampling point in the section point information set is used as the depth value, combined with the corresponding second gray value information, to render the CT Gaussian at the corresponding depth value of the ROI area; expressed as: in, gray Indicates the grayscale value finally rendered by the light; gray i Indicated in i The gray value at the sampling point.

7. An interactive surgical case guidance system based on manual annotation guidance, characterized in that: The method for executing the interactive surgical case guidance method according to claim 1 comprises: A reconstruction module, used to obtain a preoperative CT model and a first grayscale value information set thereof based on the preoperative nii volume data, and receive a point mark made by a doctor on the preoperative CT model to obtain a three-dimensional coordinate set of a 3D mark point; A registration module is used to obtain a color image frame sequence of laparoscopic surgery, receive the point marks made by the doctor on the color image frame at the current moment, obtain a 2D marker point uv coordinate set corresponding to the 3D marker point 3D coordinate set, use the Ransac-PnP algorithm to solve the rigid transformation matrix from the preoperative to the intraoperative coordinate system, and obtain a 3D point cloud set after rigid registration; A method for three-dimensional reconstruction of a laparoscopic surgery scene based on three-dimensional Gaussian sputtering is provided to obtain a 3D Gaussian three-dimensional model at the current moment; and a V2snet method is used to non-rigidly align the three-dimensional point cloud set with the 3D Gaussian three-dimensional model at the current moment to obtain an intraoperative CT model; An optimization module, used for performing CT Gaussian initialization on the intraoperative CT model to obtain a CT Gaussian model; And used to interpolate the intraoperative CT model, use the interpolated point cloud model, render based on the preset camera pose to obtain the corresponding CT grayscale image and depth map; and use the CT grayscale image and depth map as supervision signals to optimize the CT Gaussian model; A sampling module is used to receive the ROI area marked by the doctor on the color image frame at the current moment, restore the pixel curve of the edge of the area to the three-dimensional space, so as to obtain the section point information set of the intraoperative CT model; and obtain the second gray value information set of the section point information set based on the first gray value information set; The rendering module is used to separately render the current 3D Gaussian model and the optimized CT Gaussian model based on GPU sorting and block rendering strategy to obtain a fused image of the original image and the CT image for interactive surgical case guidance; wherein: The volume rendering method is used to render the 3D Gaussian outside the ROI area; The coordinate z component of each sampling point in the section point information set is used as the depth value, and combined with the corresponding second gray value information, the CT Gaussian at the corresponding depth value of the ROI area is rendered.

8. A storage medium, characterized in that: It stores a computer program for interactive surgical case guidance based on manual annotation guidance, wherein the computer program enables the computer to execute the interactive surgical case guidance method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include programs for executing the interactive surgical case guidance method as described in any one of claims 1 to 6.

Images