Postoperative review method and system for minimally invasive surgery based on multi-stage surgical scene registration

Abstract

The present invention provides a method, system, storage medium and electronic device for postoperative review of minimally invasive surgery based on multi-stage scene registration of surgery, and relates to the field of postoperative review. In the present invention, a color video frame sequence of the entire laparoscopic surgery process is first obtained; secondly, based on a preset surgical stage recognition network, each color video frame is classified into a video frame set of different surgical stages; again, a real-time space three-dimensional Gaussian three-dimensional model corresponding to each of the video frame sets is obtained; finally, each of the real-time space three-dimensional Gaussian three-dimensional models is subjected to multi-stage model registration to form a three-dimensional model of a multi-stage scene. The present invention realizes the registration of multiple scenes, which is convenient for doctors to intuitively observe the impact of surgical operations. In particular, the three-dimensional reconstruction method of laparoscopic surgery scenes based on three-dimensional Gaussian sputtering can establish a three-dimensional model of the overall environment when the color video frame sequence only captures the local in vivo environment and tissue deformation, thereby overcoming the disadvantage of limited laparoscope field of view.

Description

Technical Field

[0001] The present invention relates to the field of postoperative review, and in particular to a postoperative review method, system, storage medium and electronic device for minimally invasive surgery based on multi-stage surgery scene registration. Background Art

[0002] In modern medical practice, clinicians' postoperative review of minimally invasive surgery is an important part of the process. Minimally invasive surgery is usually performed with an endoscope or other imaging device, which can record the entire surgical process through video, providing the necessary conditions for a detailed review of every step and technical execution during the operation.

[0003] In related technologies, doctors replay surgical videos to find and analyze key steps and decision points in the operation, such as to evaluate the impact of a specific operation and compare the status before and after the operation. This usually involves watching the same clip multiple times, or comparing video screenshots at different time points. Through video review, doctors can evaluate the quality of surgical technique execution, such as the accuracy of cutting, the firmness of suturing, etc., as well as the immediate impact of the operation on the patient's physiological state. In addition, in educational and team learning environments, doctors may also use surgical videos as a basis for discussion to jointly analyze surgical strategies and techniques and discuss possible improvements.

[0004] However, since the surgical area is usually dynamic, there are mismatches between different screenshots, and the doctor can only align them in his mind based on his rich experience, and cannot visualize them intuitively. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides a method, system, storage medium and electronic device for postoperative review of minimally invasive surgery based on multi-stage surgery scene registration, which solves the technical problem of mismatch between different screenshots.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A postoperative review method for minimally invasive surgery based on multi-stage surgical scene registration, comprising:

[0010] Obtain color video frame sequences of the entire laparoscopic surgery process;

[0011] Based on a preset surgical stage recognition network, each color video frame is classified into a set of video frames of different surgical stages; wherein the surgical stages include surgical preparation stage A, pneumoperitoneum stage B, exploration and operation stage C, and cleaning and suturing stage D;

[0012] Acquire a real-time spatial three-dimensional Gaussian three-dimensional model corresponding to each of the video frame sets;

[0013] Each of the real-time spatial three-dimensional Gaussian three-dimensional models is subjected to multi-stage model registration to form a three-dimensional model of a multi-stage scene.

[0014] Preferably, the method of classifying each color video frame into a set of video frames of different surgical stages based on a preset surgical stage recognition network comprises:

[0015] Constructing the surgical stage recognition network; the surgical stage recognition network includes a three-dimensional spatial feature extraction channel, a two-dimensional spatial feature extraction channel, a temporal optical flow feature extraction channel and a softmax layer, and the extracted features of the three channels are output through the softmax layer to obtain a stage classification score;

[0016] The three-dimensional spatial feature extraction channel is used to extract the spatial geometric information of the point cloud model at different times using the PointNet network; the two-dimensional spatial feature extraction channel is composed of a plurality of convolutional layers, and is used to identify the instrument and tissue information in each of the color video frames; the temporal optical flow feature extraction channel is composed of a plurality of convolutional layers, and is used to extract the optical flow information of each of the color video frames relative to the previous frame;

[0017] The surgical stage recognition network is trained by using a cross entropy loss function until convergence, and the converged network is used to obtain video frame sets corresponding to the surgical preparation stage A, pneumoperitoneum stage B, exploration and operation stage C, and cleaning and suturing stage D, respectively. , , , ;

[0018] in, N A , N B , N C , N D Respectively represent the number of video frames in the corresponding set.

[0019] Preferably, for any of the video frame sets, a laparoscopic surgery scene 3D reconstruction method based on 3D Gaussian sputtering is used to obtain a corresponding real-time space 3D Gaussian 3D model; comprising:

[0020] Based on a color video frame sequence and a corresponding depth map sequence in the video frame set;

[0021] Based on the deformation-aware key frame pool, the initial camera pose of the color video frame at the current moment is estimated, and whether it is a key frame to be stored in the key frame pool; wherein the key frame pool maintains a relatively static regular space 3D Gaussian 3D model;

[0022] If the current color video frame is not a key frame, then

[0023] Taking the color video frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, a dynamic scene optimizer is started to estimate the deformation field used to transform the regular space 3D Gaussian 3D model into the real space 3D Gaussian 3D model;

[0024] The color video frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian model are used as input to start the pose optimizer to update the current camera pose.

[0025] Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both are converged, so as to output at least a real-time spatial 3D Gaussian 3D model at the current moment;

[0026] If the current color video frame is a key frame, then

[0027] Taking the color video frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, the static scene optimizer is started to update the regular space 3D Gaussian 3D model;

[0028] The color video frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model are used as input to start the pose optimizer to update the current camera pose;

[0029] Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least a regular space 3D Gaussian 3D model at the current moment;

[0030] The real-time space 3D Gaussian 3D model at the moment corresponding to the last non-key frame is selected as the real-time space 3D Gaussian 3D model corresponding to the video frame set.

[0031] Preferably, the deformation-aware key frame pool estimates the initial camera pose of the color video frame at the current moment, and determines whether it is a key frame to be stored in the key frame pool; including:

[0032] Extract the SIFT feature descriptor of the color video frame, traverse all the key frames in the matching key frame pool, and take the key frame with the most matching key point pairs as the best matching frame;

[0033] Calculate the pose transformation between the color video frame and the best matching frame to obtain the initial camera pose;

[0034] Based on the FPFH feature descriptor of the matching key point pair between the color video frame and the best matching frame, the deformation degree of the color video frame is estimated, and the estimated deformation measurement value is stored in the buffer pool;

[0035] Based on the best matching frame, judging the quality of the color video frame; wherein the judging index includes the number of matching key points on the color video frame and / or the area of ​​the two-dimensional convex hull formed by the matching key points;

[0036] If the deformation metric value of the color video frame at the current moment is less than the corresponding threshold and the quality is greater than the corresponding threshold, it is judged as a key frame and stored in the key frame pool;

[0037] If no new key frame is stored in the key frame pool after the preset time, the color video frame with the smallest deformation metric value is found in the current buffer pool and stored in the key frame pool, and the data in the buffer pool is cleared.

[0038] Preferably, the optimizable parameter of the pose optimizer is the current camera pose, and a corresponding loss function is designed:

[0039]

[0040] in, L pose represents the total loss function of the pose optimizer; L c , L d They represent the color image loss function and depth image loss function of the pose optimizer respectively;

[0041] Respectively represent the color video frame and its depth map at the current moment;

[0042] Represents the F norm of a matrix or vector;

[0043] express and SSIM loss;

[0044] A differentiable renderer representing a 3D Gaussian splatter, used to render 3D Gaussian 3D models With the current camera pose P t Rendering into predicted color video frames and the predicted depth map ;

[0045] , They respectively represent the current regular space 3D Gaussian 3D model and the current real-time space 3D Gaussian 3D model.

[0046] Preferably, based on the fact that the optimizable parameter of the dynamic scene optimizer is a deformation field, a corresponding loss function is designed:

[0047]

[0048] in, L dynamic represents the total loss function of the dynamic scene optimizer; They represent the color map loss function and depth map loss function of the dynamic scene optimizer respectively;

[0049] Deformation Field W t The 3D Gaussian model of the regular space at the previous moment The deformation effect;

[0050] Represents a 3D Gaussian i The three-dimensional position of the center point in canonical space; KNN ( i ) represents a 3D Gaussian i The K nearest neighbors of Represents a neighboring 3D Gaussian j The three-dimensional position of the center point in canonical space; Represents 3D Gaussian i and neighboring 3D Gaussian j The three-dimensional position of the center point in real space;

[0051] w ij Represents a 3D Gaussian i and neighboring 3D Gaussian j The connection weight between them; exp represents an exponential function with the natural constant e as the base; λ is the weight hyperparameter used to adjust w ij The value of represents the two-norm of a vector;

[0052] Preferably, based on the fact that the optimizable parameters of the static scene optimizer are the properties of the regular space 3D Gaussian 3D model at the previous moment, a corresponding loss function is designed:

[0053]

[0054] in, L staticrepresents the total loss function of the static scene optimizer; They represent the color map loss and depth map loss of the static scene optimizer respectively;

[0055] is the property of the regular space 3D Gaussian three-dimensional model at the previous moment; the color of each 3D Gaussian adopts the spherical harmonic function coefficient c lm Indicates that the shape is S Matrix and R The matrix represents R Matrix as Quaternion , transparency is adopted α Indicates that the center point position is x express.

[0056] Preferably, during the operation of the static scene optimizer, a densification process is also performed to increase the number of 3D Gaussians in the regular space 3D Gaussian three-dimensional model.

[0057] Preferably, the video frame sets , , , The corresponding real-time space 3D Gaussian 3D model is recorded as , , , ; The method of performing multi-stage model registration on each of the real-time spatial three-dimensional Gaussian three-dimensional models to form a three-dimensional model of a multi-stage scene; comprising:

[0058] Set a spherical coordinate system, set the radius range of the spherical coordinate system to the distance between the farthest three-dimensional Gaussians, set the polar angle range according to the camera's field of view, and set the azimuth angle range to the front of the model, and sample evenly on the spherical coordinate system. N The camera position is converted into Cartesian coordinates using the following conversion formula:

[0059]

[0060] in,( x cam , y cam , z cam ) represents the transformed camera position, r represents the radius, θ represents the polar angle, ϕ Indicates azimuth;

[0061] Align the camera's viewing angle to the center of the Gaussian 3D model and offset it up, down, left, and right in that viewing angle. adegrees, expanding each camera position to a The camera poses are finally sampled. aN Camera poses { P i};

[0062] After the sampling is completed, and exist aN Rendering with camera poses:

[0063]

[0064] in, Respectively expressed in i Rendering images of 3D model A and 3D model B obtained under the sampled camera poses;

[0065] Using the following energy function as the optimization objective, calculate arrive Transformation posture P BA ; The energy function is expressed as follows:

[0066]

[0067] Unify the multi-stage 3D Gaussian 3D models into the same coordinate system:

[0068]

[0069] in, Indicates that Unified to Three-dimensional Gaussian three-dimensional model after coordinate system; express The i 3D Gaussian; Represents the coordinates of the center point of the three-dimensional Gaussian; express Middle i The coordinates of the center point of a three-dimensional Gaussian; express P BA The 3*3 rotation matrix representing the quaternion transformation of the rotation; t BA Express posture P BA The translation vector in ; transform (⋅) means P BA Transformation equations into rotation matrices and translation vectors;

[0070] Similarly, get , , and finally obtain the aligned 3D model sequence , forming a three-dimensional model of a multi-stage scene.

[0071] A minimally invasive surgery postoperative review system based on multi-stage surgery scene registration, comprising:

[0072] An acquisition module is used to acquire a color video frame sequence of the entire laparoscopic surgery process;

[0073] A recognition module, configured to classify each color video frame into a set of video frames of different surgical stages based on a preset surgical stage recognition network; wherein the surgical stages include surgical preparation stage A, pneumoperitoneum stage B, exploration and operation stage C, and cleaning and suturing stage D;

[0074] A reconstruction module, used for obtaining a real-time spatial three-dimensional Gaussian three-dimensional model corresponding to each of the video frame sets;

[0075] The registration module is used to perform multi-stage model registration on each of the real-time spatial three-dimensional Gaussian three-dimensional models to form a three-dimensional model of a multi-stage scene.

[0076] A storage medium stores a computer program for postoperative review of minimally invasive surgery based on multi-stage surgery scene registration, wherein the computer program enables a computer to execute the postoperative review method of minimally invasive surgery as described above.

[0077] An electronic device, comprising:

[0078] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for performing a postoperative review of minimally invasive surgery as described above.

[0079] (III) Beneficial effects

[0080] The present invention provides a minimally invasive surgery postoperative review method, system, storage medium and electronic device based on multi-stage surgery scene registration. Compared with the prior art, it has the following beneficial effects:

[0081] In the present invention, firstly, a color video frame sequence of the whole process of laparoscopic surgery is obtained; secondly, based on a preset surgical stage recognition network, each color video frame is classified into a video frame set of different surgical stages; again, a real-time space three-dimensional Gaussian three-dimensional model corresponding to each of the video frame sets is obtained; finally, each of the real-time space three-dimensional Gaussian three-dimensional models is subjected to multi-stage model registration to form a three-dimensional model of a multi-stage scene. The present invention designs a postoperative multi-scene registration algorithm based on an intraoperative dynamic three-dimensional reconstruction framework, which can realize multi-scene registration by only obtaining color video frame information and tissue deformation of different surgical stages, so as to facilitate doctors to intuitively observe the impact of surgical operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0083] Figure 1 A block diagram of a minimally invasive surgery postoperative review method based on multi-stage surgery scene registration provided by an embodiment of the present invention;

[0084] Figure 2 A block diagram of a method for three-dimensional reconstruction of a laparoscopic surgery scene based on three-dimensional Gaussian sputtering provided in an embodiment of the present invention;

[0085] Figure 3 A complete technical roadmap for a minimally invasive surgery postoperative review method based on multi-stage surgery scene registration provided by an embodiment of the present invention;

[0086] Figure 4 An architecture diagram of a surgical stage recognition network provided by an embodiment of the present invention;

[0087] Figure 5 A complete technical roadmap for a method for three-dimensional reconstruction of a laparoscopic surgical scene provided by an embodiment of the present invention;

[0088] Figure 6 This is a reconstruction effect diagram of a real in vivo environment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0090] The embodiments of the present application solve the technical problem of mismatch between different screenshots by providing a method, system, storage medium and electronic device for postoperative review of minimally invasive surgery based on multi-stage surgery scene registration.

[0091] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0092] It is understandable that in the related art, doctors search for and analyze the key steps and decision points in the operation by replaying the surgical video. In addition to the mismatch between different screenshots pointed out in the background technology, there are also the following disadvantages: (1) Perspective limitation: The video can only provide a limited perspective and may not be able to capture all the key details of the operation. If the perspective in the surgical video is poor or important parts are blocked, the doctor may not be able to fully evaluate the impact of the surgical operation (the 2D perspective is fixed and cannot be transformed like a point cloud). (2) Limitations of two-dimensional images: Traditional endoscopic videos are two-dimensional, and it may be difficult to accurately grasp the three-dimensional relationship, depth perception and precise spatial positioning of the internal structure, and it is difficult to quantitatively and accurately measure (such as incision size, suture density, etc.), and these details are crucial to evaluating the quality of the operation.

[0093] In this context, the embodiment of the present invention creatively aligns the three-dimensional model of the surgical area at different surgical stages, helping doctors to intuitively feel the impact of different surgical operations on the changes in the three-dimensional structure of the body. This can provide a more detailed perspective than traditional 2D videos, allowing doctors to examine the surgical area from any angle, assisting in the quantitative evaluation of the execution quality of surgical techniques after surgery, such as the accuracy of cutting, the firmness of suturing, etc., as well as the immediate impact of the operation on the patient's physiological state and the attribution of postoperative complications.

[0094] In particular, the three-dimensional reconstruction method of the laparoscopic surgical scene based on three-dimensional Gaussian sputtering proposed in the embodiment of the present invention starts the dynamic scene optimizer when the color video frame at the current moment is not a key frame, estimates the deformation field used to convert the regular space 3D Gaussian three-dimensional model into the real-time space 3D Gaussian three-dimensional model; then starts the pose optimizer to update the current camera pose; and alternately iterates and runs the two optimizers until both converge, and outputs the real-time space 3D Gaussian three-dimensional model at the current moment. That is, it is possible to generate a high-fidelity dynamic in vivo three-dimensional scene, which is convenient for doctors to observe the patient's internal environment. In addition, depth map information that can provide geometric surface information is introduced, and it is used to supervise the center point coordinates of the 3D Gaussian so that it is bound around the surface, greatly improving the three-dimensional reconstruction accuracy of the tissue surface.

[0095] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0096] Embodiment 1:

[0097] like Figure 1 As shown, an embodiment of the present invention provides a minimally invasive surgery postoperative review method based on multi-stage surgery scene registration, comprising:

[0098] S1, obtaining the color video frame sequence of the whole process of laparoscopic surgery;

[0099] S2, based on a preset surgical stage recognition network, classifying each color video frame into a set of video frames of different surgical stages;

[0100] S3, obtaining a real-time spatial three-dimensional Gaussian three-dimensional model corresponding to each set of video frames;

[0101] S4. Perform multi-stage model registration on each real-time spatial three-dimensional Gaussian three-dimensional model to form a three-dimensional model of a multi-stage scene.

[0102] The embodiment of the present invention designs a postoperative multi-scene registration algorithm based on an intraoperative dynamic three-dimensional reconstruction framework, which can achieve multi-scene registration by only acquiring color video frame information and tissue deformation at different surgical stages, making it convenient for doctors to intuitively observe the impact of surgical operations.

[0103] In an optional implementation, for any of the video frame sets, the embodiment of the present invention adopts a laparoscopic surgery scene 3D reconstruction method based on 3D Gaussian sputtering to obtain a corresponding real-time space 3D Gaussian 3D model; Figure 2 As shown, including:

[0104] S31, based on a color video frame sequence and its corresponding depth map sequence in the video frame set;

[0105] S32, based on the deformation-aware key frame pool, estimating the initial camera pose of the color video frame at the current moment, and determining whether it is a key frame to be stored in the key frame pool; wherein the key frame pool maintains a relatively static regular space 3D Gaussian 3D model;

[0106] If the current color video frame is not a key frame, then

[0107] S3311, taking the color video frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, starting the dynamic scene optimizer to estimate the deformation field used to convert the regular space 3D Gaussian 3D model into the real-time space 3D Gaussian 3D model;

[0108] S3312, taking the color video frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian 3D model as input, and starting the pose optimizer to update the current camera pose;

[0109] S3313, alternately iteratively running the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least the real-time spatial 3D Gaussian three-dimensional model at the current moment;

[0110] If the current color video frame is a key frame, then

[0111] S3321, taking the color video frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, starting the static scene optimizer to update the regular space 3D Gaussian 3D model;

[0112] S3322, taking the color video frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model as input, and starting the pose optimizer to update the current camera pose;

[0113] S3323, alternately iteratively running the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least a regular space 3D Gaussian 3D model at the current moment;

[0114] S34: Select the real-time space 3D Gaussian 3D model at the moment corresponding to the last non-key frame as the real-time space 3D Gaussian 3D model corresponding to the video frame set.

[0115] The embodiment of the present invention can establish a three-dimensional model of the overall environment when the color video frame sequence only captures the local in vivo environment and tissue deformation, thereby overcoming the disadvantage of the limited field of view of the laparoscope.

[0116] like Figure 3 As shown, Figure 3A complete technical roadmap of the postoperative review method for minimally invasive surgery based on multi-stage surgical scene registration provided by an embodiment of the present invention is given. Next, the following will be combined with Figure 3 The following are the steps of the above solution:

[0117] In step S1, a color video frame sequence of the entire laparoscopic surgery process is acquired.

[0118] In this step, a color video frame sequence of the entire laparoscopic surgery process is obtained. , where t=0,1,2,… represents the time series.

[0119] In step S2, each color video frame is classified into a set of video frames of different surgical stages based on a preset surgical stage recognition network.

[0120] In this step, a surgical stage recognition network is designed to classify different surgical stages. The network architecture is as follows: Figure 4 Here, the surgical phases are divided into surgical preparation phase A, pneumoperitoneum phase B, exploration and operation phase C, and cleaning and suturing phase D.

[0121] The surgical stage recognition network includes a three-dimensional spatial feature extraction channel, a two-dimensional spatial feature extraction channel, a temporal optical flow feature extraction channel and a softmax layer. The extracted features of the three channels are output through the softmax layer to obtain the stage classification score; among them:

[0122] The three-dimensional spatial feature extraction channel is used to extract spatial geometric information of the point cloud model at different times using the PointNet network.

[0123] The two-dimensional spatial feature extraction channel is composed of a plurality of convolutional layers for identifying the instrument and tissue information in each of the color video frames.

[0124] The temporal optical flow feature extraction channel is composed of a plurality of convolutional layers, and is used to extract the optical flow information of each color video frame relative to its previous frame.

[0125] The network uses a supervised training method, which uses a cross entropy loss function for training, expressed as follows:

[0126]

[0127] in, N represents the number of samples; y ic Indicates i The samples belong to c The true label of the class, 1 means it belongs to the class, 0 means it does not belong to the class; Indicates i The samples belong to cThe predicted probability of the class, w c Indicates c The weight of the category.

[0128] After the surgical stage recognition network converges, the converged network is used to obtain video frame sets corresponding to the surgical preparation stage A, the pneumoperitoneum stage B, the exploration and operation stage C, and the cleaning and suturing stage D. , , , ;in, N A , N B , N C , N D Respectively represent the number of video frames in the corresponding set.

[0129] In step S3, a real-time spatial three-dimensional Gaussian three-dimensional model corresponding to each video frame set is obtained.

[0130] Exemplarily, for any of the video frame sets, this step obtains a corresponding real-time spatial three-dimensional Gaussian three-dimensional model based on a three-dimensional reconstruction method of a laparoscopic surgery scene using three-dimensional Gaussian sputtering.

[0131] Before formally introducing the specific steps of the 3D reconstruction method of laparoscopic surgery scenes based on 3D Gaussian sputtering, it is necessary to supplement the relevant basic mathematical expressions:

[0132] (1) Mathematical expression of three-dimensional model:

[0133] In the implementation of the present invention, a three-dimensional model of a scene (any symbol containing " " represents the three-dimensional model of the scene) consists of multiple 3D Gaussian G ( x ), the mathematical formula of each 3D Gaussian is as follows:

[0134]

[0135] Among them, 3D Gaussians are geometrically represented as ellipsoids in space, and the geometry of a three-dimensional model of a scene is "built" by these ellipsoids. x Represents the center point position of the three-dimensional Gaussian, that is, the mean of the three-dimensional Gaussian distribution, which is also the coordinate of the center point of the ellipsoid. R Represents the rotation of a three-dimensional Gaussian in space, using quaternions during optimization Q =( q 0 ,q x,q y ,q z )express. S It represents the scale factor of the three-dimensional Gaussian on the X, Y, and Z axes, that is, the radius of the ellipsoid on its X, Y, and Z axes.

[0136] Considering that the above formula only describes the shape of the 3D Gaussian, and each 3D Gaussian also carries color information and transparency information related to the viewing angle, the embodiment of the present invention further makes the following settings:

[0137] Each 3D Gaussian transparency uses α express, α ∈[0,1], color information uses three sets of spherical harmonic coefficients c lm Represents the color values ​​of the three channels of RGB respectively. l is the order of the spherical harmonic coefficients, m is an integer, - l≤m≤l .

[0138] Among them, the spherical harmonic coefficients are the spherical harmonic functions The form and number of spherical harmonics are determined after the maximum order L is determined. Therefore, the values ​​of the three color channels RGB can be represented by three sets of spherical harmonic coefficients. For example, the value of the R channel can be expressed as: ,in θ represents the zenith angle, φ Represents the azimuth, and these two angles together represent the direction of the viewing angle.

[0139] (2) Mathematical expression of deformation field:

[0140] The deformation field acts on the center point of each three-dimensional Gaussian ellipsoid, causing it to shift in space. The embodiment of the present invention uses three sets of Fourier series , Fit this deformation, each set of Fourier series fits the deformation of one dimension. For example, the displacement in the X dimension It can be calculated by the following formula:

[0141]

[0142] Similarly, the displacement in the Y dimension can be calculated using formula (2): v Y And the displacement in the Z dimension v z , and define the deformation field Wt ( x ):

[0143]

[0144] Formula (4) represents the effect on the center point x The displacement vector of 3D Gaussian and deformation field is given in the following specific form:

[0145]

[0146] in, Represents the three-dimensional coordinates of the center point after displacement, Represents the deformed 3D Gaussian.

[0147] (3) Mathematical expression of camera pose:

[0148] The embodiment of the present invention uses quaternions and translation vectors to express the camera pose, that is, ,in P t Represents the camera pose, which consists of a quaternion that expresses the camera rotation q t and a translation vector expressing the camera translation t t express.

[0149] like Figure 5 As shown, Figure 5 A complete technical roadmap of the laparoscopic surgery scene three-dimensional reconstruction method provided by the embodiment of the present invention is given. Figure 5 , a laparoscopic surgery scene 3D reconstruction method based on 3D Gaussian sputtering, and obtaining a corresponding real-time space 3D Gaussian 3D model; including:

[0150] In step S31, based on a color video frame sequence in the video frame set and a corresponding depth map sequence.

[0151] Getting a collection of video frames , , , Afterwards, the embodiment of the present invention also obtains the corresponding depth map sequence For example, the depth map here can be obtained using a depth estimation algorithm such as MIOL.

[0152] It should be noted that, considering that the principles of the three-dimensional reconstruction process of the laparoscopic surgery scene corresponding to each video frame set are completely consistent, and for the sake of ease of introduction, the following only gives a general introduction without specifying a specific video frame set.

[0153] In step S32, based on the deformation-aware key frame pool, the initial camera pose of the color video frame at the current moment is estimated, and it is determined whether it is a key frame that needs to be stored in the key frame pool; wherein the key frame pool maintains a relatively static regular space 3D Gaussian three-dimensional model.

[0154] First of all, it should be clarified that the embodiment of the present invention defines the above deformation-aware key frame pool to have the following functions:

[0155] (1) Identify and store color video frames with relatively small deformation and high quality As keyframe I k , the set of all key frames is is the key frame index. The information in the key frame pool maintains a static 3D model information, called the canonical space. In contrast to the canonical space, the space where the dynamic 3D model is located is called the real-time space.

[0156] (2) Calculate the initial camera pose corresponding to the color video frame at the current moment .

[0157] On this basis, this step specifically includes:

[0158] S321, extracting SIFT feature descriptors of color video frames, traversing all key frames in the matching key frame pool, and taking the key frame with the most matching key point pairs as the best matching frame.

[0159] Specifically: First, use the key point recognition algorithm to obtain any pair of key points on the matching video (that is, points with obvious features, such as edge points of objects); then, extract the SIFT feature descriptors of these key points, and use the SIFT feature descriptors to find matching point pairs among these key points (that is, if the vector angle of the SIFT feature descriptors between two key points is less than a certain threshold, then this is considered to be a pair of matching key points); finally, the key frame with the most matching key point pairs is taken as the best matching frame.

[0160] S322, calculating the pose transformation between the color video frame and the best matching frame to obtain the initial camera pose.

[0161] For example, 3D-2D PNP (Perspective N Points) is used to calculate the pose transformation between the color video frame and the best matching frame, thereby obtaining the initial camera pose. , as the initial value for subsequent pose optimization.

[0162] S323: Based on the FPFH feature descriptor of the matching key point pair between the color video frame and the best matching frame, estimate the deformation degree of the color video frame, and store the estimated deformation measurement value in the buffer pool.

[0163] Specifically, in S21, after the best matching frame is obtained, the three-dimensional coordinates of the matching key point pairs between the color video frame and the best matching frame are restored using the depth map data, and then the FPFH feature descriptor of the matching key point is obtained using the traditional FPFH algorithm to estimate the deformation degree of the color video frame; the deformation estimation function is as follows:

[0164]

[0165] in, F nr represents the deformation metric value of the color image frame; and It is a pair of FPFH feature descriptors matching key points; N s Represents the number of matching key point pairs; Represents the F-norm of a matrix or vector.

[0166] S324, judging the quality of the color video frame; wherein the judging index includes the number of matching key points on the color video frame and / or the area of ​​the two-dimensional convex hull formed by the matching key points.

[0167] S325: If the deformation metric value of the color video frame at the current moment is less than the corresponding threshold and the quality is greater than the corresponding threshold, it is determined as a key frame and stored in the key frame pool.

[0168] If the color video frame at the current moment is a key frame, the key frame information will be stored in the key frame pool. The stored information includes but is not limited to the frame ID, initial camera pose, color video frame, depth map, key point coordinates, and SIFT feature descriptors of the key points.

[0169] S326: If no new key frame is stored in the key frame pool after the preset time, a color video frame with the smallest deformation metric value is searched in the current buffer pool and stored in the key frame pool, and the data in the buffer pool is cleared.

[0170] like Figure 5 As shown, for whether the color video frame at the current moment is a key frame or not, the embodiment of the present invention processes it according to different processing ideas to output the regular space 3D Gaussian 3D model or the real-time space 3D Gaussian 3D model at the current moment, as shown below:

[0171] First, if the current color video frame is not a key frame, then

[0172] In step S3311, the color video frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment are taken as input, and the dynamic scene optimizer is started to estimate the deformation field used to convert the regular space 3D Gaussian 3D model into the real-time space 3D Gaussian 3D model.

[0173] The scene optimizer in the embodiment of the present invention has two types: dynamic scene optimizer OP dynamic and Static Scene Optimizer OP static , since the color video frame at the current moment is not a key frame, only the dynamic scene optimizer is started here. The dynamic scene optimizer has the following functions:

[0174] Compared to the static scene optimizer, the dynamic scene optimizer only estimates a deformation field W t , W t The regular space 3D Gaussian 3D model will be converted into a real space 3D Gaussian 3D model , which will be converted during the optimization process; the formula is as follows:

[0175]

[0176] In step S3312, the color video frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian model are used as input to start the pose optimizer to update the current camera pose.

[0177] Pose Optimizer OP pose It will be run during the scene optimizer iteration, for example, it will be run when the scene optimizer iterates T, 2T, 3T...nT times, so the input of the pose optimizer is the initial pose , current picture and And the 3D Gaussian 3D model output by the scene optimizer after iterating for a period of time. If the started scene optimizer is a static scene optimizer, the input 3D Gaussian 3D model is a regular space 3D model If the scene optimizer started is a dynamic scene optimizer, the input 3D Gaussian 3D model is a 3D model of real-time space. , the output of the pose optimizer is the pose of the current frame P t , the pose optimizer will output the best pose after optimization. The scene optimizer started here is a dynamic scene optimizer, and the formula is expressed as follows:

[0178]

[0179] In step S3313, the dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least the real-time spatial 3D Gaussian three-dimensional model at the current moment.

[0180] Exemplarily, the optimization solution methods of the two optimizers both use the Adam algorithm.

[0181] (1) Loss function and optimization parameters of the dynamic scene optimizer

[0182] Based on the fact that the optimizable parameter of the dynamic scene optimizer is the deformation field, the corresponding loss function is designed:

[0183]

[0184] in, L dynamic represents the total loss function of the dynamic scene optimizer; They represent the color map loss function and depth map loss function of the dynamic scene optimizer respectively;

[0185] Deformation Field W t The 3D Gaussian model of the regular space at the previous moment The deformation effect;

[0186] Represents a 3D Gaussian i The three-dimensional position of the center point in canonical space; KNN ( i ) represents a 3D Gaussian i The K nearest neighbors of Represents a neighboring 3D Gaussian j The three-dimensional position of the center point in canonical space; Represents 3D Gaussian i and neighboring 3D Gaussian j The three-dimensional position of the center point in real space;

[0187] w ij Represents a 3D Gaussian i and neighboring 3D Gaussian j The connection weight between them; exp represents an exponential function with the natural constant e as the base; λ is the weight hyperparameter used to adjust w ij The value of Represents the bi-norm of a vector.

[0188] (2) About the loss function and optimization parameters of the pose optimizer

[0189] Based on the optimizable parameters of the pose optimizer as the current camera pose, the corresponding loss function is designed:

[0190]

[0191] in, L pose represents the total loss function of the pose optimizer; L c , L d They represent the color image loss function and depth image loss function of the pose optimizer respectively;

[0192] Respectively represent the color video frame and its depth map at the current moment;

[0193] Represents the F norm of a matrix or vector;

[0194] express and SSIM loss;

[0195] A differentiable renderer representing a 3D Gaussian splatter, used to render 3D Gaussian 3D models With the current camera pose P t Rendering into predicted color video frames and the predicted depth map ;

[0196] , They respectively represent the current regular space 3D Gaussian 3D model and the current real-time space 3D Gaussian 3D model.

[0197] Second, if the current color video frame is a key frame, then

[0198] In step S3321, the color video frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment are taken as input to start the static scene optimizer to update the regular space 3D Gaussian 3D model.

[0199] The scene optimizer in the embodiment of the present invention has two types: dynamic scene optimizer OP dynamic and Static Scene Optimizer OP static ,Since the color video frame at the current moment is a key frame, only the static scene optimizer is started here.

[0200] For the static scene optimizer, on the one hand, the color video frame at the current moment is used to optimize and generate incremental information (Incremental information refers to the new geometric and texture information carried by the latest frame compared to the previous image sequence); on the other hand, the previous regular space 3D Gaussian 3D model is used , And the camera pose output by the pose optimizer P t , modeling the latest regular space 3D Gaussian 3D model , the formula is as follows:

[0201]

[0202] It should be noted that the incremental information will be directly added to the Gaussian 3D model during the optimization process. The color, shape and other properties of the 3D Gaussian will also be optimized to make it consistent with the observations of the latest frame video.

[0203] In step S3322, the color video frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian three-dimensional model are used as input to start the pose optimizer to update the current camera pose.

[0204] Similarly, since the scene optimizer started here is a static scene optimizer, the formula is expressed as follows:

[0205]

[0206] In step S3323, the dynamic scene optimizer and the pose optimizer are run alternately and iteratively until both converge, so as to output at least the regular space 3D Gaussian three-dimensional model at the current moment.

[0207] Exemplarily, the optimization solution methods of the two optimizers both use the Adam algorithm.

[0208] (1) Loss function and optimization parameters of static scene optimizer

[0209] Based on the fact that the optimizable parameters of the static scene optimizer are the properties of the regular space 3D Gaussian 3D model at the previous moment, the corresponding loss function is designed:

[0210]

[0211] in, L static represents the total loss function of the static scene optimizer; They represent the color map loss and depth map loss of the static scene optimizer respectively;

[0212] is the property of the regular space 3D Gaussian three-dimensional model at the previous moment; the color of each 3D Gaussian adopts the spherical harmonic function coefficient c lm Indicates that the shape is S Matrix and R The matrix represents R Matrix as Quaternion , transparency is adopted α Indicates that the center point position is x express.

[0213] (2) About the loss function and optimization parameters of the pose optimizer

[0214] It can be understood that the content of this part can be completely referred to the content of formulas (16) to (19), and will not be repeated here.

[0215] In particular, during the operation of the static scene optimizer, the embodiment of the present invention also performs a densification process to increase the number of 3D Gaussians in the regular space 3D Gaussian three-dimensional model to obtain incremental information. .

[0216] Specifically, there are two densification methods: segmentation and cloning. The Adam algorithm calculates the gradient for the 3D Gaussian center point visible from the perspective of each color video frame at the current moment. After a period of iteration, if the average gradient obtained by a 3D Gaussian is greater than a certain threshold, it is considered to need to be densified. Among them:

[0217] If the Gaussian is too large, that is, the maximum value in the Gaussian S matrix is ​​greater than or equal to the preset threshold, segmentation densification is performed. The segmentation operation splits the Gaussian into two. For example, the S of these two 3D Gaussians will be multiplied by 0.8, and the center point coordinates of the new 3D Gaussian are the center point coordinates of the original 3D Gaussian plus a bias randomly sampled from a three-dimensional Gaussian distribution with a mean of 0, and the remaining attributes are copied. If the Gaussian is too small, that is, the maximum value in the Gaussian S matrix is ​​less than the preset threshold, cloning densification is performed, that is, copying an identical 3D Gaussian at the original position. The new 3D Gaussians generated by these cloning and segmentation operations are incremental information .

[0218] Since the incremental information is generated in the scene optimizer process, the incremental information, that is, the newly generated 3D Gaussian, will be further optimized, and its properties such as shape, size and color will be optimized to be consistent with the "length" of the color video frame.

[0219] Thus, the embodiment of the present invention designs a three-dimensional reconstruction framework for complex in vivo environments, which can establish a three-dimensional model of the overall environment when the color video frame sequence only captures the local in vivo environment and tissue deformation, thus overcoming the disadvantage of limited laparoscope field of view.

[0220] In addition, in an embodiment of the present invention, depth map information that provides geometric surface information is introduced and used to supervise the center point coordinates of the 3D Gaussians so that they are bound around the surface, greatly improving the three-dimensional reconstruction accuracy of the tissue surface. The addition of the depth map provides a geometric prior for all 3D Gaussians, making it unnecessary to adjust the 3D Gaussian density.

[0221] It is worth pointing out that if Figure 5 As shown, in addition to outputting the 3D Gaussian model at the current moment, the embodiment of the present invention can also simultaneously output the optimal camera pose of the color video frame at the current moment. In other words, the embodiment of the present invention can simultaneously estimate the accurate camera pose and tissue deformation information, providing support information for motion planning for robotic surgery.

[0222] The solution provided in the embodiment of the present invention can be used for simulating real surgical scenes, and combined with VR display technology can help novice doctors perform surgical training. Figure 6 Demonstrated the effect of reconstructing the real in vivo environment.

[0223] In step S34, the real-time space 3D Gaussian 3D model at the time corresponding to the last non-key frame is selected as the real-time space 3D Gaussian 3D model corresponding to the video frame set.

[0224] Finally, the real-time spatial 3D Gaussian 3D model corresponding to each video frame set is obtained and recorded as , , , .

[0225] In step S4, each real-time spatial 3D Gaussian 3D model is subjected to multi-stage model registration to form a 3D model of a multi-stage scene.

[0226] It should be noted that, since the 3D Gaussian 3D model of each stage in the 3D reconstruction method of laparoscopic surgical scenes based on 3D Gaussian sputtering uses the camera coordinate system captured by the first color video frame of that stage as the world coordinate system, the real-time space 3D Gaussian 3D model within each stage is naturally aligned.

[0227] Performing multi-stage model registration on each of the real-time spatial three-dimensional Gaussian three-dimensional models to form a three-dimensional model of a multi-stage scene; comprising:

[0228] (1) Camera pose sampling method:

[0229] Set a spherical coordinate system, set the radius range of the spherical coordinate system to the distance between the farthest three-dimensional Gaussians, set the polar angle range according to the camera's field of view, and set the azimuth angle range to the front of the model, and sample evenly on the spherical coordinate system. NThe camera position is converted into Cartesian coordinates using the following conversion formula:

[0230]

[0231] in,( x cam , y cam , z cam ) represents the transformed camera position, r represents the radius, θ represents the polar angle, ϕ Indicates azimuth;

[0232] Align the camera's viewing angle to the center of the Gaussian 3D model and offset it up, down, left, and right in that viewing angle. a degrees, expanding each camera position to a The camera poses are finally sampled. aN Camera poses { P i}.

[0233] (2) After sampling, and exist aN Rendering with camera poses:

[0234]

[0235] in, Respectively expressed in i Rendering images of 3D model A and 3D model B obtained under 3D camera poses.

[0236] For example, here a 5 is preferred.

[0237] (3) Use the following energy function as the optimization objective and calculate arrive Transformation posture P BA ; The energy function is expressed as follows:

[0238]

[0239] Unify the multi-stage 3D Gaussian 3D models into the same coordinate system:

[0240]

[0241] in, Indicates that Unified to Three-dimensional Gaussian three-dimensional model after coordinate system; express The i 3D Gaussian; Represents the coordinates of the center point of the three-dimensional Gaussian; express Middle i The coordinates of the center point of a three-dimensional Gaussian; express P BA The 3*3 rotation matrix representing the quaternion transformation of the rotation; t BA Express posture P BA The translation vector in ; transform (⋅) means P BA Transformation equations into rotation matrices and translation vectors;

[0242] Similarly, get , , and finally obtain the aligned 3D model sequence , forming a three-dimensional model of a multi-stage scene for doctors to review.

[0243] At this point, the embodiment of the present invention completes the entire process of the postoperative review method for minimally invasive surgery based on multi-stage surgical scene registration. With the multi-stage three-dimensional model obtained after registration, doctors can review the impact of different surgical operations more efficiently and intuitively, measure the incision size, observe the suture density and evaluate the bleeding point more accurately, and reduce the perspective and proportional errors caused by two-dimensional images; the dynamic display of the three-dimensional model makes the education process more intuitive and real-time, which helps medical students and new doctors better understand complex surgical techniques and processes; and the three-dimensional model can help doctors more accurately analyze operations that may cause complications during surgery, such as improper resection methods or suturing techniques, so as to take measures to reduce the risks of future surgeries.

[0244] Embodiment 2:

[0245] The embodiment of the present invention provides a minimally invasive surgery postoperative review system based on multi-stage surgery scene registration, comprising:

[0246] An acquisition module is used to acquire a color video frame sequence of the entire laparoscopic surgery process;

[0247] A recognition module, configured to classify each color video frame into a set of video frames of different surgical stages based on a preset surgical stage recognition network; wherein the surgical stages include surgical preparation stage A, pneumoperitoneum stage B, exploration and operation stage C, and cleaning and suturing stage D;

[0248] A reconstruction module, used for obtaining a real-time spatial three-dimensional Gaussian three-dimensional model corresponding to each of the video frame sets;

[0249] The registration module is used to perform multi-stage model registration on each of the real-time spatial three-dimensional Gaussian three-dimensional models to form a three-dimensional model of a multi-stage scene.

[0250] Embodiment 3:

[0251] An embodiment of the present invention provides a storage medium storing a computer program for postoperative review of minimally invasive surgery based on multi-stage surgery scene registration, wherein the computer program enables a computer to execute the postoperative review method of minimally invasive surgery as described in Example 1.

[0252] Embodiment 4:

[0253] An embodiment of the present invention provides an electronic device, including:

[0254] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for performing a postoperative review of minimally invasive surgery as described in Example 1.

[0255] It is understandable that the minimally invasive surgery postoperative review system based on multi-stage surgery scene registration, storage medium and electronic device provided in the embodiments of the present invention correspond to the minimally invasive surgery postoperative review method based on multi-stage surgery scene registration provided in the embodiments of the present invention, and the explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the minimally invasive surgery postoperative review method, which will not be repeated here.

[0256] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0257] 1. The embodiment of the present invention designs a postoperative multi-scene registration algorithm based on an intraoperative dynamic three-dimensional reconstruction framework, which can realize multi-scene registration by only acquiring color video frame information and tissue deformation at different surgical stages, making it convenient for doctors to intuitively observe the impact of surgical operations.

[0258] 2. In the embodiment of the present invention, the multi-stage three-dimensional model obtained after registration allows doctors to review the impact of different surgical operations more efficiently and intuitively, measure the incision size, observe the suture density and evaluate the bleeding point more accurately, and reduce the perspective and proportion errors caused by two-dimensional images; the dynamic display of the three-dimensional model makes the education process more intuitive and real-time, which helps medical students and new doctors better understand complex surgical techniques and processes; and the three-dimensional model can help doctors more accurately analyze operations that may cause complications during surgery, such as improper resection methods or suturing techniques, so as to take measures to reduce the risks of future surgeries.

[0259] 3. The three-dimensional reconstruction method of laparoscopic surgical scenes based on three-dimensional Gaussian sputtering proposed in the embodiment of the present invention, when the color video frame at the current moment is not a key frame, starts the dynamic scene optimizer to estimate the deformation field used to convert the regular space 3D Gaussian three-dimensional model into the real-time space 3D Gaussian three-dimensional model; then starts the pose optimizer to update the current camera pose; and alternately iterates and runs the two optimizers until both converge, and outputs the real-time space 3D Gaussian three-dimensional model at the current moment. That is, it can generate high-fidelity dynamic in vivo three-dimensional scenes, which is convenient for doctors to observe the patient's internal environment. In addition, depth map information that can provide geometric surface information is introduced, and it is used to supervise the center point coordinates of the 3D Gaussian so that it is bound around the surface, which greatly improves the three-dimensional reconstruction accuracy of the tissue surface.

[0260] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0261] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A postoperative review method for minimally invasive surgery based on multi-stage surgical scene registration, characterized in that: include: Obtain color video frame sequences of the entire laparoscopic surgery process; Based on a preset surgical stage recognition network, each color video frame is classified into a set of video frames of different surgical stages; wherein the surgical stages include surgical preparation stage A, pneumoperitoneum stage B, exploration and operation stage C, and cleaning and suturing stage D; Acquire a real-time spatial three-dimensional Gaussian three-dimensional model corresponding to each of the video frame sets; Performing multi-stage model registration on each of the real-time spatial three-dimensional Gaussian three-dimensional models to form a three-dimensional model of a multi-stage scene; The method of classifying each color video frame into a set of video frames of different surgical stages based on a preset surgical stage recognition network includes: Constructing the surgical stage recognition network; the surgical stage recognition network includes a three-dimensional spatial feature extraction channel, a two-dimensional spatial feature extraction channel, a temporal optical flow feature extraction channel and a softmax layer, and the extracted features of the three channels are output through the softmax layer to obtain a stage classification score; The three-dimensional spatial feature extraction channel is used to extract the spatial geometric information of the point cloud model at different times using the PointNet network; the two-dimensional spatial feature extraction channel is composed of a plurality of convolutional layers, and is used to identify the instrument and tissue information in each of the color video frames; the temporal optical flow feature extraction channel is composed of a plurality of convolutional layers, and is used to extract the optical flow information of each of the color video frames relative to the previous frame; The surgical stage recognition network is trained by using a cross entropy loss function until convergence, and the converged network is used to obtain video frame sets corresponding to the surgical preparation stage A, pneumoperitoneum stage B, exploration and operation stage C, and cleaning and suturing stage D, respectively. , , , ; in, N A , N B , N C , N D Respectively represent the number of video frames in the corresponding set; For any of the video frame sets, a laparoscopic surgery scene 3D reconstruction method based on 3D Gaussian sputtering is used to obtain a corresponding real-time spatial 3D Gaussian 3D model; including: Based on a color video frame sequence and a corresponding depth map sequence in the video frame set; Based on the deformation-aware key frame pool, the initial camera pose of the color video frame at the current moment is estimated, and whether it is a key frame to be stored in the key frame pool; wherein the key frame pool maintains a relatively static regular space 3D Gaussian 3D model; If the current color video frame is not a key frame, then Taking the color video frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, a dynamic scene optimizer is started to estimate the deformation field used to transform the regular space 3D Gaussian 3D model into the real space 3D Gaussian 3D model; The color video frame and its depth map at the current moment, the current camera pose, and the current real-time spatial 3D Gaussian model are used as input to start the pose optimizer to update the current camera pose. Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both are converged, so as to output at least a real-time spatial 3D Gaussian 3D model at the current moment; If the current color video frame is a key frame, then Taking the color video frame and its depth map at the current moment, the current camera pose, and the regular space 3D Gaussian 3D model at the previous moment as input, the static scene optimizer is started to update the regular space 3D Gaussian 3D model; The color video frame and its depth map at the current moment, the current camera pose, and the current canonical space 3D Gaussian 3D model are used as input to start the pose optimizer to update the current camera pose; Alternately iterate and run the dynamic scene optimizer and the pose optimizer until both converge, so as to output at least a regular space 3D Gaussian 3D model at the current moment; The real-time space 3D Gaussian 3D model at the moment corresponding to the last non-key frame is selected as the real-time space 3D Gaussian 3D model corresponding to the video frame set.

2. The minimally invasive surgery postoperative review method according to claim 1, characterized in that: The deformation-aware key frame pool estimates the initial camera pose of the color video frame at the current moment, and determines whether it is a key frame to be stored in the key frame pool; including: Extract the SIFT feature descriptor of the color video frame, traverse all the key frames in the matching key frame pool, and take the key frame with the most matching key point pairs as the best matching frame; Calculate the pose transformation between the color video frame and the best matching frame to obtain the initial camera pose; Based on the FPFH feature descriptor of the matching key point pair between the color video frame and the best matching frame, the deformation degree of the color video frame is estimated, and the estimated deformation measurement value is stored in the buffer pool; Based on the best matching frame, judging the quality of the color video frame; wherein the judging index includes the number of matching key points on the color video frame and / or the area of ​​the two-dimensional convex hull formed by the matching key points; If the deformation metric value of the color video frame at the current moment is less than the corresponding threshold and the quality is greater than the corresponding threshold, it is judged as a key frame and stored in the key frame pool; If no new key frame is stored in the key frame pool after the preset time, the color video frame with the smallest deformation metric value is found in the current buffer pool and stored in the key frame pool, and the data in the buffer pool is cleared.

3. The minimally invasive surgery postoperative review method according to claim 2, characterized in that: Based on the optimizable parameters of the pose optimizer as the current camera pose, the corresponding loss function is designed: in, L pose represents the total loss function of the pose optimizer; L c , L d They represent the color image loss function and depth image loss function of the pose optimizer respectively; Respectively represent the color video frame and its depth map at the current moment; Represents the F norm of a matrix or vector; express and SSIM loss; A differentiable renderer representing a 3D Gaussian splatter, used to render 3D Gaussian 3D models With the current camera pose P t Rendering into predicted color video frames and the predicted depth map ; , They respectively represent the current regular space 3D Gaussian 3D model and the current real-time space 3D Gaussian 3D model.

4. The minimally invasive surgery postoperative review method according to claim 3, characterized in that: Based on the fact that the optimizable parameter of the dynamic scene optimizer is the deformation field, the corresponding loss function is designed: in, L dynamic represents the total loss function of the dynamic scene optimizer; They represent the color map loss function and depth map loss function of the dynamic scene optimizer respectively; Deformation Field W t The 3D Gaussian model of the regular space at the previous moment The deformation effect; Represents a 3D Gaussian i The three-dimensional position of the center point in canonical space; KNN ( i ) represents a 3D Gaussian i The K nearest neighbors of Represents a neighboring 3D Gaussian j The three-dimensional position of the center point in canonical space; Represents 3D Gaussian i and neighboring 3D Gaussian j The three-dimensional position of the center point in real space; w ij Represents a 3D Gaussian i and neighboring 3D Gaussian j The connection weight between them; exp represents an exponential function with the natural constant e as the base; λ is the weight hyperparameter used to adjust w ij The value of represents the two-norm of a vector; and / or Based on the fact that the optimizable parameters of the static scene optimizer are the properties of the regular space 3D Gaussian 3D model at the previous moment, the corresponding loss function is designed: in, L static represents the total loss function of the static scene optimizer; They represent the color map loss and depth map loss of the static scene optimizer respectively; is the property of the regular space 3D Gaussian three-dimensional model at the previous moment; the color of each 3D Gaussian adopts the spherical harmonic function coefficient c lm Indicates that the shape is S Matrix and R The matrix represents R Matrix as Quaternion , transparency is adopted α Indicates that the center point position is x express; and / or During the operation of the static scene optimizer, a densification process is also performed to increase the number of 3D Gaussians in the regular space 3D Gaussian three-dimensional model.

5. The minimally invasive surgery postoperative review method according to claim 1, characterized in that: Set the video frame , , , The corresponding real-time space 3D Gaussian 3D model is recorded as , , , ; The method of performing multi-stage model registration on each of the real-time spatial three-dimensional Gaussian three-dimensional models to form a three-dimensional model of a multi-stage scene comprises: Set a spherical coordinate system, set the radius range of the spherical coordinate system to the distance between the farthest three-dimensional Gaussians, set the polar angle range according to the camera's field of view, and set the azimuth angle range to the front of the model, and sample evenly on the spherical coordinate system. N The camera position is converted into Cartesian coordinates using the following conversion formula: in,( x cam , y cam , z cam ) represents the transformed camera position, r represents the radius, θ represents the polar angle, ϕ Indicates azimuth; Align the camera's viewing angle to the center of the Gaussian 3D model and offset it up, down, left, and right in that viewing angle. a degrees, expanding each camera position to a The camera poses are finally sampled. aN Camera poses { P i }; After the sampling is completed, and exist aN Rendering with camera poses: in, Respectively expressed in i Rendering images of 3D model A and 3D model B obtained under the sampled camera poses; Using the following energy function as the optimization objective, calculate arrive Transformation posture P BA ; The energy function is expressed as follows: Unify the multi-stage 3D Gaussian 3D models into the same coordinate system: in, Indicates that Unified to Three-dimensional Gaussian three-dimensional model after coordinate system; express The i 3D Gaussian; Represents the coordinates of the center point of the three-dimensional Gaussian; express Middle i The coordinates of the center point of a three-dimensional Gaussian; express P BA The 3*3 rotation matrix representing the quaternion transformation of the rotation; t BA Express posture P BA The translation vector in ; transform (⋅) means P BA Transformation equations into rotation matrices and translation vectors; Similarly, get , , and finally obtain the aligned 3D model sequence , forming a three-dimensional model of a multi-stage scene.

6. A minimally invasive surgery postoperative review system based on multi-stage surgery scene registration, characterized in that: The method for performing postoperative review of minimally invasive surgery as claimed in claim 1 comprises: An acquisition module is used to acquire a color video frame sequence of the entire laparoscopic surgery process; A recognition module, configured to classify each color video frame into a set of video frames of different surgical stages based on a preset surgical stage recognition network; wherein the surgical stages include surgical preparation stage A, pneumoperitoneum stage B, exploration and operation stage C, and cleaning and suturing stage D; A reconstruction module, used for obtaining a real-time spatial three-dimensional Gaussian three-dimensional model corresponding to each of the video frame sets; The registration module is used to perform multi-stage model registration on each of the real-time spatial three-dimensional Gaussian three-dimensional models to form a three-dimensional model of a multi-stage scene.

7. A storage medium, characterized in that: The computer program for postoperative review of minimally invasive surgery based on multi-stage surgery scene registration is stored therein, wherein the computer program enables the computer to execute the postoperative review method of minimally invasive surgery as described in any one of claims 1 to 5.

8. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including a method for executing a postoperative review method for minimally invasive surgery as described in any one of claims 1 to 5.

Images