Method and system for monitoring region of interest in laparoscopic surgery

Abstract

The present invention provides a method and system for monitoring regions of interest in laparoscopic surgery, and relates to the field of computer vision. Based on a deformation estimation algorithm, the present invention tracks the tissue of interest defined preoperatively in real time on a tissue point cloud during surgery, and combined with real-time estimation of the 6D posture of surgical instruments, it can achieve real-time estimation of the relative distance between high-risk tissue organs and instruments. When the minimum relative distance exceeds a preset threshold, it can comprehensively provide warning marks, the 3D positions of instruments and tissues in the region of interest, and the minimum relative distance between the two. By accurately estimating, visualizing, and warning the distance between surgical instruments and key tissue structures, it is helpful to avoid accidental injury to important tissue structures during surgery and improve the safety of surgery.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method and system for monitoring a region of interest in laparoscopic surgery. Background Art

[0002] In computer vision, a region of interest (ROI) refers to an area in an image or video that is designated as requiring special attention or processing.

[0003] In the related art, patent CN201910899200.8 discloses a multi-level interactive visualization method for surgical navigation. The steps are as follows: first, obtain the multimodal two-dimensional image data output by the medical imaging device, pre-process the multimodal two-dimensional image data, and obtain the tissue and organ image; then reconstruct the two-dimensional tissue and organ image in three dimensions, obtain the 3D model file of the tissue, organ and lesion location and transmit it to the client; the client parses the 3D model file to obtain the contour model; select the area of ​​interest in the 3D model and render it clearly, define the high-risk tissue model and stain it, and finally, through the AR module, superimpose the contour model and the rendered 3D model to the corresponding position of the real-world patient, and display the auxiliary information on the mobile terminal.

[0004] However, the above scheme only estimates the tip of the instrument in the instrument information part, lacking the complete 6D posture of the instrument; the above scheme does not consider the tissue depth information of the instrument occluded area, resulting in the missing tissue depth of the occluded area; the above scheme does not issue a hazard warning based on the relative distance relationship between the instrument and the tissue. In summary, the scheme cannot provide a complete relative posture of the instrument and the tissue, and cannot issue a hazard warning based on the relative distance between the instrument and the tissue. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides a method and system for monitoring the region of interest in laparoscopic surgery, which solves the technical problems that the instrument information part only estimates the tip of the instrument and lacks the complete 6D posture of the instrument; does not consider the tissue depth information of the instrument occluded area, resulting in the missing tissue depth of the occluded part of the area; and cannot provide a complete instrument-tissue relative posture and cannot provide danger warning based on the instrument-tissue relative distance.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A method for monitoring a region of interest in laparoscopic surgery, comprising:

[0010] Tracking the tissue of the region of interest defined before the operation on the intraoperative tissue point cloud based on the deformation estimation algorithm to obtain the three-dimensional coordinates of the region of interest in the camera coordinates of the current frame;

[0011] Using the instrument 6D pose tracking algorithm based on 3D Gaussian sputtering, the optimal instrument 6D pose of the instrument in the current frame camera coordinate system is estimated in real time; the pre-reconstructed instrument Gaussian model is simplified into several cylindrical sides, and the simplified instrument model is updated using the optimal instrument 6D pose to obtain the simplified instrument model in the current frame camera coordinate system;

[0012] According to the three-dimensional coordinates of the region of interest in the current frame camera coordinates, the minimum relative distance between the tissue point cloud of the region of interest and the updated simplified instrument model is obtained.

[0013] Preferably, after obtaining the minimum relative distance, the region of interest monitoring method further includes:

[0014] Projecting the tissue point cloud of the region of interest onto a two-dimensional image plane using a camera model to determine the coordinates of corresponding pixel points;

[0015] According to the relationship between the minimum relative distance and the preset danger distance warning threshold, the pixel value of the B channel of the corresponding pixel point is updated to construct a warning map corresponding to the original projection image;

[0016] and / or

[0017] Update the Z coordinate of the tissue point cloud of the region of interest with the minimum relative distance, construct a new point cloud corresponding to the original pixel points one by one, and use it as a three-dimensional visualization interface;

[0018] If the minimum relative distance is less than a preset danger distance warning threshold, a scale surface is displayed at every preset scale on the three-dimensional visualization interface.

[0019] Preferably, the method of simplifying the pre-reconstructed instrument Gaussian model into a plurality of cylindrical side faces, updating the simplified instrument model using the optimal instrument 6D posture, and obtaining the simplified instrument model in the camera coordinates of the current frame includes:

[0020] Gaussian Model M 0 Simplified instrument model represented by several cylindrical side faces , with articulated instrument center , the direction vector of the joint , the radius of the cylindrical section of the hinged part r k , height of cylindrical side of each hinged part h k To express, , k is the index of the hinged component;

[0021] Initial simplified device model , satisfying the following constraints:

[0022]

[0023] in, Representing the three-dimensional coordinates of a point on the cylindrical side of the device model; ; Indicates k The 3D direction vector of the articulated component; Articulated Equipment Center p c The three-dimensional coordinates of

[0024] Utilize the optimal instrument 6D posture , get the center of the device in the camera coordinates of the current frame and the joint direction vector for:

[0025]

[0026] in, Represents the local coordinate system F 1 to the transformation matrix of the camera coordinate system of the current frame; Represents the local coordinate system F 2 To the local coordinate system F 1's transformation matrix; θ t Represents the local coordinate system F 1. F The Euler angles of a single degree of freedom for rotation between 2; Indicates that p c Transform to local coordinate system F 1 down; Indicates that p c Transform to local coordinate system F 2 down; Represents the direction vector of the joint component in the previous camera coordinates;

[0027] After updating, obtain the simplified instrument model in the camera coordinates of the current frame .

[0028] Preferably, the updating of the pixel value of the B channel of the corresponding pixel point according to the size relationship between the minimum relative distance and the preset danger distance warning threshold value, and the construction of the warning map corresponding to the original projection image comprises:

[0029] Calculate the normalized color value to distinguish the warning, as follows:

[0030]

[0031] in, represents the normalized pixel value updated according to the relative distance between the instrument and tissue, λ Indicates the danger distance warning threshold, Represents the organized point cloud of the area of ​​interest in the camera coordinates of the current frame With updated instrument models The minimum relative distance between , N The number of tissue point clouds representing the region of interest;

[0032] Define the pixel value of the corresponding pixel point on the original projection image as , then the pixel value of the corresponding pixel point on the warning map is , calculated as follows:

[0033]

[0034] in, Respectively represent the pixel values ​​of the R, G, and B channels of the corresponding pixel points of the original projected image. They respectively represent the pixel values ​​of the R, G, and B channels corresponding to the pixel points in the warning image.

[0035] Preferably, the updating of the Z coordinate of the tissue point cloud of the region of interest with the minimum relative distance to construct a new point cloud corresponding to the original pixel points one by one includes:

[0036] The tissue point cloud defining the region of interest is , the new point cloud to be constructed is , and the set point With point Have the same RGB value and X, Y coordinates, let the Z coordinate equal to the tissue point cloud With updated instrument models The minimum relative distance between , calculated as follows:

[0037]

[0038] in, , Respectively indicate points With point RGB values.

[0039] Preferably, the method of using a 6D posture tracking algorithm of an instrument based on 3D Gaussian sputtering to estimate in real time the optimal 6D posture of the instrument in the current frame camera coordinate system comprises:

[0040] Reading the current frame image, obtaining the device mask through a semantic segmentation algorithm; and using a binocular depth estimation algorithm to perform depth estimation on the current frame image, and respectively obtaining a tissue depth map and a device depth map in combination with the device mask;

[0041] Initialize the device posture, perform 3D Gaussian sputtering using the device Gaussian model, and obtain a view, a projection mask, and a depth map under the corresponding posture;

[0042] Wherein, the initialization of the device posture comprises:

[0043] For the first frame image, the rendering view of the instrument Gaussian model on the image plane is obtained in combination with the camera model, the laparoscope is moved so that the image and the area where the rendering view is located coincide with each other, and the instrument posture at this time is used as the initial instrument posture of the first frame image;

[0044] For images other than the first frame, the device pose of the current frame is initialized based on the optimal device pose of the previous frame and the optical flow information of two consecutive frames;

[0045] A total loss function is constructed based on the rendering results, the current frame image, its instrument mask, and the instrument depth map; and the instrument posture is updated and learned through the back propagation algorithm to minimize the total loss function and obtain the optimal instrument 6D posture of the instrument in the current frame camera coordinate system.

[0046] Preferably, the reconstruction process of the Gaussian model of the device includes:

[0047] Acquire several images with known camera poses and containing all views of the device to reconstruct a sparse point cloud of the device;

[0048] Initialize each point of the sparse point cloud as a Gaussian to obtain an initialized Gaussian model of the instrument;

[0049] Combining the aforementioned image sequence consisting of images with known camera poses and containing all viewing angles of the instrument and their corresponding camera poses, iteratively optimizing the initialized instrument Gaussian model to obtain the reconstructed instrument Gaussian model; comprising:

[0050] The color consistency loss function between the 3D Gaussian rendering view and the original image that only retains the instrument part is constructed as follows:

[0051]

[0052] in, Respectively represent the center position, rotation, and size of any Gaussian; , L is the number of images included in the image sequence; L 1 indicates a norm loss; The camera pose is R l The process of performing 3D Gaussian differentiable rendering of an image; G i Indicates i Gaussian points, , I is the number of Gaussians; It indicates the original image from the image sequence where only the instrument part is retained;

[0053] A multi-region alignment loss function is introduced to constrain the projection shape of each articulated part on the reconstructed device Gaussian model on the image plane and the device mask on the original image. M lc Align them separately; the multi-region alignment loss function is as follows:

[0054]

[0055] in, , C Indicates the number of articulated parts of the device; The camera pose is R l The process of performing 3D Gaussian differentiable rendering projection;

[0056] The total loss function of the instrument Gaussian model reconstruction process is:

[0057]

[0058] in, α represents the corresponding weight coefficient;

[0059] By minimizing the total loss function L recons , obtain the reconstructed instrument Gaussian model , as shown below:

[0060]

[0061] in, Indicates i reconstructed Gaussian points; Indicates the value of the variable when the function value is minimum.

[0062] Preferably, the method constructs a total loss function based on the rendering result, the current frame image, its instrument mask, and the instrument depth map; and updates and learns the instrument posture through a back propagation algorithm to minimize the total loss function and obtain the optimal instrument 6D posture of the instrument in the current frame camera coordinate system; including:

[0063] Construct the color consistency loss function between the 3D Gaussian rendering view and the current frame image as follows:

[0064]

[0065] in, t Indicates the current moment; T t Indicates the current frame equipment posture; I t Indicates the current frame image; Represents a rendered view; , Represents the process of performing 3D Gaussian differentiable rendering of an image; M t The instrument mask representing the current image segmentation;

[0066] Construct the regional alignment loss function between the 3D Gaussian differentiable rendering projection image and the device mask of the current frame image as follows:

[0067]

[0068] in, They represent the first c The projection shape of each articulated part on the image plane and the instrument mask of the current frame image; , Represents the process of performing 3D Gaussian differentiable rendering projection; ;

[0069] Construct the depth consistency loss between the rendered depth map and the device depth map of the previous frame as follows:

[0070]

[0071] in, Respectively represent the rendered depth map and the device depth map of the previous frame image; , Represents the process of rendering the depth map of the Gaussian model of the device;

[0072] The total loss function is constructed as:

[0073]

[0074] in,α 1. α 2. α 3 represent the corresponding weight coefficients respectively;

[0075] Then the device posture is adjusted by back propagation algorithm. T t Iterate to minimize L pose , get the optimal 6D pose estimation result of the device in the current frame image , as shown below:

[0076] .

[0077] A region of interest monitoring system for laparoscopic surgery, comprising:

[0078] A region tracking module is used to track the preoperatively defined region of interest tissue on the intraoperative tissue point cloud based on a deformation estimation algorithm to obtain the three-dimensional coordinates of the region of interest in the current frame camera coordinates;

[0079] A pose estimation module is used to estimate the optimal 6D pose of the instrument in the camera coordinate system of the current frame in real time by using the instrument 6D pose tracking algorithm based on 3D Gaussian sputtering; simplify the pre-reconstructed instrument Gaussian model into several cylindrical sides, update the simplified instrument model by using the optimal 6D pose of the instrument, and obtain the simplified instrument model in the camera coordinate system of the current frame;

[0080] The distance calculation module is used to obtain the minimum relative distance between the tissue point cloud of the region of interest and the updated simplified instrument model according to the three-dimensional coordinates of the region of interest in the current frame camera coordinates.

[0081] Preferably, after the distance calculation module obtains the minimum relative distance, the region of interest monitoring system further includes:

[0082] A visualization and early warning module is used to project the tissue point cloud of the region of interest onto a two-dimensional image plane using a camera model to determine the coordinates of corresponding pixel points;

[0083] According to the relationship between the minimum relative distance and the preset danger distance warning threshold, the pixel value of the B channel of the corresponding pixel point is updated to construct a warning map corresponding to the original projection image;

[0084] and / or

[0085] Used to update the Z coordinate of the tissue point cloud of the region of interest with the minimum relative distance, construct a new point cloud corresponding to the original pixel points one by one, and serve as a three-dimensional visualization interface;

[0086] If the minimum relative distance is less than a preset danger distance warning threshold, a scale surface is displayed at every preset scale on the three-dimensional visualization interface.

[0087] A storage medium stores a computer program for monitoring a region of interest in laparoscopic surgery, wherein the computer program enables a computer to execute the region of interest monitoring method described above.

[0088] An electronic device, comprising:

[0089] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include instructions for executing the region of interest monitoring method as described above.

[0090] (III) Beneficial effects

[0091] The present invention provides a method and system for monitoring regions of interest in laparoscopic surgery. Compared with the prior art, the present invention has the following beneficial effects:

[0092] In the present invention, firstly, the tissue of the region of interest defined preoperatively is tracked on the intraoperative tissue point cloud based on the deformation estimation algorithm; secondly, the optimal 6D posture of the instrument in the current frame camera coordinate system is estimated in real time using the instrument 6D posture tracking algorithm based on 3D Gaussian sputtering; the pre-reconstructed instrument Gaussian model is simplified into several cylindrical sides, and the simplified instrument Gaussian model is updated using the optimal instrument 6D posture to obtain the instrument model in the current frame camera coordinate; then, according to the three-dimensional coordinates of the region of interest in the current frame camera coordinate, the minimum relative distance between the tissue point cloud of the region of interest and the updated simplified instrument model is obtained. The present invention, combined with the complete instrument posture, can provide more accurate navigation information, and the relative position estimation method of the instrument and tissue is more robust and can be applied to complex scenes in the body. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0094] Figure 1 A block diagram of a method for monitoring a region of interest in laparoscopic surgery is provided for an embodiment of the present invention;

[0095] Figure 2A block diagram of another method for monitoring a region of interest in laparoscopic surgery is provided for an embodiment of the present invention;

[0096] Figure 3 A flowchart of device posture estimation provided by an embodiment of the present invention;

[0097] Figure 4 A schematic diagram of an articulated motion coordinate system is provided for an embodiment of the present invention;

[0098] Figure 5 A schematic diagram of an instrument model (ie, a simplified instrument Gaussian model) is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0099] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0100] The embodiments of the present application provide a method and system for monitoring the region of interest in laparoscopic surgery, thereby solving the technical problems that the instrument information portion only estimates the tip of the instrument and lacks a complete 6D posture of the instrument; does not consider the tissue depth information of the instrument occluded area, resulting in the missing tissue depth of the occluded part of the area; and cannot provide a complete instrument-tissue relative posture and cannot provide danger warning based on the instrument-tissue relative distance.

[0101] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0102] By real-time detection and early warning of areas of interest such as surgical areas of interest, it helps doctors plan and perform operations more accurately and ensures a safe distance between surgical instruments and high-risk organ tissues, which helps improve the accuracy and success rate of operations and reduce the occurrence of surgical complications.

[0103] The embodiment of the present invention designs a method and system for monitoring and warning of regions of interest such as high-risk tissues and organs in laparoscopic surgery. By tracking the tissues of the region of interest defined preoperatively on the intraoperative tissue point cloud based on a deformation estimation algorithm (for example, the three-dimensional model of the tissues and organs built preoperatively and the point cloud of the intraoperative surgical scene are real-time registered to achieve the tracking purpose), combined with the real-time estimation of the 6D posture of the surgical instrument, the real-time estimation of the relative distance between the high-risk tissues and organs and the instrument can be achieved. When the minimum relative distance exceeds the preset threshold, the warning mark, the 3D position of the instrument and the organ, and the minimum relative distance between the two can be fully provided. By accurately estimating, visualizing and warning the distance between the surgical instrument and the key tissue structure, it is helpful to avoid accidental injury to important tissue structures during surgery and improve the safety of surgery.

[0104] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0105] Embodiment 1:

[0106] like Figure 1 As shown, an embodiment of the present invention provides a method for monitoring a region of interest in laparoscopic surgery, comprising:

[0107] S1. Tracking the tissue of the region of interest defined before the operation on the intraoperative tissue point cloud based on the deformation estimation algorithm to obtain the three-dimensional coordinates of the region of interest in the camera coordinates of the current frame;

[0108] S2. Using a 6D instrument pose tracking algorithm based on 3D Gaussian sputtering, the optimal 6D instrument pose of the instrument in the current frame camera coordinate system is estimated in real time; the pre-reconstructed instrument Gaussian model is simplified into several cylindrical sides, and the simplified instrument model is updated using the optimal 6D instrument pose to obtain the simplified instrument model in the current frame camera coordinate system;

[0109] S3. Acquire the minimum relative distance between the tissue point cloud of the region of interest and the updated simplified instrument model according to the three-dimensional coordinates of the region of interest in the current frame camera coordinates.

[0110] The embodiment of the present invention combines the complete instrument posture and removes instrument occlusion to obtain the complete three-dimensional information of the surgical scene, which can provide more accurate navigation information. The relative position estimation method of the instrument and tissue is more robust and can be applied to complex scenes inside the body.

[0111] In an optional embodiment, if Figure 2 As shown, after obtaining the minimum relative distance, the region of interest monitoring method further includes:

[0112] S41, projecting the tissue point cloud of the region of interest onto a two-dimensional image plane using a camera model, determining the coordinates of corresponding pixel points; updating the pixel value of the B channel of the corresponding pixel point according to the magnitude relationship between the minimum relative distance and a preset danger distance warning threshold, and constructing a warning map corresponding to the original projected image;

[0113] S42. Update the Z coordinate of the tissue point cloud of the area of ​​interest with the minimum relative distance, construct a new point cloud corresponding to the original pixel point one by one, and use it as a three-dimensional visualization interface; if the minimum relative distance is less than a preset danger distance warning threshold, display a scale surface at every preset scale on the three-dimensional visualization interface.

[0114] The embodiment of the present invention is based on real-time tracking of high-risk tissues and organs and other areas of interest and surgical instruments. After obtaining the relative distance information between high-risk tissues and other tissues and organs and dangerous instruments during surgery, technical means such as early warning maps and contour plane representations can be used to display the relative position of the instrument to the high-risk tissue in real time on the endoscopic image end, assisting doctors in making decisions and planning surgical paths.

[0115] Next, we will introduce each step of the above solution in detail:

[0116] In step S1, the region of interest defined before the operation is tracked on the intraoperative tissue point cloud to obtain the three-dimensional coordinates of the region of interest tissue in the current frame camera coordinates.

[0117] It should be noted that the region of interest here can be a high-risk tissue organ such as a blood vessel, or other tissue organs. In other words, this step can be set to track any type of region of interest defined before the operation.

[0118] Specifically, the physician defines the tissue of interest on a preoperative in vivo tissue model. During the operation, based on the existing preoperative and intraoperative registration algorithm, the preoperative tissue model and the intraoperative tissue point cloud are simultaneously input into the registration network, the features of the preoperative model and the intraoperative tissue point cloud are extracted, and then the point matching relationship between the preoperative model and the intraoperative point cloud is output. Through the obtained point matching relationship, the deformation variable from the preoperative model to the intraoperative point cloud (i.e., the coordinate difference between the preoperative matching points) is calculated to obtain the tissue of interest defined preoperatively. 3D shape of , and then obtain the three-dimensional position of the tissue of interest in the camera coordinate system of the current frame , t Indicates t The frame image is as follows:

[0119]

[0120] Thus, the three-dimensional coordinates of the region of interest in the current frame camera coordinates can be obtained. ;in, , N The number of tissue point clouds representing the region of interest.

[0121] In step S2, the optimal 6D pose of the instrument in the current frame camera coordinate system is estimated in real time using the instrument 6D pose tracking algorithm based on 3D Gaussian sputtering; the pre-reconstructed instrument Gaussian model is simplified into several cylindrical side faces, and the simplified instrument model is updated using the optimal 6D pose of the instrument to obtain the simplified instrument model in the current frame camera coordinate system.

[0122] On the one hand, in this step, the 6D pose tracking algorithm of the instrument based on 3D Gaussian sputtering is used to estimate the optimal 6D pose of the instrument in the camera coordinate system of the current frame in real time, including:

[0123] S101, reading the current frame image, obtaining the instrument mask through a semantic segmentation algorithm; and performing depth estimation on the current frame image using a binocular depth estimation algorithm, and respectively obtaining a tissue depth map and an instrument depth map in combination with the instrument mask.

[0124] Specifically, the image is segmented using a semantic segmentation algorithm (random forest, SAM, PerSAM, etc.) to obtain the device mask on the image. ,in K Indicates the number of articulations of the instrument. And the depth of the endoscope image is estimated using a binocular depth estimation method (MIOL algorithm, CREStereo algorithm, etc.) to obtain a depth map D t , and combined with the device mask M t , and obtain the tissue depth map of the non-device occluded area D t (1- M t ) and the device depth map of the visible part of the device D t M t .in, t Indicates the current moment.

[0125] S102, initializing the posture of the instrument, performing 3D Gaussian sputtering using the Gaussian model of the instrument, and obtaining a view, a projection mask, and a depth map under a corresponding posture.

[0126] Note that 3D Gaussian splatter rendering is mainly used for new view synthesis. The 3D position of the object represented by the 3D Gaussian is determined by the center position of each Gaussian. , Rotate ,size ,color ,density σ Together, it is not suitable for expressing the 6D pose transformation of dynamic objects.

[0127] Accordingly, the embodiment of the present invention pre-establishes a 3D Gaussian model of the surgical instrument (ie, a reconstructed Gaussian model of the instrument), thereby converting the 3D Gaussian optimization into Gaussian posture optimization, ie, posture optimization of the surgical instrument.

[0128] Specifically, the reconstruction process of the Gaussian model of the device includes:

[0129] First, obtain several images with known camera poses and all viewing angles of the instrument to reconstruct the sparse point cloud of the instrument; specifically,

[0130] a) Fix the position of the surgical instrument, aim the camera at the instrument and shoot around it to obtain several instrument images covering all viewing angles. I l , , L is the number of images. However, due to the textureless and specular reflection characteristics of surgical instruments, it is not easy to solve the relative camera pose through the image features between adjacent frames. Feature points can be provided by attaching marker points on the instrument or fixing a checkerboard in the shooting field of view.

[0131] b) The sparse point cloud of the captured scene is reconstructed through SFM, as well as the 2D-3D corresponding points between the image and the SFM point cloud.

[0132] c) Scale the sfm point cloud according to the real scale of the device, restore the absolute scale of the sparse point cloud, and use the 2D-3D corresponding points to obtain the absolute camera pose of each image R l .

[0133] d) Use the segmentation algorithm to segment the instrument and obtain the segmentation mask of the instrument to remove the background outside the instrument in the image and obtain an image with only the instrument .

[0134] e) According to the camera pose R l Known images , use the SFM algorithm to reconstruct the sparse point cloud of the instrument M sfm , used as initialization for 3D Gaussian model reconstruction.

[0135] Then, each point of the sparse point cloud is initialized as a Gaussian G i , , the properties of each Gaussian are: the center position of the Gaussian , Rotate ,size ,color ,density σ , the probability of belonging to the foreground f .

[0136] It should be noted that the parameters that need to be optimized in the reconstruction process of the surgical instrument Gaussian model in the embodiment of the present invention are only: the center position of each Gaussian , Rotate ,size .

[0137] In addition, the instrument Gaussian model at this stage { G i} is initialized as follows: Equipment point cloud M sfm Each point is initialized as a Gaussian G i , , I is the number of Gaussians, and the position and color of the point cloud are initialized to the center position of the Gaussians u and color c , Gaussian rotation q Initialized to {1,0,0,0}, the distance from each point to the nearest neighbor is initialized to the size of the Gaussian, and the density of the Gaussian σ Initialized to 1, the probability of belonging to the foreground f Initialized to 1.

[0138] On this basis, the image sequence composed of the images of the aforementioned known camera poses and containing all viewing angles of the instrument and its corresponding camera poses is combined to iteratively optimize the initialized instrument Gaussian model to obtain the reconstructed instrument Gaussian model; including:

[0139] The color consistency loss function between the 3D Gaussian rendering view and the original image that only retains the instrument part is constructed as follows:

[0140]

[0141] in, L 1 indicates a norm loss; It indicates the original image from the image sequence where only the instrument part is retained; The camera pose is R l The process of performing 3D Gaussian differentiable rendering of an image.

[0142] In addition, to ensure that the reconstructed instrument Gaussian model is an articulated structure, the embodiment of the present invention introduces a multi-region alignment loss function to constrain the projection shape of each articulated part on the reconstructed instrument Gaussian model on the image plane and the instrument mask on the original image. M lcAlign them separately; the multi-region alignment loss function is as follows:

[0143]

[0144] in, , C Indicates the number of articulated parts of the device; The camera pose is R l The process of performing 3D Gaussian differentiable rendering projection.

[0145] Then the total loss function of the instrument Gaussian model reconstruction process is:

[0146]

[0147] in, α represents the corresponding weight coefficient;

[0148] By minimizing the total loss function L recons , obtain the reconstructed instrument Gaussian model , as shown below:

[0149]

[0150] in, Indicates i reconstructed Gaussian points; Indicates the value of the variable when the function value is minimum.

[0151] Optimized Gaussian model of the device , denoted as , i Represents the index of the Gaussian kernel.

[0152] Getting the Gaussian model of the instrument Based on the rigid motion between the articulated parts of the surgical instrument T In the embodiment of the present invention, the DH parameter method is used to define the articulated motion of the instrument, and the rod and the left clamp (the right clamp is symmetrical to the left clamp) of the instrument are regarded as a two-link structure, as shown in the schematic diagram. Figure 5 As shown in the figure, local coordinate systems are defined for the instrument rod and the left clamp (the right clamp is symmetrical to the left clamp) respectively. F 1. F 2. Establish the camera coordinate system F cam The device pose estimation solves the transformation matrix from the device's own coordinate system to the camera coordinate system. At this time, the local coordinates of the device are F 1. F 2 points below X , the formula for transforming to the camera coordinate system is as follows:

[0153]

[0154] in, Represents the local coordinate system F 1. F 2 points below, Figure 5 The points of the left (right) clamp are in the local coordinate system F 2. The points on the instrument rod are in the local coordinate system F 1, is the instrument rod coordinate system F 1 to the camera coordinate system F cam The transformation matrix, It is the coordinate system of the left clamp of the instrument F 2 to the instrument rod coordinate system F 1's transformation matrix, the left clamp coordinate system of the instrument F 2 Must be around the rod coordinate system F 1 z-axis rotation, so one degree of freedom in Euler angles θ The local coordinate system F 1. F 2-room rotation.

[0155] In summary, the rigidity transformation of the articulated device T It can be expressed as ,in , R represents a 3D rotation matrix, Represents a 3D translation vector.

[0156] After clarifying the reconstruction process of the Gaussian model of the instrument and the rigid motion parameters corresponding to the Gaussian model, Figure 3 and Figure 4 As shown, based on the defined Gaussian model, the pose parameters of the instrument are optimized to obtain the 6D pose of the articulated instrument. Given the current frame binocular image, the current frame depth map is obtained by the binocular depth estimation algorithm, and the segmentation mask of the instrument is obtained by the instrument segmentation algorithm. Among them, the input image, depth map and instrument mask are used as the supervision information for subsequent pose learning. Based on the initialization position of the instrument, the view, depth map and projection mask under the corresponding pose are rendered by 3D Gaussian sputtering ( Figure 3 It is expressed as rendered image, segmentation, and depth information, and is compared with the original image, semantic segmentation mask, and depth estimation result ( Figure 3 The loss functions are constructed separately for the image, segmentation, and depth information (expressed in [1]), so that the subsequent steps can construct the total loss function based on this, and then the device posture is updated and learned through the back propagation algorithm to obtain the optimal 6D posture of the device in the current frame image.

[0157] In the stage of device posture optimization, the update of the device Gaussian model follows the following formula:

[0158]

[0159] in i represents the index of the Gaussian kernel, t Indicates the current frame. is the property of the reconstructed instrument Gaussian model, u i,t The update analogy formula (a) ~ formula (b) is as follows:

[0160] when u i,0 When it is a point belonging to the rod of the instrument, it is updated as follows:

[0161]

[0162] in, Indicates that u i,0 Convert to local coordinate system F 1 is the transformation matrix; It is determined after the instrument Gaussian model is reconstructed and the corresponding posture parameters are defined.

[0163] when u i,0 When it belongs to the left (right) clamp of the instrument, update as follows:

[0164]

[0165] in, Indicates that u i,0 Convert to local coordinate system F 2 is the transformation matrix; It is determined after the instrument Gaussian model is reconstructed and the corresponding posture parameters are defined.

[0166] The initialization of the instrument posture is mainly divided into two cases.

[0167] First, for the first frame of image, the rendering view of the instrument Gaussian model on the image plane is obtained in combination with the camera model, the laparoscope is moved so that the image and the area where the rendering view is located coincide with each other, and the instrument posture at this time is used as the initial instrument posture of the first frame of image;

[0168] Second, for images other than the first frame, the device pose of the current frame is initialized based on the optimal device pose of the previous frame and the optical flow information of two consecutive frames.

[0169] Current frame image I tInitial instrument posture From the previous frame of equipment pose T t-1 and a rough estimate of the 3D translation get:

[0170]

[0171] in, The calculation of is as follows:

[0172]

[0173] in, Represents the mean value of the predicted optical flow from the previous frame to the current frame in the instrument area, Represents the average depth value of the instrument area in the previous frame, f x , f y Respectively indicated in the camera x , y The focal length in the direction.

[0174] S103, constructing a total loss function according to the rendering result, the current frame image, its instrument mask, and the instrument depth map; and updating and learning the instrument posture through the back propagation algorithm, minimizing the total loss function, and obtaining the optimal instrument 6D posture of the instrument in the current frame camera coordinate system.

[0175] Furthermore, the present invention implements the optimization of setting the device posture mainly by rendering the view and the original image I t Loss of color consistency between L im , Rendered instrument projection mask and instrument segmentation mask M t The region alignment loss between L region , Rendered equipment depth map And the device depth estimation results D t The depth consistency loss between L depth To construct the optimization target.

[0176] In the iteratively updated device pose T t Under the action of , the color consistency loss function between the 3D Gaussian rendering view and the current frame image is constructed as follows:

[0177]

[0178] in, t Indicates the current moment; T t Indicates the current frame equipment posture; I t Indicates the current frame image; Represents a rendered view; , Represents the process of performing 3D Gaussian differentiable rendering of an image; M t The instrument mask representing the current image segmentation;

[0179] Construct the regional alignment loss function between the 3D Gaussian differentiable rendering projection image and the device mask of the current frame image as follows:

[0180]

[0181] in, They represent the first c The projection shape of each articulated part on the image plane and the instrument mask of the current frame image; , Represents the process of performing 3D Gaussian differentiable rendering projection; ;

[0182] Construct the depth consistency loss between the rendered depth map and the device depth map of the previous frame as follows:

[0183]

[0184] in, Respectively represent the rendered depth map and the device depth map of the previous frame image; , Represents the process of rendering the depth map of the Gaussian model of the device;

[0185] Therefore, the total loss function is constructed as:

[0186]

[0187] in, α 1. α 2. α 3 represent the corresponding weight coefficients respectively;

[0188] Then the device posture is adjusted by back propagation algorithm. T t Iterate to minimize L pose , get the optimal 6D pose estimation result of the device in the current frame image , as shown below:

[0189]

[0190] The embodiment of the present invention converts the instrument posture optimization into 3D Gaussian position optimization based on the 3D Gaussian instrument posture tracking algorithm, utilizes the fast rendering of 3D Gaussian sputtering, and uses the optical flow information to perform rough posture estimation to reduce the number of iterations of 3D Gaussian optimization, thereby solving the time-consuming problem of traditional iterative optimization of posture.

[0191] On the other hand, the embodiment of the present invention takes into account the problem that it takes a long time to calculate the instrument-tissue relative distance by traversing the instrument point cloud, which is difficult to apply in surgery. Therefore, before calculating the instrument-tissue relative distance, in this step, the pre-reconstructed instrument Gaussian model is simplified into several cylindrical side faces, and the simplifier model is updated using the optimal instrument 6D pose to obtain a simplified instrument model in the current frame camera coordinates; including:

[0192] like Figure 5 As shown, the Gaussian model of the instrument M 0 to articulated device center , the direction vector of the joint , the radius of the cylindrical section of the hinged part r k , height of cylindrical side of each hinged part h k To express, , k is the index of the hinged component;

[0193] Initial device model , satisfying the following constraints:

[0194]

[0195] in, Representing the three-dimensional coordinates of a point on the cylindrical side of the device model; ; Indicates k The 3D direction vector of the articulated component; Articulated Equipment Center p c The three-dimensional coordinates of

[0196] Utilize the optimal instrument 6D posture , get the center of the device in the camera coordinates of the current frame and the joint direction vector for:

[0197] The center of the device in the camera coordinates of the current frame The calculation of is as follows:

[0198]

[0199] The direction vector of the joint in the camera coordinates of the current frame The calculation of is as follows:

[0200]

[0201] in, Represents the local coordinate system F 1 to the transformation matrix of the camera coordinate system of the current frame; Represents the local coordinate system F 2 To the local coordinate system F 1's transformation matrix; θ t Represents the local coordinate system F 1. F The Euler angles of a single degree of freedom for rotation between 2; Indicates that p c Transform to local coordinate system F 1 down; Indicates that p c Transform to local coordinate system F 2 down; Represents the direction vector of the joint component in the previous camera coordinates;

[0202] After updating, obtain the simplified instrument model in the camera coordinates of the current frame .

[0203] In step S3, the minimum relative distance between the tissue point cloud of the region of interest and the updated simplified instrument model is obtained according to the three-dimensional coordinates of the region of interest in the current frame camera coordinates.

[0204] Obtain the three-dimensional coordinates of the region of interest in the current frame camera coordinates , and instrument models Then, the minimum relative distance between the tissue point cloud of the region of interest and the updated simplified instrument model can be calculated according to formula (c). , , N The number of tissue point clouds representing the region of interest, Solve according to the Lagrange multiplier method:

[0205]

[0206] Among them, point ( x,y,z) satisfies the constraints given in formula (1); then, a Lagrangian function is constructed for the above objective function and constraints to solve the shortest distance from each tissue point cloud to the device. .

[0207] In particular, after obtaining the minimum relative distance, the embodiment of the present invention further includes a step of visualizing the surgical region of interest and providing an early warning, specifically:

[0208] In step S41, the tissue point cloud of the area of ​​interest is projected onto a two-dimensional image plane using a camera model to determine the coordinates of the corresponding pixel points; based on the relationship between the minimum relative distance and the preset danger distance warning threshold, the pixel value of the B channel of the corresponding pixel point is updated to construct a warning map corresponding to the original projected image.

[0209] First, the high-risk tissue is passed through the camera model to obtain the position coordinates of the pixel points on the two-dimensional image plane. , and then obtain the original projection image of the region of interest on the display end of the endoscope video in real time, which is calculated as follows:

[0210]

[0211] in, It represents the process of projecting a 3D point in the camera coordinate system into a 2D image plane in combination with the camera intrinsic parameters, where K represents the camera intrinsic parameters;

[0212] Afterwards, according to the magnitude relationship between the minimum relative distance and the preset danger distance warning threshold, the pixel value of the B channel of the corresponding pixel point is updated to construct a warning map corresponding to the original projection image; including:

[0213] Calculate the normalized color value for distinguishing warnings as follows (after the following formula, the normalized color value will tend to 0 and 1):

[0214]

[0215] in, represents the normalized pixel value updated according to the relative distance between the instrument and tissue, λ Indicates the danger distance warning threshold, Represents the organized point cloud of the area of ​​interest in the camera coordinates of the current frame With updated instrument models The minimum relative distance between , N The number of tissue point clouds representing the region of interest.

[0216] Define the pixel value of the corresponding pixel point on the original projection image as , then the pixel value of the corresponding pixel point on the warning map is , calculated as follows:

[0217]

[0218] in, Respectively represent the pixel values ​​of the R, G, and B channels of the corresponding pixel points of the original projected image. They respectively represent the pixel values ​​of the R, G, and B channels corresponding to the pixel points in the warning image.

[0219] It should be noted that the early warning image here refers to the area of ​​interest that is significantly highlighted based on the relative distance between the device and the tissue on the basis of the original image. The early warning image will significantly display the high-risk area when the minimum relative distance between the device and the high-risk tissue is less than the set threshold. When the minimum relative distance between the device and the high-risk tissue is less than the set threshold, the high-risk area will be significantly displayed, and when the doctor needs it, the minimum relative distance between the device and the high-risk tissue, as well as the predicted contact point between the tissue and the device can be viewed.

[0220] Furthermore, considering that the above-mentioned qualitative visualization interface is not intuitive enough for doctors to give instructions and is ambiguous, the embodiment of the present invention also adds a three-dimensional visualization interface for quantitative display.

[0221] Specifically:

[0222] In step S42, the Z coordinate of the tissue point cloud of the area of ​​interest is updated with the minimum relative distance, and a new point cloud corresponding one-to-one to the original pixel point is constructed and used as a three-dimensional visualization interface; if the minimum relative distance is less than the preset danger distance warning threshold, a scale surface is displayed on the three-dimensional visualization interface at every preset scale.

[0223] The updating of the Z coordinates of the tissue point cloud of the region of interest with the minimum relative distance to construct a new point cloud corresponding to the minimum relative distance comprises:

[0224] The tissue point cloud defining the region of interest is , the new point cloud to be constructed is , and the set point With point Have the same RGB value and X, Y coordinates, let the Z coordinate equal to the tissue point cloud With updated instrument models The minimum relative distance between , calculated as follows:

[0225]

[0226] in, , Respectively indicate points With point RGB values.

[0227] For example, when the minimum relative distance Less than the warning threshold λ When using the instrument, it can be set to display a scale every 2mm, thus providing doctors with more accurate quantitative and qualitative data.

[0228] In fact, in the embodiment of the present invention, the early warning standard of the surgical region of interest can be set according to the specific situation of the patient, so that the surgical process is safer and more effective.

[0229] At this point, the embodiment of the present invention has completed real-time detection of intraoperative regions of interest by real-time tracking of regions of interest such as high-risk tissues and organs in the body and combining them with the three-dimensional position of surgical instruments, providing doctors with quantitative relative positions of instruments and high-risk tissues to assist them in surgical path planning.

[0230] Embodiment 2:

[0231] The embodiment of the present invention provides a system for monitoring a region of interest in laparoscopic surgery, comprising:

[0232] A region tracking module is used to track the preoperatively defined region of interest tissue on the intraoperative tissue point cloud based on a deformation estimation algorithm to obtain the three-dimensional coordinates of the region of interest in the current frame camera coordinates;

[0233] A pose estimation module is used to estimate the optimal 6D pose of the instrument in the camera coordinate system of the current frame in real time by using the instrument 6D pose tracking algorithm based on 3D Gaussian sputtering; simplify the pre-reconstructed instrument Gaussian model into several cylindrical sides, update the simplified instrument model by using the optimal 6D pose of the instrument, and obtain the simplified instrument model in the camera coordinate system of the current frame;

[0234] The distance calculation module is used to obtain the minimum relative distance between the tissue point cloud of the region of interest and the updated simplified instrument model according to the three-dimensional coordinates of the region of interest in the current frame camera coordinates.

[0235] In an optional embodiment, after the distance calculation module obtains the minimum relative distance, the region of interest monitoring system further includes:

[0236] A visualization and early warning module is used to project the tissue point cloud of the region of interest onto a two-dimensional image plane using a camera model to determine the coordinates of corresponding pixel points;

[0237] According to the relationship between the minimum relative distance and the preset danger distance warning threshold, the pixel value of the B channel of the corresponding pixel point is updated to construct a warning map corresponding to the original projection image;

[0238] and for updating the Z coordinate of the tissue point cloud of the region of interest with the minimum relative distance, constructing a new point cloud corresponding to the original pixel points one by one, and serving as a three-dimensional visualization interface;

[0239] If the minimum relative distance is less than a preset danger distance warning threshold, a scale surface is displayed at every preset scale on the three-dimensional visualization interface.

[0240] Embodiment 3:

[0241] An embodiment of the present invention provides a storage medium storing a computer program for monitoring a region of interest in laparoscopic surgery, wherein the computer program enables a computer to execute the region of interest monitoring method as described in Example 1.

[0242] Embodiment 4:

[0243] An embodiment of the present invention provides an electronic device, including:

[0244] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the region of interest monitoring method as described in Example 1.

[0245] It is understandable that the region of interest monitoring system, storage medium and electronic device for laparoscopic surgery provided in the embodiments of the present invention correspond to the region of interest monitoring method for laparoscopic surgery provided in the embodiments of the present invention, and the explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the region of interest monitoring method, which will not be repeated here.

[0246] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0247] 1. The embodiment of the present invention combines the complete instrument posture and removes the occlusion of the instrument on the tissue to obtain the complete three-dimensional information of the surgical scene, which can provide more accurate navigation information. The relative position estimation method of the instrument and tissue is more robust and can be applied to complex scenes in the body.

[0248] 2. The embodiment of the present invention is based on real-time tracking of high-risk tissues and organs and other areas of interest and surgical instruments. After obtaining the relative distance information between high-risk tissues and other tissues and organs and dangerous instruments during surgery, technical means such as two-dimensional early warning maps or three-dimensional contour surface visualization interfaces can be used to display the relative position of the instrument to the high-risk tissue in real time on the endoscopic image end, thereby assisting doctors in making decisions and planning surgical paths.

[0249] 3. The embodiment of the present invention converts the device posture optimization into 3D Gaussian position optimization based on the 3D Gaussian device posture tracking algorithm, utilizes the fast rendering speed of 3D Gaussian sputtering, and performs rough estimation of the device posture based on optical flow information to reduce the number of iterations of 3D Gaussian optimization, thereby speeding up the optimization speed and solving the time-consuming problem of traditional iterative optimization of posture.

[0250] 4. In the embodiment of the present invention, the early warning standard of the surgical area of ​​interest can be set according to the patient's specific situation, making the surgical process safer and more effective.

[0251] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0252] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring a region of interest in laparoscopic surgery, characterized in that: include: Tracking the tissue region of interest defined preoperatively on the intraoperative tissue point cloud based on a deformation estimation algorithm to obtain the three-dimensional coordinates of the tissue region of interest in the current frame camera coordinates; Using the instrument 6D pose tracking algorithm based on 3D Gaussian sputtering, the optimal instrument 6D pose of the instrument in the current frame camera coordinate system is estimated in real time; the pre-reconstructed instrument Gaussian model is simplified into several cylindrical sides, and the simplified instrument model is updated using the optimal instrument 6D pose to obtain the simplified instrument model in the current frame camera coordinate system; According to the three-dimensional coordinates of the region of interest in the current frame camera coordinates, obtaining the minimum relative distance between the tissue point cloud of the region of interest and the updated simplified instrument model; The pre-reconstructed instrument Gaussian model is simplified into a number of cylindrical sides, and the simplified instrument model is updated using the optimal instrument 6D posture to obtain the simplified instrument model in the camera coordinates of the current frame; include: Gaussian Model M 0 Simplified instrument model represented by several cylindrical side faces , with articulated instrument center , the direction vector of the joint , radius of the cylindrical section of the hinged part r k , height of cylindrical side of each hinged part h k To express, , k is the index of the hinged component; Initial simplified device model , satisfying the following constraints: in, The three-dimensional coordinates of a point on the cylindrical side surface representing the simplified device model; ; Indicates k The 3D direction vector of the articulated component; Articulated Equipment Center p c The three-dimensional coordinates of Utilize the optimal instrument 6D posture , get the center of the device in the camera coordinates of the current frame and the joint direction vector for: in, Represents the local coordinate system F 1 to the transformation matrix of the camera coordinate system of the current frame; Represents the local coordinate system F 2 To the local coordinate system F 1's transformation matrix; θ t Represents the local coordinate system F 1. F The Euler angles of a single degree of freedom for rotation between 2; Indicates that p c Transform to local coordinate system F 1 down; Indicates that p c Transform to local coordinate system F 2 down; Represents the direction vector of the joint component in the previous camera coordinates; After updating, obtain the simplified instrument model in the camera coordinates of the current frame .

2. The method for monitoring a region of interest according to claim 1, wherein: After obtaining the minimum relative distance, the region of interest monitoring method further includes: Projecting the tissue point cloud of the region of interest onto a two-dimensional image plane using a camera model to determine the coordinates of corresponding pixel points; According to the relationship between the minimum relative distance and the preset danger distance warning threshold, the pixel value of the B channel of the corresponding pixel point is updated to construct a warning map corresponding to the original projection image; and / or Update the Z coordinate of the tissue point cloud of the region of interest with the minimum relative distance, construct a new point cloud corresponding to the original pixel points one by one, and use it as a three-dimensional visualization interface; If the minimum relative distance is less than a preset danger distance warning threshold, a scale surface is displayed at every preset scale on the three-dimensional visualization interface.

3. The method for monitoring a region of interest according to claim 2, wherein: The method of updating the pixel value of the B channel of the corresponding pixel point according to the size relationship between the minimum relative distance and the preset danger distance warning threshold value, and constructing a warning map corresponding to the original projection image, comprises: Calculate the normalized color value to distinguish the warning, as follows: in, represents the normalized pixel value updated according to the relative distance between the instrument and tissue, λ Indicates the danger distance warning threshold, Represents the organized point cloud of the area of ​​interest in the camera coordinates of the current frame With updated instrument models The minimum relative distance between , N The number of tissue point clouds representing the region of interest; Define the pixel value of the corresponding pixel point on the original projection image as , then the pixel value of the corresponding pixel point on the warning map is , calculated as follows: in, Respectively represent the pixel values ​​of the R, G, and B channels of the corresponding pixel points of the original projected image. Respectively represent the pixel values ​​of the R, G, and B channels corresponding to the pixel points in the warning image; and / or The method of updating the Z coordinate of the tissue point cloud of the region of interest with the minimum relative distance to construct a new point cloud corresponding to the original pixel points one by one includes: The tissue point cloud defining the region of interest is , the new point cloud to be constructed is , and the set point With point Have the same RGB value and X, Y coordinates, let the Z coordinate equal to the tissue point cloud With updated instrument models The minimum relative distance between , calculated as follows: in, , Respectively indicate points With point RGB values.

4. The method for monitoring a region of interest according to claim 1 or 2, wherein: The device 6D posture tracking algorithm based on 3D Gaussian sputtering is used to estimate the optimal device 6D posture of the device in the current frame camera coordinate system in real time; including: Reading the current frame image, obtaining the device mask through a semantic segmentation algorithm; and performing depth estimation on the current frame image using a binocular depth estimation algorithm, and respectively obtaining a tissue depth map and a device depth map in combination with the device mask; Initialize the device posture, perform 3D Gaussian sputtering using the device Gaussian model, and obtain a view, a projection mask, and a depth map under the corresponding posture; Wherein, the initialization of the device posture comprises: For the first frame of image, a rendering view of the instrument Gaussian model on the image plane is obtained in combination with the camera model, the laparoscope is moved so that the image coincides with the area where the rendering view is located, and the instrument posture at this time is used as the initial instrument posture of the first frame of image; For images other than the first frame, the device pose of the current frame is initialized based on the optimal device pose of the previous frame and the optical flow information of two consecutive frames; A total loss function is constructed based on the rendering results, the current frame image, its instrument mask, and the instrument depth map; and the instrument posture is updated and learned through the back propagation algorithm to minimize the total loss function and obtain the optimal instrument 6D posture of the instrument in the current frame camera coordinate system.

5. The method for monitoring a region of interest according to claim 4, wherein: The reconstruction process of the Gaussian model of the device includes: Acquire several images with known camera poses and containing all views of the device to reconstruct a sparse point cloud of the device; Initialize each point of the sparse point cloud as a Gaussian to obtain an initialized Gaussian model of the instrument; Combining the aforementioned image sequence consisting of images with known camera poses and containing all viewing angles of the instrument and their corresponding camera poses, iteratively optimizing the initialized instrument Gaussian model to obtain the reconstructed instrument Gaussian model; comprising: The color consistency loss function between the 3D Gaussian rendering view and the original image that only retains the instrument part is constructed as follows: in, Respectively represent the center position, rotation, and size of any Gaussian; , L is the number of images included in the image sequence; L 1 indicates a norm loss; The camera pose is R l The process of performing 3D Gaussian differentiable rendering of images; G i Indicates i Gaussian points, , I is the number of Gaussians; It indicates the original image from the image sequence where only the instrument part is retained; A multi-region alignment loss function is introduced to constrain the projection shape of each articulated part on the reconstructed device Gaussian model on the image plane and the device mask on the original image. M lc Align them separately; the multi-region alignment loss function is as follows: in, , C Indicates the number of articulated parts of the device; The camera pose is R l The process of performing 3D Gaussian differentiable rendering projection; The total loss function of the instrument Gaussian model reconstruction process is: in, α represents the corresponding weight coefficient; By minimizing the total loss function L recons , obtain the reconstructed instrument Gaussian model , as shown below: in, Indicates i reconstructed Gaussian points; Indicates the value of the variable when the function value is minimum; and / or The method constructs a total loss function based on the rendering result, the current frame image, the device mask, and the device depth map; and updates and learns the device pose through a back propagation algorithm to minimize the total loss function and obtain the optimal device 6D pose of the device in the current frame camera coordinate system; including: Construct the color consistency loss function between the 3D Gaussian rendering view and the current frame image as follows: in, t Indicates the current moment; T t Indicates the current frame equipment posture; I t Indicates the current frame image; Represents a rendered view; , Represents the process of performing 3D Gaussian differentiable rendering of an image; M t The instrument mask representing the current image segmentation; Construct the regional alignment loss function between the 3D Gaussian differentiable rendering projection image and the device mask of the current frame image as follows: in, They represent the first c The projection shape of each articulated part on the image plane and the instrument mask of the current frame image; , Represents the process of performing 3D Gaussian differentiable rendering projection; ; Construct the depth consistency loss between the rendered depth map and the device depth map of the previous frame as follows: in, Respectively represent the rendered depth map and the device depth map of the previous frame image; , Represents the process of rendering the depth map of the Gaussian model of the device; The total loss function is constructed as: in, α 1. α 2. α 3 represent the corresponding weight coefficients respectively; Then the device posture is adjusted by back propagation algorithm. T t Iterate to minimize L pose , get the optimal 6D pose estimation result of the device in the current frame image , as shown below: 。 6. A region of interest monitoring system for laparoscopic surgery, characterized in that: The method for performing the region of interest monitoring method according to claim 1 comprises: A region tracking module is used to track the preoperatively defined region of interest tissue on the intraoperative tissue point cloud based on a deformation estimation algorithm to obtain the three-dimensional coordinates of the region of interest tissue in the current frame camera coordinates; A pose estimation module is used to estimate the optimal 6D pose of the instrument in the camera coordinate system of the current frame in real time by using the instrument 6D pose tracking algorithm based on 3D Gaussian sputtering; simplify the pre-reconstructed instrument Gaussian model into several cylindrical sides, update the simplified instrument model by using the optimal 6D pose of the instrument, and obtain the simplified instrument model in the camera coordinate system of the current frame; The distance calculation module is used to obtain the minimum relative distance between the tissue point cloud of the region of interest and the updated simplified instrument model according to the three-dimensional coordinates of the region of interest in the current frame camera coordinates.

7. The region of interest monitoring system according to claim 6, characterized in that: After the distance calculation module obtains the minimum relative distance, the region of interest monitoring system further includes: A visualization and early warning module is used to project the tissue point cloud of the region of interest onto a two-dimensional image plane using a camera model to determine the coordinates of corresponding pixel points; According to the relationship between the minimum relative distance and the preset danger distance warning threshold, the pixel value of the B channel of the corresponding pixel point is updated to construct a warning map corresponding to the original projection image; and / or Used to update the Z coordinate of the tissue point cloud of the region of interest with the minimum relative distance, construct a new point cloud corresponding to the original pixel points one by one, and use it as a three-dimensional visualization interface; If the minimum relative distance is less than a preset danger distance warning threshold, a scale surface is displayed at every preset scale on the three-dimensional visualization interface.

8. A storage medium, characterized in that: The computer program for monitoring a region of interest in laparoscopic surgery is stored therein, wherein the computer program enables a computer to execute the region of interest monitoring method as claimed in any one of claims 1 to 5.

9. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a program for executing the region of interest monitoring method according to any one of claims 1 to 5.

Images