A surgical operation force feedback guidance method based on virtual marker tracking and instrument posture

Abstract

The present invention provides a surgical operation force feedback guidance method, system, storage medium and electronic device based on virtual marker tracking and instrument posture, which relates to the field of medical devices. In the present invention, real-time three-dimensional point cloud reconstruction is performed on the in vivo scene, and a virtual marker dynamic tracking method based on key frame guidance is used to track the surgical path marked by the doctor on the first image frame, and the three-dimensional surgical path is obtained based on the real-time three-dimensional point cloud. Combined with the surgical instrument 6D posture method based on 3D Gaussian tracking, a guiding virtual fixture along the dynamic three-dimensional surgical path is finally established to effectively help the doctor complete the operation along the pre-set surgical path through the force feedback mechanism, and realize human-machine co-operation that meets the doctor's control intention.

Description

Technical Field

[0001] The present invention relates to the field of medical devices, and in particular to a surgical operation force feedback guidance method, system, storage medium and electronic device based on virtual marker tracking and device posture. Background Art

[0002] As a minimally invasive surgical technique, robotic surgery has been widely recognized by surgeons due to its high precision. Among them, force tactile feedback is a major challenge currently faced by surgical robots.

[0003] In the related technology, patent CN202211328226.5 discloses a force feedback method and device for a main operator of a surgical robot, including: obtaining first force feedback data and scene data of the main operator of the surgical robot; based on the scene data, generating a virtual protection domain corresponding to the main operator of the surgical robot; based on the first force feedback data and the virtual protection domain, generating target force feedback data using an impedance control algorithm; sending a first instruction to a force feedback device controller, the first instruction is used to instruct the force feedback device controller to control the main operator of the surgical robot based on the target force feedback data.

[0004] However, the original plan for establishing the tension field in the virtual protection domain requires planning the marking path in advance. Due to the large changes in the body environment during surgery, the marking path is difficult to be accurately and real-time confirmed. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a surgical operation force feedback guidance method, system, storage medium and electronic device based on virtual marker tracking and instrument posture, which solves the technical problem that the establishment of a tension field in a virtual protection domain requires planning the marking path in advance, and the marking path is difficult to be accurately and real-time confirmed due to large changes in the in vivo environment during surgery.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A surgical operation force feedback guidance method based on virtual marker tracking and instrument posture, comprising:

[0010] Acquire the three-dimensional point cloud of the internal scene based on depth estimation;

[0011] The 6D pose estimation method of the instrument based on 3D Gaussian sputtering obtains the optimal 6D pose of the instrument in the current frame image to estimate the end position of the surgical instrument;

[0012] Based on the key frame-guided virtual marker dynamic tracking method, the surgical path marked by the doctor on the first image frame is tracked to obtain the tracking path of the current frame image; wherein the tracking path is represented by a two-dimensional target point set;

[0013] Based on the three-dimensional point cloud and the two-dimensional target point set, obtaining a three-dimensional surgical path;

[0014] The point cloud position closest to the end position of the surgical instrument on the three-dimensional surgical path is searched, an additional investigation radius is introduced, an artificial vector of the end position of the surgical instrument is obtained, and based on the artificial vector, a guiding virtual force field along the three-dimensional surgical path is established to guide the doctor to control the surgical robot system through a force feedback mechanism.

[0015] Preferably, the instrument 6D pose estimation method based on 3D Gaussian sputtering obtains the optimal 6D pose of the instrument in the current frame image to estimate the end position of the surgical instrument; comprising:

[0016] Reading the current frame image, obtaining the device mask through a semantic segmentation algorithm; and performing depth estimation on the current frame image using a binocular depth estimation algorithm, and respectively obtaining a tissue depth map and a device depth map in combination with the device mask;

[0017] Initialize the device pose, use the pre-reconstructed device Gaussian model to perform 3D Gaussian sputtering, and obtain the view, projection mask and depth map under the corresponding pose;

[0018] According to the rendering result, the current frame image, the device mask, and the device depth map, a total loss function is constructed; and the device posture is updated and learned through the back propagation algorithm to minimize the total loss function and obtain the optimal 6D posture of the device in the current frame image;

[0019] Based on the optimal 6D posture of the instrument in the current frame image, the position of the Gaussian kernel corresponding to the end of the instrument is determined and used as the end position of the surgical instrument.

[0020] Preferably, the initializing the device posture comprises:

[0021] For the first frame of image, a rendering view of the instrument Gaussian model on the image plane is obtained in combination with the camera model, the laparoscope is moved so that the image coincides with the area where the rendering view is located, and the instrument posture at this time is used as the initial instrument posture of the first frame of image;

[0022] For images other than the first frame, the device pose of the current frame is initialized based on the optimal device pose of the previous frame and the optical flow information of two consecutive frames.

[0023] Preferably, the reconstruction process of the Gaussian model of the device includes:

[0024] Acquire several images with known camera poses and containing all views of the device to reconstruct a sparse point cloud of the device;

[0025] Initialize each point of the sparse point cloud as a Gaussian to obtain an initialized Gaussian model of the instrument;

[0026] In combination with the aforementioned image sequence consisting of images with known camera poses and containing all viewing angles of the instrument and their corresponding camera poses, the initialized instrument Gaussian model is iteratively optimized to obtain the reconstructed instrument Gaussian model.

[0027] Preferably, the image sequence composed of the images of the aforementioned known camera poses and containing all viewing angles of the instrument and the corresponding camera poses is combined to iteratively optimize the initialized instrument Gaussian model to obtain the reconstructed instrument Gaussian model; comprising:

[0028] The color consistency loss function between the 3D Gaussian rendering view and the original image that only retains the instrument part is constructed as follows:

[0029]

[0030] in, Respectively represent the center position, rotation, and size of any Gaussian; , L is the number of images included in the image sequence; L 1 indicates a norm loss; The camera pose is R l The process of performing 3D Gaussian differentiable rendering of images; G i Indicates i Gaussian points, , I is the number of Gaussians; It indicates the original image from the image sequence where only the instrument part is retained;

[0031] A multi-region alignment loss function is introduced to constrain the projection shape of each articulated part on the reconstructed device Gaussian model on the image plane and the device mask on the original image. M lc Align them separately; the multi-region alignment loss function is as follows:

[0032]

[0033] in, , C Indicates the number of articulated parts of the device; The camera pose is R lThe process of performing 3D Gaussian differentiable rendering projection;

[0034] The total loss function of the instrument Gaussian model reconstruction process is:

[0035]

[0036] in, α represents the corresponding weight coefficient;

[0037] By minimizing the total loss function L recons , obtain the reconstructed instrument Gaussian model , as shown below:

[0038]

[0039] in, Indicates i reconstructed Gaussian points; Indicates the value of the variable when the function value is minimum.

[0040] Preferably, the method constructs a total loss function based on the rendering result, the current frame image, the device mask, and the device depth map; and updates and learns the device pose through a back propagation algorithm to minimize the total loss function and obtain the optimal 6D pose of the device in the current frame image; including:

[0041] Construct the color consistency loss function between the 3D Gaussian rendering view and the current frame image as follows:

[0042]

[0043] in, t Indicates the current moment; T t Indicates the current frame equipment posture; Represents a rendered view; , Represents the process of performing 3D Gaussian differentiable rendering of an image; M t The instrument mask representing the current image segmentation;

[0044] Construct the regional alignment loss function between the 3D Gaussian differentiable rendering projection image and the device mask of the current frame image as follows:

[0045]

[0046] in, They represent the first c The projection shape of each articulated part on the image plane and the instrument mask of the current frame image; , Represents the process of performing 3D Gaussian differentiable rendering projection; ;

[0047] Construct the depth consistency loss between the rendered depth map and the device depth map of the previous frame as follows:

[0048]

[0049] in, Respectively represent the rendered depth map and the device depth map of the previous frame image; , Represents the process of rendering the depth map of the Gaussian model of the device;

[0050] The total loss function is constructed as:

[0051]

[0052] in, α 1. α 2. α 3 represent the corresponding weight coefficients respectively;

[0053] Then the device posture is adjusted by back propagation algorithm. T t Iterate to minimize L pose , get the optimal 6D pose estimation result of the device in the current frame image , as shown below:

[0054]

[0055] Preferably, the virtual marker dynamic tracking method based on key frame guidance tracks the surgical path marked by the doctor on the first image frame to obtain the tracking path of the current frame image; including:

[0056] Receiving a surgical path marked by a doctor on a first image frame, wherein the surgical path is represented by a two-dimensional target point set at a pixel level;

[0057] Obtain the target area where the two-dimensional target point set on the first image frame is located, and use the optical flow algorithm to predict the trajectory of each point, obtain the two-dimensional target point set and the target area where it is located on the subsequent multiple image frames to initialize the key frame pool;

[0058] For the current frame image, the key frame pool is traversed, similar key frames are queried using a sparse matching method, and the best template is obtained based on the sparse matching results between the queried similar key frames and the current frame image;

[0059] When the best template is not an empty set, obtaining a tracking area on the current frame image based on the target area on each similar key frame in the best template;

[0060] Based on the two-dimensional target point set in each of the target areas, the motion trajectory of each point is predicted using an optical flow algorithm to obtain a two-dimensional target point set on the corresponding tracking area; the two-dimensional target point set on each of the tracking areas is merged to obtain a final two-dimensional target point set, and the final two-dimensional target point set is used as the tracking path on the current frame image;

[0061] When the current frame image is determined to be a new key frame based on a preset condition, it is placed in the key frame pool; wherein the preset condition includes:

[0062] (1) Compared with the surgical path, a complete tracking path was obtained;

[0063] (2) obtaining a tracking area where the final two-dimensional target point set is located, wherein the center of the tracking area is always within a preset range of the current frame image;

[0064] (3) if the number of similar key frames in the best template is greater than one, the number of image frames that have passed since the last new key frame was put into the key frame pool exceeds a first preset number;

[0065] If the similar keyframe in the best template is equal to one, more than a second preset number of image frames have passed since the last new keyframe was put into the keyframe pool, and the ratio of the length of the tracking path to the surgical path is consistent with the scale change of the respective image frames;

[0066] (4) The maximum capacity of the keyframe pool is not exceeded.

[0067] Preferably, the initialization of the key frame pool specifically includes:

[0068] use represents a key frame, where j is the key frame index in the key frame pool; Represents RGB value, W.H. Represents width and height respectively; Indicates the target area; , They represent the extracted Feature points and feature descriptions in; represents a two-dimensional target point set;

[0069] The 0th image frame F 0 specifies a key frame K 0∈ K , the two-dimensional target point set on it is defined as , Indicates s The coordinates of the pixels, n Indicates the number of pixels. represents the set of real numbers;

[0070] by The central area Generated by:

[0071]

[0072] Among them, the center point , for The arithmetic mean of the maximum and minimum values ​​of the horizontal coordinates of all pixels in , for The arithmetic mean of the maximum and minimum values ​​of the ordinates of all pixels in ; L is the side length of the target area, L Not less than The difference between the maximum and minimum values ​​of the horizontal and vertical coordinates of all pixels in;

[0073] Define the target area Store four vertices and the center point , expressed as ,in Depend on It can be seen that The four vertices of: upper left, upper right, lower left, lower right;

[0074] The optical flow algorithm is used to predict the next two image frames. F 1 and F 2 Movement V 0→1 ∈ , V 0→2 ∈ , and calculate each pixel point on the surgical path according to the optical flow result exist F 1 and F Trajectory on 2:

[0075]

[0076] in, Interp (∙) represents the sampling process, and the corresponding target point set is obtained and , and refer to formula (1) to obtain the target area and ;

[0077] Will F 1.F 2 corresponds to the key frame K 1 、K 2, with K 0 together form the initial key frame pool P ={ K 0, K 1, K 2}.

[0078] Preferably, for the current frame image, traversing the key frame pool, using a sparse matching method to query similar key frames, and obtaining the best template based on the sparse matching results between the queried similar key frames and the current frame image; specifically includes:

[0079] Get the current frame image F i Feature points and feature descriptions;

[0080] Traverse each key frame in the key frame pool K j , using the sparse matching method to obtain F i and K j The matching points between

[0081] statistics K j The number of matching points in the upper target area n ji , and select n ji The three largest keyframes are used as similar keyframes K simi ={ K a ,K b ,K c}⊆ P ,

[0082] The best template is obtained using the following formula:

[0083]

[0084] in, K j Indicates the first keyframe in the keyframe pool. j keyframes, j The value of a, b or c;n ji express K j Target area on With the current frame image Fi The number of matching points between ; σ is the first threshold; K simi Represents all similar keyframes queried;

[0085] and / or

[0086] The step of obtaining a tracking area on the current frame image based on the target area on each similar key frame in the best template specifically includes:

[0087] based on K j ⊆ K best and F i The matching points between them are used to estimate the homography matrix using RANSAC H ji , H ji The accuracy of get, n in is the number of interior points;

[0088] when ≥λ , 0< λ<1 , λ is the second threshold, then for the target area The four vertices on Perform homography transformation and get exist F i The corresponding position on :

[0089]

[0090] Otherwise, the vertex coordinates are transformed using the neighborhood similarity transformation method:

[0091] Extract from sparse matching results F i and K j A set of matches , x im ∈ and y jm ∈ is a two-dimensional column vector representing the position of the feature point;

[0092] Search target area vertices of N Neighbors , get the corresponding matching point , to get the point exist F i The corresponding position on :

[0093]

[0094] in , w t and and is inversely proportional to the distance between them;

[0095] Based on vertex coordinates , determine an initial region, expressed as:

[0096]

[0097] in, , Respectively exist F i The corresponding position on The horizontal and vertical coordinates of u min 、u max Respectively represent the maximum and minimum values ​​of the horizontal coordinates of the four vertices; v min 、v max Respectively represent the maximum and minimum values ​​of the ordinates of the four vertices;

[0098] Based on the coordinates of the four vertices of the initial area, determine the tracking area The center point ;

[0099]

[0100] in, 、 Respectively represent the tracking area The center point The horizontal and vertical coordinates of

[0101] Refer to formula (1) to obtain the tracking area on the current frame image .

[0102] Preferably, based on the two-dimensional target point set in each of the target areas, the motion trajectory of each point is predicted using an optical flow algorithm to obtain a two-dimensional target point set on the corresponding tracking area; and the two-dimensional target point set on each of the tracking areas is merged to obtain a final two-dimensional target point set; specifically including:

[0103] 2D target point set Each point in p j,s In the whole frame K j The coordinates on the target area Coordinates The mapping relationship is as follows:

[0104]

[0105] Optical flow in the target area and tracking area The optical flow prediction result is V j→i , the sparse matching result should be consistent with the optical flow prediction result, and the difference between the two is calculated using the following formula:

[0106]

[0107] like d ≤ γ , γ is the third threshold, then the tracking area The target point set on Each point in p ji,s Obtained through the following methods:

[0108]

[0109] otherwise, Not participating Calculation of

[0110] Merge the two-dimensional target point set on each tracking area , get the final two-dimensional target point set , expressed as:

[0111]

[0112] in, λ ji is the weight, and =1, λ ji and n ji Directly proportional.

[0113] Preferably, the step of searching for the point cloud position closest to the end position of the surgical instrument on the three-dimensional surgical path, introducing an additional investigation radius, and obtaining an artificial vector of the end position of the surgical instrument comprises:

[0114] Using the K nearest neighbor algorithm, search for the distance from the end of the surgical instrument on the three-dimensional surgical path Nearest point cloud location , get the relative distance between two points x d ;

[0115] Assume that all points on the three-dimensional surgical path are included in the virtual marker point set V r For the end position of the surgical instrument and the distance along the 3D surgical path Point cloud location closest to the point , seek V r The sub-point set ;in For additional investigation radius;

[0116] The end position of the surgical instrument The corresponding artificial vector is expressed as follows:

[0117]

[0118] is an artificial vector length function, calculated as follows:

[0119]

[0120] in, End position of surgical instrument With virtual markers v The distance between r and R are the upper and lower bounds of the distance respectively. When the distance is less than or equal to r When , it is a virtual marker point v Assign the greatest weight q max , when the distance is greater than r and less than R When the virtual marker v The weight of q max to 0, when the distance is greater than or equal to R When the virtual marker v The weight of is 0; Indicated by Pointing Point v A vector of card (∙) represents the cardinality or number of elements of a set.

[0121] Preferably, the guided virtual force field is expressed as:

[0122]

[0123] in, K vf represents the proportionality coefficient between gravity and the artificial vector, D vf ( x d ) represents the damping coefficient express speed.

[0124] A surgical operation force feedback guidance system based on virtual marker tracking and instrument posture, comprising:

[0125] A depth estimation module, used to obtain a three-dimensional point cloud of an in-vivo scene based on depth estimation;

[0126] The pose estimation module is used for the instrument 6D pose estimation method based on 3D Gaussian sputtering to obtain the optimal 6D pose of the instrument in the current frame image to estimate the end position of the surgical instrument;

[0127] A marker tracking module is used to track the surgical path marked by the doctor on the first image frame based on a virtual marker dynamic tracking method guided by a key frame, and obtain the tracking path of the current frame image; wherein the tracking path is represented by a two-dimensional target point set;

[0128] A path conversion module, used for acquiring a three-dimensional surgical path based on the three-dimensional point cloud and the two-dimensional target point set;

[0129] The force field establishment module is used to search for the point cloud position on the three-dimensional surgical path that is closest to the end position of the surgical instrument, introduce an additional investigation radius, obtain an artificial vector of the end position of the surgical instrument, and establish a guiding virtual force field along the three-dimensional surgical path based on the artificial vector, so as to guide the doctor to control the surgical robot system through a force feedback mechanism.

[0130] A storage medium stores a computer program for force feedback guidance of surgical operations based on virtual marker tracking and instrument posture, wherein the computer program enables a computer to execute the force feedback guidance method for surgical operations as described above.

[0131] An electronic device, comprising:

[0132] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for performing a surgical operation force feedback guidance method as described above.

[0133] (III) Beneficial effects

[0134] The present invention provides a surgical operation force feedback guidance method, system, storage medium and electronic device based on virtual marker tracking and instrument posture. Compared with the prior art, it has the following beneficial effects:

[0135] In the present invention, real-time 3D point cloud reconstruction is performed on the in vivo scene, and the surgical path marked by the doctor on the first image frame is tracked based on a key frame-guided virtual marker dynamic tracking method, and the 3D surgical path is acquired based on the real-time 3D point cloud. Combined with the 6D posture method of surgical instruments based on 3D Gaussian tracking, a guided virtual fixture along the dynamic 3D surgical path is finally established, so as to effectively help the doctor complete the operation along the pre-set surgical path through the force feedback mechanism, thereby realizing human-machine collaborative control that meets the doctor's control intentions. BRIEF DESCRIPTION OF THE DRAWINGS

[0136] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0137] Figure 1 A block diagram of a surgical operation force feedback guidance method based on virtual marker tracking and instrument posture provided by an embodiment of the present invention;

[0138] Figure 2 An overall flow chart of a method for estimating 6D posture of an instrument based on 3D Gaussian sputtering provided in an embodiment of the present invention;

[0139] Figure 3 A schematic diagram of an articulated motion coordinate system provided by an embodiment of the present invention;

[0140] Figure 4 An overall flow chart of a virtual marker dynamic tracking method based on key frame guidance provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0141] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0142] The embodiments of the present application provide a surgical operation force feedback guidance method, system, storage medium and electronic device based on virtual marker tracking and instrument posture, thereby solving the technical problem that the establishment of a tension field in a virtual protection domain requires planning the marking path in advance, and the marking path is difficult to be accurately and real-time confirmed due to large changes in the internal body environment during surgery.

[0143] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0144] In an embodiment of the present invention, real-time three-dimensional point cloud reconstruction is performed on the in vivo scene, and a virtual marker dynamic tracking method guided by key frames is used to dynamically track the surgical path marked by the doctor on the first image frame, and the three-dimensional surgical path is acquired based on the real-time three-dimensional point cloud. Combined with the 6D posture method of surgical instruments based on 3D Gaussian tracking, a guided virtual fixture along the dynamic three-dimensional surgical path is finally established, so as to effectively help the doctor complete the operation along the pre-set surgical path through a force feedback mechanism, thereby achieving human-machine collaborative control that meets the doctor's control intentions.

[0145] In particular, a binocular depth estimation and instrument 6D pose estimation method is proposed. Aiming at the difficulty of 6D pose estimation and tracking of surgical instruments due to their fast movement, low texture, and severe reflection, as well as the problem of decreased accuracy of tissue depth estimation caused by the presence of instruments, the 3D position of the instruments can be accurately estimated, and the complete 3D reconstruction of the surgical scene, 3D position estimation and tracking of surgical instruments can be achieved.

[0146] A keyframe-guided virtual marker dynamic tracking method is proposed to achieve pixel-level precision tracking of two-dimensional virtual markers on laparoscopic images, ensuring the intuitiveness and accuracy of information transmission. The two-stage tracking method adopted reduces erroneous tracking. In addition, by dynamically updating the keyframe pool and enriching the diversity of the keyframe pool, the robustness and accuracy of virtual marker dynamic tracking are further improved.

[0147] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0148] Embodiment 1:

[0149] like Figure 1 As shown, an embodiment of the present invention provides a surgical operation force feedback guidance method based on virtual marker tracking and instrument posture, comprising:

[0150] S1, obtain the three-dimensional point cloud of the internal scene based on depth estimation;

[0151] S2, a 6D pose estimation method of an instrument based on 3D Gaussian sputtering, to obtain the optimal 6D pose of the instrument in the current frame image, so as to estimate the end position of the surgical instrument;

[0152] S3, a virtual marker dynamic tracking method based on key frame guidance, tracking the surgical path marked by the doctor on the first image frame, and obtaining the tracking path of the current frame image; wherein the tracking path is represented by a two-dimensional target point set;

[0153] S4. Acquire a three-dimensional surgical path based on the three-dimensional point cloud and the two-dimensional target point set;

[0154] S5. Search for the point cloud position on the three-dimensional surgical path that is closest to the end position of the surgical instrument, introduce an additional investigation radius, obtain an artificial vector of the end position of the surgical instrument, and based on the artificial vector, establish a guiding virtual force field along the three-dimensional surgical path to guide the doctor to control the surgical robot system through a force feedback mechanism.

[0155] In an embodiment of the present invention, real-time three-dimensional point cloud reconstruction is performed on the in vivo scene, and the surgical path marked by the doctor on the first image frame is tracked based on a key frame-guided virtual marker dynamic tracking method, and the three-dimensional surgical path is acquired based on the real-time three-dimensional point cloud. Combined with the 6D posture method of surgical instruments based on 3D Gaussian tracking, a guided virtual fixture along the dynamic three-dimensional surgical path is finally established, so as to effectively help the doctor complete the operation along the pre-set surgical path through the force feedback mechanism, thereby realizing human-machine collaborative control that meets the doctor's control intentions.

[0156] Next, we will introduce the steps of the above scheme in detail:

[0157] In step S1, a three-dimensional point cloud of the in-vivo scene is acquired based on depth estimation.

[0158] In this step, the three-dimensional point cloud of the in-vivo scene can be obtained based on the image frame sequence showing the in-vivo scene obtained by laparoscopic surgery and the depth estimation algorithm can be used. V d .

[0159] For example, a neural network can be used here for depth estimation to present a stereoscopic three-dimensional field of view to the doctor, thereby creating a surgical experience that meets the doctor's intuition.

[0160] In step S2, the instrument 6D pose estimation method based on 3D Gaussian sputtering is used to obtain the optimal 6D pose of the instrument in the current frame image to estimate the end position of the surgical instrument.

[0161] It can be understood that the estimated results of the instrument posture, the relative position and interaction status of the instrument and tissue can provide high-precision positioning and navigation information in real time, which helps to improve the accuracy and safety of minimally invasive surgery. Based on this, the embodiment of the present invention proposes a method for estimating the instrument-tissue interaction distance based on binocular depth estimation, instrument 6D posture estimation and depth completion, which specifically includes:

[0162] S21, reading the current frame image, obtaining the device mask through the semantic segmentation algorithm; and using the binocular depth estimation algorithm to perform depth estimation on the current frame image, and respectively obtaining the tissue depth map and the device depth map in combination with the device mask;

[0163] S22, initializing the device posture, performing 3D Gaussian sputtering using a pre-reconstructed device Gaussian model, and obtaining a view, a projection mask, and a depth map at a corresponding posture;

[0164] S23, constructing a total loss function according to the rendering result and the current frame image and its instrument mask and instrument depth map; and updating and learning the instrument posture through the back propagation algorithm, minimizing the total loss function, and obtaining the optimal 6D posture of the instrument in the current frame image;

[0165] S24. Based on the optimal 6D posture of the instrument in the current frame image, determine the position of the Gaussian kernel corresponding to the end of the instrument and use it as the end position of the surgical instrument.

[0166] The above scheme estimates the instrument pose by aligning the instrument Gaussian model and image features, which can accurately and robustly restore the 6D pose of the instrument in complex surgical scenarios; and implements instrument pose estimation based on a 3D Gaussian sputtering framework, which can quickly restore the instrument pose in a shorter iteration time and a small number of iterations.

[0167] Figure 2 The overall flow chart of the above scheme is given. The 6D pose estimation and depth estimation method based on the instrument designed in the embodiment of the present invention includes but is not limited to the 6D pose estimation and instrument-tissue relative distance estimation of surgical instruments in laparoscopic surgery or robot-assisted surgery. It mainly includes: depth estimation and instrument segmentation based on binocular laparoscopic images, 3D Gaussian model reconstruction of surgical instruments, 6D pose estimation of surgical instruments based on 3D Gaussian optimization, etc.

[0168] Next, we will combine Figure 2 The specific contents of step S2 are introduced in detail:

[0169] In step S21, the current frame image is read, and the instrument mask is obtained by a semantic segmentation algorithm; and the depth of the current frame image is estimated by a binocular depth estimation algorithm, and a tissue depth map and an instrument depth map are respectively obtained in combination with the instrument mask;

[0170] In this step, the semantic segmentation algorithm (Random Forest, SAM, PerSAM, etc.) is used to perform multi-value segmentation on the image to obtain the device mask on the image. ,in K Indicates the number of articulations of the instrument. And the depth of the endoscope image is estimated using a binocular depth estimation method (MIOL algorithm, CREStereo algorithm, etc.) to obtain a depth map D t , and combined with the device mask M t , and obtain the tissue depth map of the non-device occluded area D t (1- M t ) and the device depth map of the visible part of the device D t M t .in, t Indicates the current moment.

[0171] In step S22, the device posture is initialized, and 3D Gaussian sputtering is performed using the pre-reconstructed device Gaussian model to obtain the view, projection mask and depth map at the corresponding posture.

[0172] Note that 3D Gaussian splatter rendering is mainly used for new view synthesis. The 3D position of the object represented by the 3D Gaussian is determined by the center position of each Gaussian. , Rotate ,size ,color ,density σ Together, it is not suitable for expressing the 6D pose transformation of dynamic objects.

[0173] Accordingly, the embodiment of the present invention pre-establishes a 3D Gaussian model of the surgical instrument (ie, a reconstructed Gaussian model of the instrument), thereby converting the 3D Gaussian optimization into Gaussian posture optimization, ie, posture optimization of the surgical instrument.

[0174] Specifically, the reconstruction process of the Gaussian model of the device includes:

[0175] First, obtain several images with known camera poses and all viewing angles of the instrument to reconstruct the sparse point cloud of the instrument; specifically,

[0176] a) Fix the position of the surgical instrument, aim the camera at the instrument and shoot around it to obtain several instrument images covering all viewing angles. I l , , Lis the number of images. However, due to the textureless and specular reflection characteristics of surgical instruments, it is not easy to solve the relative camera pose through the image features between adjacent frames. Feature points can be provided by attaching marker points on the instrument or fixing a checkerboard in the shooting field of view.

[0177] b) The sparse point cloud of the captured scene is reconstructed through SFM, as well as the 2D-3D corresponding points between the image and the SFM point cloud.

[0178] c) Scale the sfm point cloud according to the real scale of the device, restore the absolute scale of the sparse point cloud, and use the 2D-3D corresponding points to obtain the absolute camera pose of each image R l .

[0179] d) Use the segmentation algorithm to segment the instrument and obtain the segmentation mask of the instrument to remove the background outside the instrument in the image and obtain an image with only the instrument .

[0180] e) According to the camera pose R l Known images , use the SFM algorithm to reconstruct the sparse point cloud of the instrument M sfm , used as initialization for 3D Gaussian model reconstruction.

[0181] Then, each point of the sparse point cloud is initialized as a Gaussian G i , , the properties of each Gaussian are: the center position of the Gaussian , Rotate ,size ,color ,density σ , the probability of belonging to the foreground f .

[0182] It should be noted that the parameters that need to be optimized in the reconstruction process of the surgical instrument Gaussian model in the embodiment of the present invention are only: the center position of each Gaussian , Rotate ,size .

[0183] In addition, the instrument Gaussian model at this stage { G i} is initialized as follows: Equipment point cloud M sfm Each point is initialized as a Gaussian G i , , Iis the number of Gaussians, and the position and color of the point cloud are initialized to the center position of the Gaussians u and color c , Gaussian rotation q Initialized to {1,0,0,0}, the distance from each point to the nearest neighbor is initialized to the size of the Gaussian, and the density of the Gaussian σ Initialized to 1, the probability of belonging to the foreground f Initialized to 1.

[0184] On this basis, the image sequence composed of the images of the aforementioned known camera poses and containing all viewing angles of the instrument and its corresponding camera poses is combined to iteratively optimize the initialized instrument Gaussian model to obtain the reconstructed instrument Gaussian model; including:

[0185] The color consistency loss function between the 3D Gaussian rendering view and the original image that only retains the instrument part is constructed as follows:

[0186]

[0187] in, L 1 indicates a norm loss; It indicates the original image from the image sequence where only the instrument part is retained; The camera pose is R l .

[0188] In addition, to ensure that the reconstructed instrument Gaussian model is an articulated structure, the embodiment of the present invention introduces a multi-region alignment loss function to constrain the projection shape of each articulated part on the reconstructed instrument Gaussian model on the image plane and the instrument mask on the original image. M lc Align them separately; the multi-region alignment loss function is as follows:

[0189]

[0190] in, , C Indicates the number of articulated parts of the device; The camera pose is R l The process of performing 3D Gaussian differentiable rendering projection.

[0191] Then the total loss function of the instrument Gaussian model reconstruction process is:

[0192]

[0193] in, α represents the corresponding weight coefficient;

[0194] By minimizing the total loss function L recons , obtain the reconstructed instrument Gaussian model , as shown below:

[0195]

[0196] in, Indicates i reconstructed Gaussian points; Indicates the value of the variable when the function value is minimum.

[0197] Optimized Gaussian model of the device , denoted as , i Represents the index of the Gaussian kernel.

[0198] Getting the Gaussian model of the instrument Based on the rigid motion between the articulated parts of the surgical instrument T In the embodiment of the present invention, the DH parameter method is used to define the articulated motion of the instrument, and the rod and the left clamp (the right clamp is symmetrical to the left clamp) of the instrument are regarded as a two-link structure, as shown in the schematic diagram. Figure 3 As shown in the figure, local coordinate systems are defined for the instrument rod and the left clamp (the right clamp is symmetrical to the left clamp) respectively. F 1. F 2. Establish the camera coordinate system F cam The device pose estimation solves the transformation matrix from the device's own coordinate system to the camera coordinate system. At this time, the local coordinates of the device are F 1. F 2 points below X , the formula for transforming to the camera coordinate system is as follows:

[0199]

[0200] in, Represents the local coordinate system F 1. F 2 points below, Figure 3 The points of the left (right) clamp are in the local coordinate system F 2. The points on the instrument rod are in the local coordinate system F 1, is the instrument rod coordinate system F 1 to the camera coordinate system F cam The transformation matrix, It is the coordinate system of the left clamp of the instrument F 2 to the instrument rod coordinate system F 1's transformation matrix, the left clamp coordinate system of the instrument F 2 Must be around the rod coordinate systemF 1 z-axis rotation, so one degree of freedom in Euler angles θ The local coordinate system F 1. F 2-room rotation.

[0201] In summary, the rigidity transformation of the articulated device T It can be expressed as ,in , R represents a 3D rotation matrix, Represents a 3D translation vector.

[0202] After clarifying the reconstruction process of the Gaussian model of the instrument and the rigid motion parameters corresponding to the Gaussian model, Figure 2 and Figure 3 As shown, based on the defined Gaussian model, the pose parameters of the instrument are optimized to obtain the 6D pose of the articulated instrument. Given the current frame binocular image, the current frame depth map is obtained by the binocular depth estimation algorithm, and the segmentation mask of the instrument is obtained by the instrument segmentation algorithm. Among them, the input image, depth map and instrument mask are used as the supervision information for subsequent pose learning. Based on the initialization position of the instrument, the view, depth map and projection mask under the corresponding pose are rendered by 3D Gaussian sputtering ( Figure 2 It is expressed as rendered image, segmentation, and depth information, and is compared with the original image, semantic segmentation mask, and depth estimation result ( Figure 2 The loss functions are constructed separately for the image, segmentation, and depth information (expressed in the figure), so that the subsequent steps can construct the total loss function based on this, and then the device posture is updated and learned through the back propagation algorithm to obtain the optimal 6D posture of the device in the current frame image.

[0203] In the stage of device posture optimization, the update of the device Gaussian model follows the following formula:

[0204]

[0205] in i represents the index of the Gaussian kernel, t Indicates the current frame. is the property of the reconstructed instrument Gaussian model, u i,t The update analogy formula (5) is as follows:

[0206] when u i,0 When it is a point belonging to the rod of the instrument, it is updated as follows:

[0207]

[0208] in, Indicates that ui,0 Convert to local coordinate system F 1 is the transformation matrix; It is determined after the instrument Gaussian model is reconstructed and the corresponding posture parameters are defined.

[0209] when u i,0 When it belongs to the left (right) clamp of the instrument, update as follows:

[0210]

[0211] in, Indicates that u i,0 Convert to local coordinate system F 2 is the transformation matrix; It is determined after the instrument Gaussian model is reconstructed and the corresponding posture parameters are defined.

[0212] The initialization of the instrument posture is mainly divided into two cases.

[0213] First, for the first frame of image, the rendering view of the instrument Gaussian model on the image plane is obtained in combination with the camera model, the laparoscope is moved so that the image and the area where the rendering view is located coincide with each other, and the instrument posture at this time is used as the initial instrument posture of the first frame of image;

[0214] Second, for images other than the first frame, the device pose of the current frame is initialized based on the optimal device pose of the previous frame and the optical flow information of two consecutive frames.

[0215] Current frame image I t Initial instrument posture From the previous frame of equipment pose T t-1 and a rough estimate of the 3D translation get:

[0216]

[0217] in, The calculation of is as follows:

[0218]

[0219] in, Represents the mean value of the predicted optical flow from the previous frame to the current frame in the instrument area, Represents the average depth value of the instrument area in the previous frame, f x , f y Respectively indicated in the camera x ,y The focal length in the direction.

[0220] In step S23, a total loss function is constructed based on the rendering result and the current frame image and its instrument mask and instrument depth map; and the instrument posture is updated and learned through the back propagation algorithm to minimize the total loss function and obtain the optimal 6D posture of the instrument in the current frame image.

[0221] Furthermore, the present invention implements the optimization of setting the device posture mainly by rendering the view and the original image I t Loss of color consistency between L im , Rendered instrument projection mask and instrument segmentation mask M t The region alignment loss between L region , Rendered equipment depth map And the device depth estimation results D t The depth consistency loss between L depth To construct the optimization target.

[0222] In the iteratively updated device pose T t Under the action of , the color consistency loss function between the 3D Gaussian rendering view and the current frame image is constructed as follows:

[0223]

[0224] in, t Indicates the current moment; T t Indicates the current frame equipment posture; Represents a rendered view; , Represents the process of performing 3D Gaussian differentiable rendering of an image; M t The instrument mask representing the current image segmentation;

[0225] Construct the regional alignment loss function between the 3D Gaussian differentiable rendering projection image and the device mask of the current frame image as follows:

[0226]

[0227] in, They represent the first c The projection shape of each articulated part on the image plane and the instrument mask of the current frame image; , Represents the process of performing 3D Gaussian differentiable rendering projection; ;

[0228] Construct the depth consistency loss between the rendered depth map and the device depth map of the previous frame as follows:

[0229]

[0230] in, Respectively represent the rendered depth map and the device depth map of the previous frame image; , Represents the process of rendering the depth map of the Gaussian model of the device;

[0231] Therefore, the total loss function is constructed as:

[0232]

[0233] in, α 1. α 2. α 3 represent the corresponding weight coefficients respectively.

[0234] Then the device posture is adjusted by back propagation algorithm. T t Iterate to minimize L pose , get the optimal 6D pose estimation result of the device in the current frame image , as shown below:

[0235]

[0236] In step S24, based on the optimal 6D posture of the instrument in the current frame image, the position of the Gaussian kernel corresponding to the end of the instrument is determined and used as the end position of the surgical instrument.

[0237] So far, given the binocular laparoscope image, the embodiment of the present invention successfully gives the surgical instrument end position of the articulated instrument .

[0238] In step S3, based on the key frame-guided virtual marker dynamic tracking method, the surgical path marked by the doctor on the first image frame is tracked to obtain the tracking path of the current frame image; wherein the tracking path is represented by a two-dimensional target point set; including:

[0239] S31, receiving the surgical path marked by the doctor on the first image frame, wherein the surgical path is represented by a two-dimensional target point set at the pixel level;

[0240] S32, obtaining the target area where the two-dimensional target point set on the first image frame is located, and using the optical flow algorithm to predict the trajectory of each point, obtaining the two-dimensional target point set and the target area where they are located on the subsequent multiple image frames, so as to initialize the key frame pool;

[0241] S33, for the current frame image, traverse the key frame pool, use a sparse matching method to query similar key frames, and obtain the best template based on the sparse matching results between the queried similar key frames and the current frame image;

[0242] S34, when the best template is not an empty set, obtaining a tracking area on the current frame image based on the target area on each similar key frame in the best template;

[0243] S35, based on the two-dimensional target point set in each of the target areas, using the optical flow algorithm to predict the motion trajectory of each point, and obtaining the two-dimensional target point set on the corresponding tracking area; merging the two-dimensional target point set on each of the tracking areas to obtain the final two-dimensional target point set, and using it as the tracking path on the current frame image;

[0244] S36: When the current frame image is determined to be a new key frame based on a preset condition, the current frame image is placed in the key frame pool; wherein the preset condition includes:

[0245] (1) Compared with the surgical path, a complete tracking path was obtained;

[0246] (2) obtaining a tracking area where the final two-dimensional target point set is located, wherein the center of the tracking area is always within a preset range of the current frame image;

[0247] (3) if the number of similar key frames in the best template is greater than one, the number of image frames that have passed since the last new key frame was put into the key frame pool exceeds a first preset number;

[0248] If the similar keyframe in the best template is equal to one, more than a second preset number of image frames have passed since the last new keyframe was put into the keyframe pool, and the ratio of the length of the tracking path to the surgical path is consistent with the scale change of the respective image frames;

[0249] (4) The maximum capacity of the keyframe pool is not exceeded.

[0250] The above scheme achieves pixel-level precision tracking of two-dimensional virtual markers on laparoscopic images, ensuring intuitiveness and accuracy of information transmission.

[0251] like Figure 4 As shown, Figure 4 The overall flow chart of the virtual marker dynamic tracking method is given. Figure 4The specific content of step S30 is introduced in detail:

[0252] In step S31, a surgical path marked by a doctor on a first image frame is received; wherein the surgical path is represented by a two-dimensional target point set at the pixel level;

[0253] It can be understood that in minimally invasive surgery, the virtual marker tracks the position of the dynamic positioning target and provides real-time feedback to the doctor about the surgical position. The doctor selects the template frame as the 0th image frame F 0∈ , and use tools such as a mouse or stylus to mark virtual markers such as planned surgical trajectories on the display. W.H. Respectively F 0 width and height.

[0254] This step receives the above surgical path and uses a two-dimensional target point set at the pixel level To express, represents a virtual marker on frame 0, p 0,s ∈ Indicates s The coordinates of the pixels, n Indicates the number of pixels, Represents the set of real numbers.

[0255] Through the above settings, it is convenient to follow the movement of the laparoscope to i frame F i ∈ , use the proposed tracking method to obtain the tracking path on this frame , the tracking results Annotate to the current frame image F i It can provide visual guidance for doctors.

[0256] In step S32, the target area where the two-dimensional target point set on the first image frame is located is obtained, and the trajectory of each point is predicted using the optical flow algorithm to obtain the two-dimensional target point set and the target area where it is located on the subsequent multiple image frames to initialize the key frame pool.

[0257] The core principle of the keyframe strategy is that it can identify and record key moments as reference points for error correction. Using keyframes not only reduces computational complexity, but also improves tracking efficiency and ensures the stability of long-term tracking.

[0258] Exemplarily, this step needs to determine the initial three key frames as the template for subsequent tracking. Accordingly, the specific steps for initializing the key frame pool are as follows:

[0259] It should be noted that in the embodiments of the present invention, represents a key frame, where j is the key frame index in the key frame pool; Represents RGB value, W.H. Represents width and height respectively; Indicates the target area; , They represent the extracted Feature points and feature descriptions in; Represents a two-dimensional target point set.

[0260] First, the 0th color image frame F 0 specifies a key frame K 0∈ K , the two-dimensional target point set on it is defined as , Indicates s The coordinates of the pixels, n Indicates the number of pixels. Represents the set of real numbers.

[0261] Secondly, The central area Generated by:

[0262]

[0263] Among them, the center point , for The arithmetic mean of the maximum and minimum values ​​of the horizontal coordinates of all pixels in , for The arithmetic mean of the maximum and minimum values ​​of the ordinates of all pixels in ; L is the side length of the target area, L Not less than The difference between the maximum and minimum values ​​of the horizontal and vertical coordinates of all pixels in .

[0264] Again, using existing descriptor-based sparse matching methods such as SIFT, lightglue, etc. F 0 to extract feature points and descriptors from the entire image and locate them in the target area The internal feature points and descriptors are recorded as .

[0265] Next, define the target area Store four vertices and the center point , expressed as ,in Depend on It can be seen that The four vertices are: upper left, upper right, lower left, and lower right.

[0266] Then, the optical flow algorithm is used to predict the F 1 and F 2 Movement V 0→1 ∈ , V 0→2 ∈ , and calculate each pixel point on the virtual marker according to the optical flow result exist F 1 and F Trajectory on 2:

[0267]

[0268] in, Interp (∙) represents the sampling process, and the corresponding target point set is obtained and , and refer to formula (1) to obtain the target area and .

[0269] Finally, F 1. F 2 corresponds to the key frame K 1 、K 2, with K 0 together form the initial key frame pool P ={ K 0, K 1, K 2}.

[0270] In step S33, for the current frame image, the key frame pool is traversed, a similar key frame query is performed using a sparse matching method, and the best template is obtained based on the sparse matching results between the queried similar key frames and the current frame image; specifically, the following steps are included:

[0271] Get the current frame image F i Feature points and feature descriptions;

[0272] Traverse each key frame in the key frame pool K j , using the sparse matching method to obtain F i and K j The matching points between

[0273] statistics K jThe number of matching points in the upper target area n ji , and select n ji The three largest keyframes are used as similar keyframes K simi ={ K a ,K b ,K c}⊆ P ,

[0274] The best template is obtained using the following formula:

[0275]

[0276] in, K j Indicates the first keyframe in the keyframe pool. j keyframes, j The value of a, b or c ; n ji express K j Target area on With the current frame image F i The number of matching points between ; σ is the first threshold; K simi Represents all similar keyframes found.

[0277] It should be noted that if Figure 3 As shown, based on the number of similar keyframes in the best template card ( K best ), current frame image F i It can be divided into two states:

[0278] (1) If card ( K best )>0, that is, when the best template is not an empty set, it means that there is an area in the current frame that is highly similar to the target area on the key frame, and there is a mark that needs to be tracked. At this time, the current frame is considered to be a normal frame or a candidate key frame, and then it goes to the next step to track the target area and accurately locate the pixel position of the surgical path.

[0279] It can be understood that setting this condition can identify the situation where the surgical path is lost and then reappears.

[0280] (2) If card (K best )=0, that is, when the best template is an empty set, it means that the similarity between the current frame and all key frames is not high, the marker has moved out of the surgical path field of view, and there is no marker that needs to be tracked. The current frame is considered to be a frame that does not need to be tracked, and the next frame image is read directly.

[0281] Then, the embodiment of the present invention adopts a two-stage tracking method. When tracking the surgical path, a two-stage matching from coarse to fine is adopted. First, similar key frames are determined by traversing the key frame pool and sparse matching. The local area is first determined by using the matching results, and then the local area is tracked to reduce error tracking; see steps S34-S35 for details:

[0282] In step S34, when the best template is not an empty set, a tracking area on the current frame image is obtained based on the target area on each similar key frame in the best template;

[0283] This step determines the similar area between the current frame image and the best template frame. It should be noted that tracking is based on the area, not the entire image.

[0284] Accordingly, the tracking area on the current frame image is obtained, including:

[0285] First, based on homography transformation or neighborhood similarity transformation, determine the target area The four vertices on exist F i The corresponding position on ,include:

[0286] based on K j ⊆ K best and F i The matching points between them are used to estimate the homography matrix using RANSAC H ji , H ji The accuracy of get, n in is the number of interior points;

[0287] when ≥λ , 0< λ<1 , λ is the second threshold, then for the target area The four vertices on Perform homography transformation and get exist F i The corresponding position on :

[0288]

[0289] Otherwise, the vertex coordinates are transformed using the neighborhood similarity transformation method:

[0290] Extract from sparse matching results F i and K j A set of matches , x im ∈ and y jm ∈ is a two-dimensional column vector representing the position of the feature point;

[0291] Search target area vertices of N Neighbors , get the corresponding matching point , to get the point exist F i The corresponding position on :

[0292]

[0293] in , w t and and is inversely proportional to the distance between them.

[0294] Exemplarily, N here can be four.

[0295] Next, based on the vertex coordinates , determine an initial region, expressed as:

[0296]

[0297] in, , Respectively exist F i The corresponding position on The horizontal and vertical coordinates of u min 、u max Respectively represent the maximum and minimum values ​​of the horizontal coordinates of the four vertices; v min 、vmax Respectively represent the maximum and minimum values ​​of the vertical coordinates of the four vertices.

[0298] Then, based on the coordinates of the four vertices of the initial area, the tracking area is determined. The center point ;

[0299]

[0300] in, 、 Respectively represent the tracking area The center point The horizontal and vertical coordinates of .

[0301] Finally, refer to formula (1) to obtain the tracking area on the current frame image .

[0302] In step S35, based on the two-dimensional target point set in each of the target areas, the motion trajectory of each point is predicted using the optical flow algorithm to obtain the two-dimensional target point set on the corresponding tracking area; the two-dimensional target point set on each of the tracking areas is merged to obtain the final two-dimensional target point set, and used as the tracking path on the current frame image.

[0303] This step marks and tracks the target area and the tracking area; specifically includes:

[0304] 2D target point set Each point in p j,s In the whole frame K j The coordinates on the target area Coordinates The mapping relationship is as follows:

[0305]

[0306] Optical flow in the target area and tracking area The optical flow prediction result is V j→i , the sparse matching result should be consistent with the optical flow prediction result, and the difference between the two is calculated using the following formula:

[0307]

[0308] like d ≤ γ , γ is the third threshold, then the tracking area The target point set on Each point inp ji,s Obtained through the following methods:

[0309]

[0310] otherwise, Not participating Calculation.

[0311] Merge the target point sets on each tracking area , get the final target point set , expressed as:

[0312]

[0313] in, λ ji is the weight, and =1, λ ji and n ji Directly proportional.

[0314] It should be noted that in fact, the embodiment of the present invention can obtain relatively accurate results by selecting one or two key frames with the highest similarity. Whether each pixel point in the tracking path is on the image, the complete tracking path or the partial tracking path is overlaid on the current frame image to provide real-time guidance for the doctor.

[0315] In step S36, when the current frame image is determined to be a new key frame based on preset conditions, it is placed in the key frame pool; wherein the preset conditions include:

[0316] (1) Compared with the surgical path, a complete tracking path is obtained; it is expressed as ,The key frame must contain the complete surgical path to have sufficient information.

[0317] (2) Obtaining the final two-dimensional target point set Tracking area , the center of the tracking area Always in the current frame image F i within the preset range; expressed as ∈ ,in l is the fourth threshold. In order to eliminate redundancy, and Needs to be satisfied , η 1 is the fifth threshold, and η1<1. This condition hopes that there is a certain difference between the target area and the key frame, and the target area on the key frame is as far away from the edge of the image as possible.

[0318] (3) If the number of similar key frames in the best template is greater than one, the distance from the last new key frame to the key frame pool exceeds a first preset number of image frames.

[0319] Exemplarily, the first preset number here is ten, and the corresponding expression is card ( K best )>1, i - f >10, of which f The index of the most recently added keyframe, 10 frames since the last keyframe was inserted.

[0320] If the similar keyframe in the best template is equal to one, more than a second preset number of image frames have passed since the last new keyframe was put into the keyframe pool, and the ratio of the length of the tracking path to the surgical path is consistent with the scale change of the respective image frames;

[0321] Exemplarily, the second preset number here is four, which is correspondingly expressed as card ( K best )=1, i - f >4, and ,in η 2. η 3 are the sixth threshold and the seventh threshold respectively, and η 2<1< η 3. r The calculation formula is r =max{ h 0 ,h 1}, h 0 、h 1 respectively indicates F i and K j The homography matrix H ji The first two diagonal elements of h 0 、h 1 corresponds to the image scaling factor in the horizontal and vertical directions respectively.

[0322] (4) does not exceed the maximum capacity of the keyframe pool; expressed as card ( P )≤ n max , nmax Keyframe Pool P The maximum capacity, for example, here n max Take 16. In order to prevent unnecessary calculations from being added to the repeated key frames, when a certain capacity is reached, it is considered that the key frames have saved the key information from each perspective.

[0323] Therefore, the embodiment of the present invention can also handle the situation of target loss and re-tracking, and achieve robust tracking.

[0324] So far, the embodiment of the present invention has successfully obtained a tracking path represented by a two-dimensional target point set.

[0325] In step S4, a three-dimensional surgical path is acquired based on the three-dimensional point cloud and the two-dimensional target point set.

[0326] In this step, a 3D point cloud based on depth estimation of laparoscopic images is used. V d , combined with the virtual marker dynamic tracking method to obtain the two-dimensional target point set , we can get a three-dimensional virtual marker point set V r .

[0327]

[0328] Among them, Π(∙) is a function that obtains 3D virtual markers based on 3D point cloud and 2D target point set. V r ∈ V d .

[0329] In step S5, searching for the point cloud position closest to the end position of the surgical instrument on the three-dimensional surgical path, introducing an additional investigation radius, obtaining an artificial vector of the end position of the surgical instrument, and establishing a guiding virtual force field along the three-dimensional surgical path based on the artificial vector to guide the doctor to control the surgical robot system through a force feedback mechanism; comprising:

[0330] Using the K nearest neighbor algorithm, search for the distance from the end of the surgical instrument on the three-dimensional surgical path Nearest point cloud location , get the relative distance between two points x d , thereby determining whether the surgical instrument deviates from the pre-set surgical path.

[0331] Assume that all points on the three-dimensional surgical path are included in the virtual marker point set V r For the end position of the surgical instrument and the distance along the 3D surgical path Point cloud location closest to the point , seek V r The sub-point set ;in For additional investigation radius;

[0332] The end position of the surgical instrument The corresponding artificial vector is expressed as follows:

[0333]

[0334] is an artificial vector length function, calculated as follows:

[0335]

[0336] in, End position of surgical instrument With virtual markers v The distance between r and R are the upper and lower bounds of the distance respectively. When the distance is less than or equal to r When , it is a virtual marker point v Assign the greatest weight q max , when the distance is greater than r and less than R When the virtual marker v The weight of q max to 0, when the distance is greater than or equal to R When the virtual marker v The weight of is 0; Indicated by Pointing Point v A vector of card (∙) represents the cardinality or number of elements of a set.

[0337] Next, based on the artificial vector, a virtual force field including gravity and viscous resistance is established along the three-dimensional surgical path, i.e., a guiding virtual fixture, so as to guide the doctor to manipulate the surgical robot system through a force feedback mechanism; wherein the guiding virtual force field is expressed as:

[0338]

[0339] in, K vf represents the proportionality coefficient between gravity and the artificial vector, D vf ( xd ) represents the damping coefficient express speed.

[0340] At this point, the embodiment of the present invention establishes a human-machine touch interaction mechanism between the doctor and the robot through force control technology, constructs a guiding virtual force field in space, achieves precise guidance of the doctor's movement trajectory, and ensures that the doctor can control the surgical robot according to the predetermined trajectory.

[0341] Embodiment 2:

[0342] An embodiment of the present invention provides a surgical operation force feedback guidance system based on virtual marker tracking and instrument posture, comprising:

[0343] A depth estimation module, used to obtain a three-dimensional point cloud of an in-vivo scene based on depth estimation;

[0344] The pose estimation module is used for the instrument 6D pose estimation method based on 3D Gaussian sputtering to obtain the optimal 6D pose of the instrument in the current frame image to estimate the end position of the surgical instrument;

[0345] A marker tracking module is used to track the surgical path marked by the doctor on the first image frame based on a virtual marker dynamic tracking method guided by a key frame, and obtain the tracking path of the current frame image; wherein the tracking path is represented by a two-dimensional target point set;

[0346] A path conversion module, used for acquiring a three-dimensional surgical path based on the three-dimensional point cloud and the two-dimensional target point set;

[0347] The force field establishment module is used to search for the point cloud position on the three-dimensional surgical path that is closest to the end position of the surgical instrument, introduce an additional investigation radius, obtain an artificial vector of the end position of the surgical instrument, and establish a guiding virtual force field along the three-dimensional surgical path based on the artificial vector, so as to guide the doctor to control the surgical robot system through a force feedback mechanism.

[0348] Embodiment 3:

[0349] An embodiment of the present invention provides a storage medium storing a computer program for force feedback guidance of surgical operations based on virtual marker tracking and instrument posture, wherein the computer program enables a computer to execute the force feedback guidance method for surgical operations as described above.

[0350] Embodiment 4:

[0351] An embodiment of the present invention provides an electronic device, including:

[0352] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for performing a surgical operation force feedback guidance method as described above.

[0353] It can be understood that the surgical operation force feedback guidance system based on virtual marker tracking and instrument posture, storage medium and electronic device provided in the embodiments of the present invention correspond to the surgical operation force feedback guidance method based on virtual marker tracking and instrument posture provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the surgical operation force feedback guidance method, and will not be repeated here.

[0354] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0355] 1. In an embodiment of the present invention, a real-time three-dimensional point cloud reconstruction is performed on the in vivo scene. A virtual marker dynamic tracking method guided by a key frame is used to track the surgical path marked by the doctor on the first image frame, and the three-dimensional surgical path is obtained based on the real-time three-dimensional point cloud. Combined with a 6D posture method of surgical instruments based on 3D Gaussian tracking, a guided virtual fixture along a dynamic three-dimensional surgical path is finally established, so as to effectively help the doctor complete the operation along the pre-set surgical path through a force feedback mechanism, thereby realizing human-machine collaborative control that meets the doctor's control intentions.

[0356] 2. The embodiment of the present invention does not require additional equipment or data, and only uses binocular endoscopic images to achieve complete 3D reconstruction of the surgical scene, and assists in surgical planning and navigation through real-time estimated 3D information of instruments and tissues. Based on the research idea of ​​"reconstruction-optimization", the instrument pose estimation is estimated by aligning the reconstructed instrument model and image features, which can accurately and robustly restore the 6D pose of the instrument in complex surgical scenes. And the instrument pose estimation is realized based on the framework of 3D Gaussian sputtering, which can quickly restore the instrument pose in a few iterations.

[0357] 3. The embodiment of the present invention realizes pixel-level precision tracking of the two-dimensional surgical path on the endoscopic image, ensuring the intuitiveness and accuracy of information transmission. A two-stage tracking method is adopted. When tracking the surgical path, a two-stage matching from coarse to fine is adopted. First, similar key frames are determined by traversing the key frame pool and sparse matching. The matching results are used to first determine the local area, and then the local area is tracked to reduce erroneous tracking. And by dynamically updating the key frame pool, the diversity of the key frame pool is enriched, and the robustness and accuracy of the dynamic tracking of the virtual marker are further improved.

[0358] 4. In the embodiment of the present invention, on the basis of improving the surgical accuracy, the surgeon does not need to wear other equipment to prevent interference with the surgeon's surgical operation.

[0359] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0360] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A surgical operation force feedback guidance method based on virtual marker tracking and instrument posture, characterized in that: include: Acquire the three-dimensional point cloud of the internal scene based on depth estimation; The 6D pose estimation method of the instrument based on 3D Gaussian sputtering obtains the optimal 6D pose of the instrument in the current frame image to estimate the end position of the surgical instrument; Based on the keyframe-guided virtual marker dynamic tracking method, the surgical path marked by the doctor on the first image frame is tracked to obtain the tracking path of the current frame image; The tracking path is represented by a two-dimensional target point set; Based on the three-dimensional point cloud and the two-dimensional target point set, obtaining a three-dimensional surgical path; Searching for the point cloud position closest to the end position of the surgical instrument on the three-dimensional surgical path, introducing an additional investigation radius, obtaining an artificial vector of the end position of the surgical instrument, and establishing a guiding virtual force field along the three-dimensional surgical path based on the artificial vector, so as to guide the doctor to control the surgical robot system through a force feedback mechanism; The 6D pose estimation method of the instrument based on 3D Gaussian sputtering obtains the optimal 6D pose of the instrument in the current frame image to estimate the end position of the surgical instrument; comprising: Reading the current frame image, obtaining the device mask through a semantic segmentation algorithm; and performing depth estimation on the current frame image using a binocular depth estimation algorithm, and respectively obtaining a tissue depth map and a device depth map in combination with the device mask; Initialize the device pose, use the pre-reconstructed device Gaussian model to perform 3D Gaussian sputtering, and obtain the view, projection mask and depth map under the corresponding pose; According to the rendering result, the current frame image, the device mask, and the device depth map, a total loss function is constructed; and the device posture is updated and learned through the back propagation algorithm to minimize the total loss function and obtain the optimal 6D posture of the device in the current frame image; Based on the optimal 6D posture of the instrument in the current frame image, determine the position of the Gaussian kernel corresponding to the end of the instrument and use it as the end position of the surgical instrument; The initialization device posture comprises: For the first frame of image, a rendering view of the instrument Gaussian model on the image plane is obtained in combination with the camera model, the laparoscope is moved so that the image coincides with the area where the rendering view is located, and the instrument posture at this time is used as the initial instrument posture of the first frame of image; For images other than the first frame, the device pose of the current frame is initialized based on the optimal device pose of the previous frame and the optical flow information of two consecutive frames; The reconstruction process of the Gaussian model of the device includes: Acquire several images with known camera poses and containing all views of the device to reconstruct a sparse point cloud of the device; Initialize each point of the sparse point cloud as a Gaussian to obtain an initialized Gaussian model of the instrument; Combining the aforementioned image sequence consisting of the images with known camera poses and containing all viewing angles of the instrument and their corresponding camera poses, iteratively optimizing the initialized instrument Gaussian model to obtain the reconstructed instrument Gaussian model; The image sequence composed of the images of the aforementioned known camera poses and containing all viewing angles of the instrument and the corresponding camera poses is combined to iteratively optimize the initialized instrument Gaussian model to obtain the reconstructed instrument Gaussian model; comprising: The color consistency loss function between the 3D Gaussian rendering view and the original image that only retains the instrument part is constructed as follows: in, Respectively represent the center position, rotation, and size of any Gaussian; , L is the number of images included in the image sequence; L 1 indicates a norm loss; The camera pose is R l The process of performing 3D Gaussian differentiable rendering of images; G i Indicates i Gaussian points, , I is the number of Gaussians; It indicates the original image from the image sequence where only the instrument part is retained; A multi-region alignment loss function is introduced to constrain the projection shape of each articulated part on the reconstructed device Gaussian model on the image plane and the device mask on the original image. M lc Align them separately; the multi-region alignment loss function is as follows: in, , C Indicates the number of articulated parts of the device; The camera pose is R l The process of performing 3D Gaussian differentiable rendering projection; The total loss function of the instrument Gaussian model reconstruction process is: in, α represents the corresponding weight coefficient; By minimizing the total loss function L recons , obtain the reconstructed instrument Gaussian model , as shown below: in, Indicates i reconstructed Gaussian points; Indicates the value of the variable when the function value is minimum; The method constructs a total loss function based on the rendering result, the current frame image, its device mask, and the device depth map; and updates and learns the device posture through a back propagation algorithm to minimize the total loss function and obtain the optimal 6D posture of the device in the current frame image; including: Construct the color consistency loss function between the 3D Gaussian rendering view and the current frame image as follows: in, t Indicates the current moment; T t Indicates the current frame equipment posture; Represents a rendered view; , Represents the process of performing 3D Gaussian differentiable rendering of an image; M t The instrument mask representing the current image segmentation; Construct the regional alignment loss function between the 3D Gaussian differentiable rendering projection image and the device mask of the current frame image as follows: in, They represent the first c The projection shape of each articulated part on the image plane and the instrument mask of the current frame image; , Represents the process of performing 3D Gaussian differentiable rendering projection; ; Construct the depth consistency loss between the rendered depth map and the device depth map of the previous frame as follows: in, Respectively represent the rendered depth map and the device depth map of the previous frame image; , Represents the process of rendering the depth map of the Gaussian model of the device; The total loss function is constructed as: in, α 1. α 2. α 3 represent the corresponding weight coefficients respectively; Then the device posture is adjusted by back propagation algorithm. T t Iterate to minimize L pose , get the optimal 6D pose estimation result of the device in the current frame image , as shown below:

2. The surgical operation force feedback guidance method according to claim 1, characterized in that: The virtual marker dynamic tracking method based on key frame guidance tracks the surgical path marked by the doctor on the first image frame to obtain the tracking path of the current frame image; including: Receiving a surgical path marked by a doctor on a first image frame, wherein the surgical path is represented by a two-dimensional target point set at a pixel level; Obtain the target area where the two-dimensional target point set on the first image frame is located, and use the optical flow algorithm to predict the trajectory of each point, obtain the two-dimensional target point set and the target area where it is located on the subsequent multiple image frames to initialize the key frame pool; For the current frame image, the key frame pool is traversed, similar key frames are queried using a sparse matching method, and the best template is obtained based on the sparse matching results between the queried similar key frames and the current frame image; When the best template is not an empty set, obtaining a tracking area on the current frame image based on the target area on each similar key frame in the best template; Based on the two-dimensional target point set in each of the target areas, the motion trajectory of each point is predicted using an optical flow algorithm to obtain a two-dimensional target point set on the corresponding tracking area; the two-dimensional target point set on each of the tracking areas is merged to obtain a final two-dimensional target point set, and the final two-dimensional target point set is used as the tracking path on the current frame image; When the current frame image is determined to be a new key frame based on a preset condition, it is placed in the key frame pool; wherein the preset condition includes: (1) Compared with the surgical path, a complete tracking path was obtained; (2) obtaining a tracking area where the final two-dimensional target point set is located, wherein the center of the tracking area is always within a preset range of the current frame image; (3) if the number of similar key frames in the best template is greater than one, the number of image frames that have passed since the last new key frame was put into the key frame pool exceeds a first preset number; If the similar keyframe in the best template is equal to one, more than a second preset number of image frames have passed since the last new keyframe was put into the keyframe pool, and the ratio of the length of the tracking path to the surgical path is consistent with the scale change of the respective image frames; (4) The maximum capacity of the keyframe pool is not exceeded.

3. The surgical operation force feedback guidance method according to claim 2, characterized in that: The initialization key frame pool specifically includes: use represents a key frame, where j is the key frame index in the key frame pool; Represents RGB value, W.H. Represents width and height respectively; Indicates the target area; , They represent the extracted Feature points and feature descriptions in; represents a two-dimensional target point set; The 0th image frame F 0 specifies a key frame K 0∈ K , the two-dimensional target point set on it is defined as , Indicates s The coordinates of the pixels, n Indicates the number of pixels. represents the set of real numbers; by The central area Generated by: Among them, the center point , for The arithmetic mean of the maximum and minimum values ​​of the horizontal coordinates of all pixels in , for The arithmetic mean of the maximum and minimum values ​​of the ordinates of all pixels in ; L is the side length of the target area, L Not less than The difference between the maximum and minimum values ​​of the horizontal and vertical coordinates of all pixels in; Define the target area Store four vertices and the center point , expressed as ,in Depend on It can be seen that The four vertices of: upper left, upper right, lower left, lower right; The optical flow algorithm is used to predict the next two image frames. F 1 and F 2 Movement V 0→1 ∈ , V 0→2 ∈ , and calculate each pixel point on the surgical path according to the optical flow result exist F 1 and F Trajectory on 2: in, Interp (∙) represents the sampling process, and the corresponding target point set is obtained and , and refer to formula (1) to obtain the target area and ; Will F 1. F 2 corresponds to the key frame K 1 、K 2, with K 0 together form the initial key frame pool P ={ K 0, K 1, K 2}; and / or The method of traversing the key frame pool for the current frame image, performing a similar key frame query using a sparse matching method, and obtaining an optimal template based on a sparse matching result between the queried similar key frame and the current frame image, specifically includes: Get the current frame image F i Feature points and feature descriptions; Traverse each key frame in the key frame pool K j , using the sparse matching method to obtain F i and K j The matching points between statistics K j The number of matching points in the upper target area n ji , and select n ji The three largest keyframes are used as similar keyframes K simi ={ K a ,K b ,K c }⊆ P , The best template is obtained using the following formula: in, K j Indicates the first keyframe in the keyframe pool. j keyframes, j The value of a, b or c;n ji express K j Target area on With the current frame image F i The number of matching points between ; σ is the first threshold; K simi Represents all similar keyframes queried; and / or The step of obtaining a tracking area on the current frame image based on the target area on each similar key frame in the best template specifically includes: based on K j ⊆ K best and F i The matching points between them are used to estimate the homography matrix using RANSAC H ji , H ji The accuracy of get, n in is the number of interior points; when ≥λ , 0< λ<1 , λ is the second threshold, then for the target area The four vertices on Perform homography transformation and get exist F i The corresponding position on : Otherwise, the vertex coordinates are transformed using the neighborhood similarity transformation method: Extract from sparse matching results F i and K j A set of matches , x im ∈ and y jm ∈ is a two-dimensional column vector representing the position of the feature point; Search target area vertices of N Neighbors , get the corresponding matching point , to get the point exist F i The corresponding position on : in , w t and and is inversely proportional to the distance between them; Based on vertex coordinates , determine an initial region, expressed as: in, , Respectively exist F i The corresponding position on The horizontal and vertical coordinates of u min 、u max Respectively represent the maximum and minimum values ​​of the horizontal coordinates of the four vertices; v min 、v max Respectively represent the maximum and minimum values ​​of the ordinates of the four vertices; Based on the coordinates of the four vertices of the initial area, determine the tracking area The center point ; in, 、 Respectively represent the tracking area The center point The horizontal and vertical coordinates of Refer to formula (1) to obtain the tracking area on the current frame image ; and / or The method of predicting the motion trajectory of each point based on the two-dimensional target point set in each target area by using an optical flow algorithm to obtain a two-dimensional target point set on the corresponding tracking area; merging the two-dimensional target point set on each tracking area to obtain a final two-dimensional target point set specifically includes: 2D target point set Each point in p j,s In the whole frame K j The coordinates on the target area Coordinates The mapping relationship is as follows: Optical flow in the target area and tracking area The optical flow prediction result is V j→i , the sparse matching result should be consistent with the optical flow prediction result, and the difference between the two is calculated using the following formula: like d ≤ γ , γ is the third threshold, then the tracking area The target point set on Each point in p ji,s Obtained through the following methods: otherwise, Not participating Calculation of Merge the two-dimensional target point set on each tracking area , get the final two-dimensional target point set , expressed as: in, λ ji is the weight, and =1, λ ji and n ji Directly proportional.

4. The surgical operation force feedback guidance method according to claim 1, characterized in that: The step of searching for the point cloud position closest to the end position of the surgical instrument on the three-dimensional surgical path, introducing an additional investigation radius, and obtaining an artificial vector of the end position of the surgical instrument comprises: Using the K nearest neighbor algorithm, search for the distance from the end of the surgical instrument on the three-dimensional surgical path Nearest point cloud location , get the relative distance between two points x d ; Assume that all points on the three-dimensional surgical path are included in the virtual marker point set V r For the end position of the surgical instrument and the distance along the three-dimensional surgical path Point cloud location closest to the point , seek V r The sub-point set ;in For additional investigation radius; The end position of the surgical instrument The corresponding artificial vector is expressed as follows: is an artificial vector length function, calculated as follows: in, End position of surgical instrument With virtual markers v The distance between r and R are the upper and lower bounds of the distance respectively. When the distance is less than or equal to r When , it is a virtual marker point v Assign the greatest weight q max , when the distance is greater than r and less than R When the virtual marker v The weight of q max to 0, when the distance is greater than or equal to R When the virtual marker v The weight of is 0; Indicated by Pointing Point v A vector of card (∙) represents the cardinality or number of elements of a set.

5. The surgical operation force feedback guidance method according to claim 4, characterized in that: The guided virtual force field is expressed as: in, K vf represents the proportionality coefficient between gravity and the artificial vector, D vf ( x d ) represents the damping coefficient express speed.

6. A surgical operation force feedback guidance system based on virtual marker tracking and instrument posture, characterized in that: The method for performing the surgical operation force feedback guidance method according to claim 1 comprises: A depth estimation module, used to obtain a three-dimensional point cloud of an in-vivo scene based on depth estimation; The pose estimation module is used for the instrument 6D pose estimation method based on 3D Gaussian sputtering to obtain the optimal 6D pose of the instrument in the current frame image to estimate the end position of the surgical instrument; A marker tracking module is used to track the surgical path marked by the doctor on the first image frame based on a virtual marker dynamic tracking method guided by a key frame, and obtain the tracking path of the current frame image; wherein the tracking path is represented by a two-dimensional target point set; A path conversion module, used for acquiring a three-dimensional surgical path based on the three-dimensional point cloud and the two-dimensional target point set; The force field establishment module is used to search for the point cloud position on the three-dimensional surgical path that is closest to the end position of the surgical instrument, introduce an additional investigation radius, obtain an artificial vector of the end position of the surgical instrument, and establish a guiding virtual force field along the three-dimensional surgical path based on the artificial vector, so as to guide the doctor to control the surgical robot system through a force feedback mechanism.

7. A storage medium, characterized in that: It stores a computer program for force feedback guidance of surgical operations based on virtual marker tracking and instrument posture, wherein the computer program enables the computer to execute the surgical operation force feedback guidance method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the surgical operation force feedback guidance method as described in any one of claims 1 to 5.

Images