Guided three-dimensional virtual jig generation method for dynamic surgical path guidance

Abstract

The application provides a guide type three-dimensional virtual clamp generation method and system facing a dynamic operation path guide, a storage medium and an electronic device, and relates to the technical field of virtual clamps.In the application, a local area is determined through the position of a key point, the local area is tracked to reduce the error tracking of three-dimensional key points, the texture information and shape information of an endoscope image are combined to avoid the influence caused by the unobvious features of the environment in the body to a certain extent, the three-dimensional key points are accurately positioned on the dimension-reduced point cloud by constructing an optimization function, and the inconsistency of the curve shapes caused by different perspectives is avoided.Further, a virtual force field containing gravity and viscous resistance is established along the three-dimensional curve obtained through tracking, that is, a guide type virtual clamp, and the guide type virtual clamp accurately guides the movement track of a surgical instrument through a force feedback mechanism.

Description

Technical Field

[0001] This invention relates to the field of virtual fixture technology, and more specifically to a guided three-dimensional virtual fixture generation method, system, storage medium, and electronic device for dynamic surgical path guidance. Background Technology

[0002] The emergence of autonomous surgical robots for remote surgery has solved the problem of uneven distribution of medical resources. By tracking three-dimensional key points and then the surgical path, the quality of remote surgical guidance and the autonomous operation capability of surgical robots can be improved.

[0003] Virtual jigs are often used in intraoperative surgical scenarios in the form of force interaction to solve the problem of controlling surgical instruments with non-periodic motion, such as organ protection, target guidance, and obstacle avoidance constraints during surgery, thereby ensuring the safety and precision of the surgical process.

[0004] Because current technologies cannot update the pre-operatively planned surgical path in real time during surgery, surgeons often struggle to locate the corresponding position of the pre-operatively planned path during the operation. They also cannot visually assess deviations in the surgical path during surgery, increasing the difficulty for surgeons in controlling the robot. This directly impacts virtual jigs, especially guided virtual jigs, which are unable to provide precise guidance for the three-dimensional surgical path. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a method, system, storage medium, and electronic device for generating guided three-dimensional virtual fixtures for dynamic surgical path guidance, solving the technical problem of the inability to achieve precise guidance of surgical three-dimensional paths.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for generating guided 3D virtual fixtures for dynamic surgical path guidance includes:

[0010] S1. Read the endoscopic image, obtain the surgical path curve on the current frame image according to the doctor's selection, and obtain several first two-dimensional key points through which the surgical path curve passes.

[0011] S2. Track the first local region containing the first two-dimensional key point on the current frame image, and obtain the second local region on the next frame image;

[0012] S3. Based on the mapping relationship between the endoscopic image and the point cloud, the first local region is mapped onto the first local point cloud and the second local region is mapped onto the second local point cloud respectively; and the first two-dimensional key point is determined as the first three-dimensional key point on the first local point cloud, and the second three-dimensional key point on the second local point cloud is obtained through coordinate transformation.

[0013] S4. Reduce the dimensionality of the first local point cloud to obtain the first two-dimensional point cloud, and obtain the second two-dimensional key point of the first three-dimensional key point on the first two-dimensional point cloud;

[0014] The second local point cloud is reduced in dimensionality to obtain the second two-dimensional point cloud, and the second three-dimensional key points are obtained as the third two-dimensional key points on the second two-dimensional point cloud.

[0015] Based on the second and third two-dimensional key points, the two-dimensional coordinates of the tracked key points on the two-dimensional point cloud are obtained by minimizing the preset optimization function.

[0016] S5. Based on the mapping relationship between the point clouds before and after dimensionality reduction, obtain the three-dimensional coordinates of each tracked key point, and perform curve fitting to finally obtain a three-dimensional curve.

[0017] S6. Along the three-dimensional curve obtained by tracking, a guided virtual fixture is established. The guided virtual fixture guides the movement trajectory of the surgical instruments through a force feedback mechanism.

[0018] Preferably, S6 specifically includes:

[0019] S61. Convert the real-time three-dimensional position information of the surgical instrument and the point cloud obtained in the previous steps to the same coordinate system, and obtain the distance p from the end of the surgical instrument on the three-dimensional curve. ins The nearest point p nearest and the relative distance x between the two points. d ;

[0020] S62, Establish information about p ins Artificial vector field:

[0021] Define all points on a 3D curve as contained in the point set V. r In the middle, find V r The subset V p ={v|||p ins -v||≤||p ins -p nearest ||+Δr}, where Δr is the additional radius of investigation;

[0022]

[0023] Where g is p ins Artificial vector field; g(||pins -v||) is the artificial vector length function, n(p ins v) is the value from point p ins The vector pointing to point v, when v corresponds to different organizations, has different artificial vector length functions g(||p). ins -v||), satisfying the condition card(V p ) is a subset V p The number of elements in the middle;

[0024] S63. Establish a virtual force field that includes gravity and viscous drag:

[0025]

[0026] Among them, F vf For virtual force field; K vf D is the proportionality coefficient between the gravitational force and the artificial vector at that location. vf (x d ) is the damping coefficient, which varies with the shortest distance x d Move; For p ins The speed.

[0027] Preferably, S2 includes

[0028] S21. First, define the (k-1)th frame image as... Where W represents the width of the endoscopic image, and H represents the height of the endoscopic image; the k-th frame image is I(k); several first two-dimensional key points

[0029] Determine the maximum and minimum values ​​of all first-dimensional keypoints on the u-axis and v-axis of the image coordinate system, and select the position of the median value on each axis as p. c (k-1)=(u1,v1)∈R 2 The first local region is determined based on the preset region shape and side length.

[0030] S22. Use optical flow to perform feature matching on image I(k-1) and I(k), and obtain the feature points on image I(k) that are related to p. c The center p of the second local region corresponding to (k-1) c (k),

[0031]

[0032] in, Let m represent the feature points on image I(k), and m represent the number of feature points on image I(k).

[0033] S23, according to center p c (k), and the preset region shape and side length, determine the second local region.

[0034] Preferably, S3 includes:

[0035] S31. Perform depth estimation on the endoscopic image to obtain a depth image corresponding to the endoscopic image; obtain the spatial and color information of each pixel from the depth image and the endoscopic image respectively by reading line by line to obtain the first local point cloud. Second local point cloud

[0036] S32. Determine the first two-dimensional key points. First local point cloud The first three-dimensional key point

[0037] P i (k-1)=ψ(p i (k-1))

[0038] ψ represents arrive The mapping relationship between them is denoted as

[0039] S33. Obtaining local regions using optical flow methods Let the feature point pairs be denoted as X and Y, then there is a coordinate transformation relationship between X and Y:

[0040]

[0041] in, These are the parameters of the fitted function; ω can be obtained using the least squares method, as follows:

[0042] ω=([X 1] T [X 1]) -1 [X 1] T Y

[0043] but The transformation matrix of the affine transformation between them is:

[0044]

[0045] Among them, 0 T = (0, 0, 0);

[0046] S34, By analyzing the first three-dimensional key points Perform a 3D affine transformation on the second local point cloud. Search for the nearest point to obtain the second and third-dimensional key points. The initial position,

[0047]

[0048] Preferably, in step S31, depth estimation is performed on the endoscopic image to obtain a depth image corresponding to the endoscopic image. The binocular depth estimation network used has the ability to quickly overlearn and can continuously adapt to new scenes using self-supervised information. Specifically, this includes:

[0049] S311. Acquire binocular endoscope images and extract multi-scale features of the current frame image using the encoder network of the current binocular depth estimation network.

[0050] S312. Using the decoder network of the current binocular depth estimation network, multi-scale features are fused to obtain the disparity of each pixel in the current frame image.

[0051] S313. Based on the camera's intrinsic and extrinsic parameters, convert the parallax into depth and output it as the result of the current frame image;

[0052] S314. Without introducing external ground truth, update the parameters of the current stereo depth estimation network using self-supervised loss for depth estimation of the next frame image.

[0053] Preferably, the optimization function in S4 refers to:

[0054]

[0055] in, Represents the optimization function;

[0056] Cosine similarity of SIFT feature vectors:

[0057]

[0058] φ(P i (k-1) represents the i-th second two-dimensional key point P. i The feature descriptor of (k-1) is a vector; φ(P) i (k) represents the i-th third two-dimensional key point P in the middle. i The feature descriptor of the neighborhood points of (k) is a vector; ||·|| represents the magnitude of the vector;

[0059] This represents the difference in the cosine of the angle between adjacent key points on different curves:

[0060]

[0061] Among them, g(P) i (k) is the cosine of the included angle, calculated as follows:

[0062]

[0063] a=(P i+1 (k)-P i (k)) T b = (P i-1 (k)-P i (k)) T

[0064] In step S4, based on the second and third two-dimensional key points, the two-dimensional coordinates of the tracked key points on the two-dimensional point cloud are obtained by minimizing a preset optimization function, including:

[0065] By traversing and searching all third-dimensional key points P i The neighborhood points of (k) make Minimize it to satisfy:

[0066]

[0067] By minimizing the optimization function, a set of ideal key points P can be obtained. i .

[0068] Preferably, in step S5, curve fitting is performed to finally track and obtain a three-dimensional curve, including:

[0069] The lines are interpolated and fitted using the B-spline curve equation, where the overall equation of the B-spline curve is:

[0070]

[0071] Among them, P i These are the characteristic points of the control curve, F. i，k (t) represents the k-th order B-spline basis function, which is used to achieve three-dimensional curve tracking through curve interpolation fitting;

[0072] Preferably, the first two-dimensional key point acquisition process in S1 includes:

[0073] definition This represents the number of pixels along the surgical path curve.

[0074] For a point on the curve, the curvature of the j-th pixel on the curve is:

[0075]

[0076] in, This represents the coordinates of the (j+α)th pixel on the curve, where α represents the number of pixel intervals when solving for the curvature of a pixel.

[0077] For the curvature of two consecutive pixels, when |K j+α -K j |≥ε, where ε is the curvature threshold; As a key point on the surgical path curve, along with the start and end points of the curve, all first-dimensional key points are determined. n is the total number of the first two-dimensional keys.

[0078] A guided 3D virtual fixture generation system for dynamic surgical path guidance includes:

[0079] The selection module is used to read endoscopic images, obtain the surgical path curve on the current frame image according to the doctor's selection, and obtain several first two-dimensional key points passed by the surgical path curve.

[0080] The tracking module is used to track a first local region containing a first two-dimensional key point on the current frame image and obtain a second local region on the next frame image;

[0081] The mapping module is used to map a first local region onto a first local point cloud and a second local region onto a second local point cloud according to the mapping relationship between the endoscopic image and the point cloud; and to determine the first three-dimensional key points of the first two-dimensional key points on the first local point cloud, and to obtain the second three-dimensional key points on the second local point cloud through coordinate transformation.

[0082] The optimization module is used to reduce the dimensionality of the first local point cloud to obtain the first two-dimensional point cloud, and to obtain the second two-dimensional key points of the first three-dimensional key points on the first two-dimensional point cloud.

[0083] The second local point cloud is reduced in dimensionality to obtain the second two-dimensional point cloud, and the second three-dimensional key points are obtained as the third two-dimensional key points on the second two-dimensional point cloud.

[0084] Based on the second and third two-dimensional key points, the two-dimensional coordinates of the tracked key points on the two-dimensional point cloud are obtained by minimizing the preset optimization function.

[0085] The fitting module is used to obtain the 3D coordinates of each tracked key point based on the mapping relationship between the point clouds before and after dimensionality reduction, and to perform curve fitting to finally obtain a 3D curve.

[0086] The guidance module is used to establish a guided virtual fixture along the tracked three-dimensional curve. The guided virtual fixture guides the movement trajectory of the surgical instruments through a force feedback mechanism.

[0087] A storage medium storing a computer program for generating a guided three-dimensional virtual fixture for dynamic surgical path guidance, wherein the computer program causes a computer to execute the guided three-dimensional virtual fixture generation method as described above.

[0088] An electronic device, characterized in that it comprises:

[0089] One or more processors;

[0090] Memory; and

[0091] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the guided 3D virtual fixture generation method as described above.

[0092] (III) Beneficial Effects

[0093] This invention provides a method, system, storage medium, and electronic device for generating guided 3D virtual fixtures for dynamic surgical path guidance. Compared with existing technologies, it has the following advantages:

[0094] In this invention, local regions are determined by the location of key points, and these regions are tracked to reduce erroneous tracking of 3D key points. Combining texture and shape information from endoscopic images mitigates the impact of indistinct internal environmental features. Furthermore, an optimization function is constructed on the dimensionality-reduced point cloud to accurately locate 3D key points, avoiding inconsistencies in curve shapes caused by different viewpoints. Moreover, a virtual force field incorporating gravity and viscous drag is established along the tracked 3D curve—a guided virtual clamp—which precisely guides the movement trajectory of surgical instruments through a force feedback mechanism. Attached Figure Description

[0095] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0096] Figure 1 This is a flowchart illustrating a guided 3D virtual fixture generation method for dynamic surgical path guidance provided in an embodiment of the present invention.

[0097] Figure 2 A schematic diagram of a network training architecture provided in an embodiment of the present invention;

[0098] Figure 3This is a schematic diagram of left and right parallax acquisition provided in an embodiment of the present invention;

[0099] Figure 4 This is a schematic diagram of a network application provided in an embodiment of the present invention. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0101] This application provides a guided 3D virtual fixture generation method, system, storage medium, and electronic device for dynamic surgical path guidance, solving the technical problem of the inability to achieve precise guidance of the 3D surgical path.

[0102] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0103] This invention proposes a guided 3D virtual fixture generation method for dynamic surgical path guidance. Its main application, but not limited to, is in minimally invasive endoscopic surgery scenarios, which can improve the quality of remote surgical guidance and the autonomous operation capability of surgical robots. During surgery, tactile feedback is crucial for the surgeon's control. Force feedback can effectively help surgeons solve problems related to the control of surgical instruments with non-periodic motion, such as organ protection, target guidance, and obstacle avoidance, thereby ensuring the safety and precision of the surgical procedure.

[0104] This invention, based on an in vivo flexible environment 3D reconstruction algorithm, guides surgeons to draw an initial surgical path on a 2D image. The computer then uses intraoperative images to perform real-time 3D point cloud reconstruction, updating the 3D surgical path and ultimately establishing a dynamic 3D surgical path. A virtual force field incorporating gravity and viscous drag—a guided virtual fixture—is constructed. Through a force feedback mechanism, the surgeon is guided to manipulate the surgical robot system along a pre-set path. This improves the surgeon's feel for controlling the robot during surgery, effectively helping them complete the operation along the pre-set path and achieving human-machine collaborative control that aligns with the surgeon's intentions.

[0105] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0106] Example:

[0107] like Figure 1As shown, this embodiment of the invention provides a method for generating a guided 3D virtual fixture for dynamic surgical path guidance, including:

[0108] S1. Read the endoscopic image, obtain the surgical path curve on the current frame image according to the doctor's selection, and obtain several first two-dimensional key points through which the surgical path curve passes.

[0109] S2. Track the first local region containing the first two-dimensional key point on the current frame image, and obtain the second local region on the next frame image;

[0110] S3. Based on the mapping relationship between the endoscopic image and the point cloud, the first local region is mapped onto the first local point cloud and the second local region is mapped onto the second local point cloud respectively; and the first two-dimensional key point is determined as the first three-dimensional key point on the first local point cloud, and the second three-dimensional key point on the second local point cloud is obtained through coordinate transformation.

[0111] S4. Reduce the dimensionality of the first local point cloud to obtain the first two-dimensional point cloud, and obtain the second two-dimensional key point of the first three-dimensional key point on the first two-dimensional point cloud;

[0112] The second local point cloud is reduced in dimensionality to obtain the second two-dimensional point cloud, and the second three-dimensional key points are obtained as the third two-dimensional key points on the second two-dimensional point cloud.

[0113] Based on the second and third two-dimensional key points, the two-dimensional coordinates of the tracked key points on the two-dimensional point cloud are obtained by minimizing the preset optimization function.

[0114] S5. Based on the mapping relationship between the point clouds before and after dimensionality reduction, obtain the three-dimensional coordinates of each tracked key point, and perform curve fitting to finally obtain a three-dimensional curve.

[0115] S6. Along the three-dimensional curve obtained by tracking, a guided virtual fixture is established. The guided virtual fixture guides the movement trajectory of the surgical instruments through a force feedback mechanism.

[0116] In this embodiment of the invention, local regions are determined by the location of key points, and these local regions are tracked to reduce erroneous tracking of 3D key points. Combining the texture and shape information of endoscopic images mitigates the impact of indistinct internal environmental features to some extent. Furthermore, an optimization function is constructed on the dimensionality-reduced point cloud to accurately locate the 3D key points, avoiding inconsistencies in curve shapes caused by different viewpoints. Moreover, a virtual force field containing gravity and viscous drag, i.e., a guided virtual clamp, is established along the tracked 3D curve. This guided virtual clamp precisely guides the movement trajectory of surgical instruments through a force feedback mechanism.

[0117] The following section will detail each step of the above plan, with specific examples:

[0118] In step S1, an endoscopic image is captured, and the surgical path curve on the current frame image is obtained according to the doctor's selection. Several first two-dimensional key points passed through by the surgical path curve are also obtained.

[0119] The process of obtaining the first two-dimensional key points in S1 includes:

[0120] definition This represents the pixels along the surgical path curve.

[0121] For a point on the curve, the curvature of the j-th pixel on the curve is:

[0122]

[0123] in, This represents the coordinates of the (j+α)th pixel on the curve, where α represents the number of pixel intervals when solving for the curvature of a pixel.

[0124] For the curvature of two consecutive pixels, when |K j+α -K j |≥ε, where ε is the curvature threshold; As a key point on the surgical path curve, along with the start and end points of the curve, all first-dimensional key points are determined. n is the total number of the first two-dimensional keys.

[0125] In this step, the surgeon marks 3D curves on intraoperative images for subsequent display and updating on a 3D point cloud, ensuring intuitive and accurate information transmission and improving surgical efficiency. Furthermore, key points are determined based on the curve's curvature, and tracking these key points ensures computational speed.

[0126] In step S2, a first local region containing the first two-dimensional keypoint is tracked in the current frame image, and a second local region in the next frame image is obtained; specifically including...

[0127] S21. First, define the (k-1)th frame image as... Where W represents the width of the endoscopic image, and H represents the height of the endoscopic image; the k-th frame image is I(k); several first two-dimensional key points

[0128] Determine the maximum and minimum values ​​of all first-dimensional keypoints on the u-axis and v-axis of the image coordinate system, and select the position of the median value on each axis as p. c (k-1)=(u1,v1)∈R 2 The first local region is determined based on the preset region shape and side length.

[0129] S22. Use optical flow to perform feature matching on image I(k-1) and I(k), and obtain the feature points on image I(k) that are related to p. c The center p of the second local region corresponding to (k-1) c (k),

[0130]

[0131] in, Let m represent the feature points on image I(k), and m represent the number of feature points on image I(k).

[0132] S23, according to center p c (k), and the preset region shape and side length, determine the second local region.

[0133] This invention reduces erroneous tracking of 3D key points by determining local regions based on the location of key points and tracking these local regions.

[0134] In step S3, based on the mapping relationship between the endoscopic image and the point cloud, the first local region is mapped onto the first local point cloud and the second local region is mapped onto the second local point cloud, respectively; and the first two-dimensional key point is determined as the first three-dimensional key point on the first local point cloud, and the second three-dimensional key point on the second local point cloud is obtained through coordinate transformation.

[0135] This step is actually the initial localization of 3D key points, which can be achieved through the following two steps.

[0136] First, based on the mapping relationship between the endoscopic image and the point cloud, the three-dimensional key points corresponding to the two-dimensional key points are determined on the point cloud by using the position of the two-dimensional key points. Second, the tissue in the local area can be approximated as a rigid body, and the transformation matrix between the point clouds can be solved by using three-dimensional affine transformation. The three-dimensional key points on the target point cloud are obtained through coordinate transformation.

[0137] Accordingly, S3 specifically includes:

[0138] S31. Perform depth estimation on the endoscopic image to obtain a depth image corresponding to the endoscopic image; obtain the spatial and color information of each pixel from the depth image and the endoscopic image respectively by reading line by line to obtain the first local point cloud. Second local point cloud

[0139] S32. Determine the first two-dimensional key points. First local point cloud The first three-dimensional key point

[0140] P i (k-1)=ψ(p i (k-1))

[0141] ψ represents arrive The mapping relationship between them is denoted as

[0142] S33. Obtaining local regions using optical flow methods Let the feature point pairs be denoted as X and Y, then there is a coordinate transformation relationship between X and Y:

[0143]

[0144] in, These are the parameters of the fitted function; ω can be obtained using the least squares method, as follows:

[0145] ω=([X 1] T [X 1]) -1 [X 1] T Y

[0146] but The transformation matrix of the affine transformation between them is:

[0147]

[0148] Among them, 0 T = (0, 0, 0);

[0149] S34, By analyzing the first three-dimensional key points Perform a 3D affine transformation on the second local point cloud. Search for the nearest point to obtain the second and third-dimensional key points. The initial position,

[0150]

[0151] Specifically, in step S31, depth estimation is performed on the endoscopic image to obtain a depth image corresponding to the endoscopic image. The binocular depth estimation network used has the ability to quickly overlearn and can continuously adapt to new scenes using self-supervised information. Specifically, this includes:

[0152] S311. Acquire binocular endoscope images and extract multi-scale features of the current frame image using the encoder network of the current binocular depth estimation network.

[0153] S312. Using the decoder network of the current binocular depth estimation network, multi-scale features are fused to obtain the disparity of each pixel in the current frame image.

[0154] S313. Based on the camera's intrinsic and extrinsic parameters, convert the parallax into depth and output it as the result of the current frame image;

[0155] S314. Without introducing external ground truth, update the parameters of the current stereo depth estimation network using self-supervised loss for depth estimation of the next frame image.

[0156] The aforementioned method for acquiring depth images utilizes the similarity of consecutive frames to extend the overfitting concept from a pair of binocular images to overfitting over time series. By continuously updating the model parameters through online learning, it can obtain high-precision tissue depth in various binocular endoscopic surgical environments.

[0157] Specifically, the pre-training stage of the network model abandons the traditional training mode and adopts the idea of ​​meta-learning, which allows the network to learn the depth of one image to predict the depth of another image, thereby calculating the loss and updating the network. This can effectively promote the network's generalization to new scenes and improve its robustness to low-texture complex lighting, while significantly reducing the time required for subsequent overfitting.

[0158] like Figure 2 As shown, the initial model parameters corresponding to the stereo depth estimation network are obtained through meta-learning training, specifically including:

[0159] S100, Randomly select an even number of pairs of stereo images {e1, e2, ..., e...} 2K} and equally divide it into support sets and query set and Images are randomly paired to form K tasks

[0160] S200, Inner Circulation Training: Based on The loss is calculated from the support set image to perform a parameter update;

[0161]

[0162] in, This represents the network parameters after the inner loop update; Let α represent the derivative, where α is the learning rate of the inner loop. For the support set image of the k-th task, It is based on the initial parameters φ of the model m Calculated loss;

[0163] S300, External Loop Training: Based on The query set image is used to calculate the meta-learning loss using the updated model, and the initial parameters φ of the model are directly updated. m For φ m+1 ;

[0164]

[0165] Where β is the learning rate of the outer loop; This is the query set image for the k-th task. This is the learning loss of the meta-learning.

[0166] In step S311, as Figure 3 As shown, binocular endoscopic images are acquired, and the encoder network of the current binocular depth estimation network is used to extract multi-scale features of the current frame image.

[0167] For example, in this step, the encoder of the binocular depth estimation network is selected to be a ResNet18 network, which is used to extract feature maps at five scales from the endoscopic image.

[0168] In step S312, as Figure 3 As shown, the decoder network of the current binocular depth estimation network is used to fuse multi-scale features and obtain the disparity of each pixel in the current frame image; specifically, it includes:

[0169] The decoder is used to pass the coarse-scale feature map through a convolutional block and upsampling, and then concatenate it with the fine-scale feature map. The feature map is then fused through another convolutional block, wherein the convolutional block is constructed by combining reflection padding, convolutional layers, and non-linear activation units (ELUs).

[0170] Calculate the disparity directly based on the output with the highest network resolution:

[0171] d = k·(sigmoid(conv(Y)) - 0.5)

[0172] Where d represents the disparity estimate of a pixel; k is the preset maximum disparity range; Y is the highest resolution output; conv is the convolutional layer; and sigmoid is used for range normalization.

[0173] In step S313, the parallax is converted into depth based on the camera's intrinsic and extrinsic parameters and output as the result of the current frame image.

[0174] In this step, converting parallax to depth means:

[0175]

[0176] Where D is the depth estimate of the pixel; f x c x2 cx1 'b' represents the intrinsic parameters of the binocular camera; 'b' represents the baseline length, i.e., the extrinsic parameters of the binocular camera.

[0177] In step S314, as Figure 4 As shown, without introducing external ground truth, the parameters of the current stereo depth estimation network are updated using self-supervised loss for depth estimation of the next frame image.

[0178] Self-monitoring losses include:

[0179] (1) Geometric consistency loss D diff (p):

[0180]

[0181] in, D′ represents the right eye depth obtained after transforming the left eye depth map. r This represents the right eye depth obtained by sampling on the right eye depth map;

[0182] By incorporating geometric consistency constraints into the training loss, the network's general applicability to hardware is ensured, enabling it to autonomously adapt to unconventional binocular images such as surgical endoscopes.

[0183] (2) Photometric loss

[0184]

[0185] Where p represents a valid pixel, and P represents the set of valid pixels; I t and I′ t Let λ represent the original image and the reconstructed image, respectively. i and λ s To balance the parameters, SSIM tt′ Indicates image structural similarity;

[0186] (3) Smoothing loss

[0187]

[0188] Among them, D n Represents a normalized depth map. and This represents the first derivative along the horizontal and vertical directions of the image.

[0189] In summary, the above-described method for acquiring depth images treats depth estimation for each frame of the binocular image as an independent task, and uses real-time overfitting to obtain a high-precision model suitable for the current frame; moreover, it can quickly learn new scenes through online learning to obtain high-precision depth estimation results.

[0190] In step S4, the first local point cloud is dimensionality reduced to obtain a first two-dimensional point cloud, and the first three-dimensional key point is obtained as a second two-dimensional key point on the first two-dimensional point cloud;

[0191] The second local point cloud is reduced in dimensionality to obtain the second two-dimensional point cloud, and the second three-dimensional key points are obtained as the third two-dimensional key points on the second two-dimensional point cloud.

[0192] Based on the second and third two-dimensional key points, the two-dimensional coordinates of the tracked key points on the two-dimensional point cloud are obtained by minimizing the preset optimization function.

[0193] This step is essentially about precisely locating the 3D key points. To ensure accuracy when the curve changes dynamically, the initial positions of the key points on the curve need to be optimized.

[0194] This step first combines the texture and shape information of the endoscopic image to avoid the influence of the unclear features of the internal environment to a certain extent; secondly, an optimization function is constructed on the dimensionality-reduced point cloud to accurately locate the 3D key points, avoiding the inconsistency of curve shape caused by different viewpoints.

[0195] Specifically, firstly, the first local point cloud... Dimensionality reduction is performed to obtain the first two-dimensional point cloud Q(k-1), and the first three-dimensional keypoints Q(k-1) are obtained in the first two-dimensional point cloud. The i-th second two-dimensional key point P on i (k-1);

[0196] The second local point cloud Dimensionality reduction is performed to obtain the second two-dimensional point cloud Q(k), and the i-th third two-dimensional keypoint P of the second three-dimensional keypoint Q(k) on the second two-dimensional point cloud is obtained. i (k).

[0197] Then, based on the second and third two-dimensional key points, the two-dimensional coordinates of the tracked key points on the two-dimensional point cloud are obtained by minimizing a preset optimization function, including:

[0198] The optimization function refers to:

[0199]

[0200] in, Represents the optimization function;

[0201] Cosine similarity of SIFT feature vectors:

[0202]

[0203] φ(Pi (k-1) represents the i-th second two-dimensional key point P. i The feature descriptor of (k-1) is a vector; φ(P) i (k) represents the i-th third two-dimensional key point P in the middle. i The feature descriptor of the neighborhood points of (k) is a vector; ||·|| represents the magnitude of the vector;

[0204] This represents the difference in the cosine of the angle between adjacent key points on different curves:

[0205]

[0206] Among them, g(P) i (k) is the cosine of the included angle, calculated as follows:

[0207]

[0208] a=(P i+1 (k)-P i (k)) T b = (P i-1 (k)-P i (k)) T .

[0209] By traversing and searching all third-dimensional key points P i The neighborhood points of (k) make Minimize it to satisfy:

[0210]

[0211] By minimizing the optimization function, a set of ideal key points P can be obtained. i .

[0212] In this embodiment of the invention, after obtaining the transformation matrix through three-dimensional affine transformation, the texture and shape information of the endoscopic image are combined with the constructed optimization function to accurately locate the three-dimensional key points, thereby avoiding the influence of the indistinct features of the internal environment on the tracking results to a certain extent.

[0213] In step S5, based on the mapping relationship between the point clouds before and after dimensionality reduction, the three-dimensional coordinates of each tracked key point are obtained, and curve fitting is performed to finally obtain a three-dimensional curve.

[0214] In this step, the B-spline curve equation is used to interpolate and fit the lines. The overall equation of the B-spline curve is:

[0215]

[0216] Among them, Pi These are the characteristic points of the control curve, F. i，k (t) represents the k-th order B-spline basis function, which is used to achieve three-dimensional curve tracking through curve interpolation fitting.

[0217] In step S6, a guided virtual fixture is established along the tracked three-dimensional curve. This guided virtual fixture guides the movement trajectory of the surgical instruments through a force feedback mechanism; specifically, it includes:

[0218] S61. Convert the real-time three-dimensional position information of the surgical instrument and the point cloud obtained in the previous steps to the same coordinate system, and use the K-nearest neighbor algorithm to search for the distance p from the end of the surgical instrument on the three-dimensional curve. ins The nearest point p nearest and the relative distance x between the two points. d This allows us to determine whether the surgical instruments have deviated from the pre-set surgical path curve.

[0219] Specifically, during the surgery, the surgeon monitors the real-time three-dimensional position information of the surgical instruments. ins The position of the three-dimensional curve obtained by measuring the position and depth of the surgical instrument is transformed into the same coordinate system after the relative position coordinate transformation is performed.

[0220] S62, Establish information about p ins Artificial vector field:

[0221] Define all points on a 3D curve as contained in the point set V. r In the middle, find V r The subset V p ={v|||p ins -v||≤||p ins -p nearest ||+Δr}, where Δr is the additional radius of investigation;

[0222]

[0223] Where g is p ins Artificial vector field; g(||p ins -v||) is the artificial vector length function, n(p ins v) is the value from point p ins The vector pointing to point v, when v corresponds to different organizations, has different artificial vector length functions g(||p). ins -v||), satisfying the condition card(V p ) is a subset V p The number of elements in the middle;

[0224] S63. Establish a virtual force field that includes gravity and viscous drag:

[0225]

[0226] Among them, F vf For virtual force field; K vf D is the proportionality coefficient between the gravitational force and the artificial vector at that location. vf (x d ) is the damping coefficient, which varies with the shortest distance x d Move; For p ins The speed.

[0227] This invention establishes a human-machine tactile interaction mechanism between the surgeon and the robot through haptic feedback technology. A guided virtual gripper is constructed in space to precisely guide the surgeon's movement trajectory, ensuring the surgeon can control the robot according to a predetermined path. While improving surgical precision, the surgeon does not need to wear other equipment, preventing interference with the surgical procedure.

[0228] This invention provides a guided 3D virtual fixture generation system for dynamic surgical path guidance, comprising:

[0229] The selection module is used to read endoscopic images, obtain the surgical path curve on the current frame image according to the doctor's selection, and obtain several first two-dimensional key points passed by the surgical path curve.

[0230] The tracking module is used to track a first local region containing a first two-dimensional key point on the current frame image and obtain a second local region on the next frame image;

[0231] The mapping module is used to map a first local region onto a first local point cloud and a second local region onto a second local point cloud according to the mapping relationship between the endoscopic image and the point cloud; and to determine the first three-dimensional key points of the first two-dimensional key points on the first local point cloud, and to obtain the second three-dimensional key points on the second local point cloud through coordinate transformation.

[0232] The optimization module is used to reduce the dimensionality of the first local point cloud to obtain the first two-dimensional point cloud, and to obtain the second two-dimensional key points of the first three-dimensional key points on the first two-dimensional point cloud.

[0233] The second local point cloud is reduced in dimensionality to obtain the second two-dimensional point cloud, and the second three-dimensional key points are obtained as the third two-dimensional key points on the second two-dimensional point cloud.

[0234] Based on the second and third two-dimensional key points, the two-dimensional coordinates of the tracked key points on the two-dimensional point cloud are obtained by minimizing the preset optimization function.

[0235] The fitting module is used to obtain the 3D coordinates of each tracked key point based on the mapping relationship between the point clouds before and after dimensionality reduction, and to perform curve fitting to finally obtain a 3D curve.

[0236] The guidance module is used to establish a guided virtual fixture along the tracked three-dimensional curve. The guided virtual fixture guides the movement trajectory of the surgical instruments through a force feedback mechanism.

[0237] This invention provides a storage medium storing a computer program for generating guided three-dimensional virtual fixtures for dynamic surgical path guidance, wherein the computer program causes a computer to execute the guided three-dimensional virtual fixture generation method as described above.

[0238] This invention provides an electronic device, comprising:

[0239] One or more processors;

[0240] Memory; and

[0241] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the guided 3D virtual fixture generation method as described above.

[0242] It is understood that the guided three-dimensional virtual jig generation system, storage medium and electronic device for dynamic surgical path guidance provided in the embodiments of the present invention correspond to the guided three-dimensional virtual jig generation method for dynamic surgical path guidance provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can be referred to the corresponding parts of the guided three-dimensional virtual jig generation method, and will not be repeated here.

[0243] In summary, compared with existing technologies, it has the following beneficial effects:

[0244] 1. In this embodiment of the invention, local regions are determined by the location of key points, and these local regions are tracked to reduce erroneous tracking of 3D key points. Combining the texture and shape information of endoscopic images helps to mitigate the impact of indistinct internal environmental features. An optimization function is constructed on the dimensionality-reduced point cloud to accurately locate the 3D key points, avoiding inconsistencies in curve shapes caused by different viewpoints. Furthermore, a virtual force field containing gravity and viscous drag is established along the tracked 3D curve, i.e., a guided virtual clamp. This guided virtual clamp precisely guides the movement trajectory of surgical instruments through a force feedback mechanism.

[0245] 2. In this step, the surgeon marks 3D curves on the intraoperative images for subsequent display and updating on the 3D point cloud, ensuring the intuitiveness and accuracy of information transmission and improving surgical efficiency. Furthermore, key points are determined based on the curvature of the curves, and tracking these key points ensures computational speed.

[0246] 3. In this embodiment of the invention, after obtaining the transformation matrix through three-dimensional affine transformation, the texture and shape information of the endoscopic image are combined with the construction of an optimization function to accurately locate the three-dimensional key points, thereby avoiding the influence of the indistinct features of the internal environment on the tracking results to a certain extent.

[0247] 4. The pre-training stage of the binocular depth estimation network model abandons the traditional training mode and adopts the idea of ​​meta-learning, which allows the network to learn the depth of one image to predict the depth of another image, thereby calculating the loss and updating the network. This can effectively promote the network's generalization to new scenes and improve its robustness to low-texture complex lighting, while significantly reducing the time required for subsequent overfitting.

[0248] 5. In the implementation of this invention, the depth estimation of each frame of binocular image is treated as an independent task, and a high-precision model suitable for the current frame is obtained through real-time overfitting; and through online learning, new scenes can be learned quickly to obtain high-precision depth estimation results.

[0249] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0250] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating guided 3D virtual fixtures for dynamic surgical path guidance, characterized in that, include: S1. Read the endoscopic image, obtain the surgical path curve on the current frame image according to the doctor's selection, and obtain several first two-dimensional key points through which the surgical path curve passes. S2. Track the first local region containing the first two-dimensional key point on the current frame image, and obtain the second local region on the next frame image; S3. Based on the mapping relationship between the endoscopic image and the point cloud, the first local region is mapped onto the first local point cloud and the second local region is mapped onto the second local point cloud respectively; and the first two-dimensional key point is determined as the first three-dimensional key point on the first local point cloud, and the second three-dimensional key point on the second local point cloud is obtained through coordinate transformation. S4. Reduce the dimensionality of the first local point cloud to obtain the first two-dimensional point cloud, and obtain the second two-dimensional key point of the first three-dimensional key point on the first two-dimensional point cloud; The second local point cloud is reduced in dimensionality to obtain the second two-dimensional point cloud, and the second three-dimensional key points are obtained as the third two-dimensional key points on the second two-dimensional point cloud. Based on the second and third two-dimensional key points, the two-dimensional coordinates of the tracked key points on the two-dimensional point cloud are obtained by minimizing the preset optimization function. S5. Based on the mapping relationship between the point clouds before and after dimensionality reduction, obtain the three-dimensional coordinates of each tracked key point, and perform curve fitting to finally obtain a three-dimensional curve. S6. Along the three-dimensional curve obtained by tracking, a guided virtual fixture is established. The guided virtual fixture guides the movement trajectory of the surgical instruments through a force feedback mechanism.

2. The guided three-dimensional virtual fixture generation method as described in claim 1, characterized in that, S6 specifically includes: S61. Convert the real-time three-dimensional position information of the surgical instrument and the point cloud obtained in the previous steps to the same coordinate system, and obtain the distance from the end of the surgical instrument on the three-dimensional curve. nearest point and the relative distance between the two points. ; S62, Establishing a framework regarding Artificial vector field: Define all points on a 3D curve as contained in a point set. In the middle, seek subset ,in For additional investigation radius; in, for Artificial vector field; For artificial vector length function, As a point Point of view The vector, when Different artificial vector length functions are established for different organizations. The conditions are met. ; It is a subset The number of elements in the middle; S63. Establish a virtual force field that includes gravity and viscous drag: in, For virtual force fields; This is the proportionality coefficient between the gravitational force and the artificial vector at that location. The damping coefficient varies with relative distance. Move; for The speed.

3. The guided three-dimensional virtual fixture generation method as described in claim 1 or 2, characterized in that, The S2 includes S21, First define the first Frame image is ,in Indicates the width of the endoscopic image. Indicates the height of the endoscopic image; the first Frame image is Several first-dimensional key points ; Determine the maximum and minimum values ​​of all first-dimensional keypoints on the u-axis and v-axis of the image coordinate system, and select the position of the median value on each axis as... The first local region is determined based on the preset region shape and side length. ; S22. Image processing using optical flow method. Perform feature matching of feature points to obtain the image. Above and The center of the corresponding second local region , in, Representing an image Feature points on, Representing an image The number of feature points; S23, according to the center The second local region is determined by the preset region shape and side length. .

4. The guided three-dimensional virtual fixture generation method as described in claim 3, characterized in that, S3 includes: S31. Perform depth estimation on the endoscopic image to obtain a depth image corresponding to the endoscopic image; obtain the spatial and color information of each pixel from the depth image and the endoscopic image respectively by reading line by line to obtain the first local point cloud. Second local point cloud ; S32. Determine the first two-dimensional key points. First local point cloud The first three-dimensional key point , express arrive The mapping relationship between them is denoted as ; S33. Obtaining local regions using optical flow methods Feature point pairs, respectively denoted as and ,but and There is a coordinate transformation relationship between them: in, These are the parameters of the fitted function; by using least squares, It can be obtained from the following formula: but The transformation matrix of the affine transformation between them is: in, ; S34, By analyzing the first three-dimensional key points Perform a 3D affine transformation on the second local point cloud. Search for the nearest point to obtain the second and third-dimensional key points. The initial position, 。 5. The guided three-dimensional virtual fixture generation method as described in claim 4, characterized in that, In step S31, depth estimation is performed on the endoscopic image to obtain a depth image corresponding to the endoscopic image. The binocular depth estimation network used has fast overlearning capabilities and can continuously adapt to new scenes using self-supervised information. Specifically, it includes: S311. Acquire binocular endoscope images and extract multi-scale features of the current frame image using the encoder network of the current binocular depth estimation network. S312. Using the decoder network of the current binocular depth estimation network, multi-scale features are fused to obtain the disparity of each pixel in the current frame image. S313. Based on the camera's intrinsic and extrinsic parameters, convert the parallax into depth and output it as the result of the current frame image; S314. Without introducing external ground truth, update the parameters of the current stereo depth estimation network using self-supervised loss for depth estimation of the next frame image.

6. The guided three-dimensional virtual fixture generation method as described in claim 1 or 2, characterized in that, The optimization function in S4 refers to: in, Represents the optimization function; Cosine similarity of SIFT feature vectors: Indicates the first A second two-dimensional key point The feature descriptor is a vector; Indicates the middle A third two-dimensional key point The feature descriptor of the neighborhood points is a vector; Represents the magnitude of a vector; This represents the difference in the cosine of the angle between adjacent key points on different curves: in, Let be the cosine of the included angle, calculated as follows: In step S4, based on the second and third two-dimensional key points, the two-dimensional coordinates of the tracked key points on the two-dimensional point cloud are obtained by minimizing a preset optimization function, including: By iterating through and searching all third-dimensional key points The neighborhood points make Minimize it to satisfy: By minimizing the optimization function, a set of ideal key points can be obtained. .

7. The guided three-dimensional virtual fixture generation method as described in claim 6, characterized in that, In step S5, curve fitting is performed to ultimately obtain a three-dimensional curve, including: The lines are interpolated and fitted using the B-spline curve equation, where the overall equation of the B-spline curve is: in, These are the characteristic points of the control curve. express B-order spline basis functions are used to achieve three-dimensional curve tracking through curve interpolation fitting. And / or the first two-dimensional key point acquisition process in S1 includes: definition This represents the number of pixels along the surgical path curve. For a point on the curve, the first point on the curve... The curvature of each pixel is: in, Indicates the first curve The coordinates of each pixel This represents the number of pixel intervals when calculating pixel curvature. For the curvature of two consecutive pixels, when , Curvature threshold; As a key point on the surgical path curve, along with the start and end points of the curve, all first-dimensional key points are determined. , This represents the total number of the first two-dimensional keys.

8. A guided 3D virtual fixture generation system for dynamic surgical path guidance, characterized in that, include: The selection module is used to read endoscopic images, obtain the surgical path curve on the current frame image according to the doctor's selection, and obtain several first two-dimensional key points passed by the surgical path curve. The tracking module is used to track a first local region containing a first two-dimensional key point on the current frame image and obtain a second local region on the next frame image; The mapping module is used to map a first local region onto a first local point cloud and a second local region onto a second local point cloud according to the mapping relationship between the endoscopic image and the point cloud; and to determine the first three-dimensional key points of the first two-dimensional key points on the first local point cloud, and to obtain the second three-dimensional key points on the second local point cloud through coordinate transformation. The optimization module is used to reduce the dimensionality of the first local point cloud to obtain the first two-dimensional point cloud, and to obtain the second two-dimensional key points of the first three-dimensional key points on the first two-dimensional point cloud. The second local point cloud is reduced in dimensionality to obtain the second two-dimensional point cloud, and the second three-dimensional key points are obtained as the third two-dimensional key points on the second two-dimensional point cloud. Based on the second and third two-dimensional key points, the two-dimensional coordinates of the tracked key points on the two-dimensional point cloud are obtained by minimizing the preset optimization function. The fitting module is used to obtain the 3D coordinates of each tracked key point based on the mapping relationship between the point clouds before and after dimensionality reduction, and to perform curve fitting to finally obtain a 3D curve. The guidance module is used to establish a guided virtual fixture along the tracked three-dimensional curve. The guided virtual fixture guides the movement trajectory of the surgical instruments through a force feedback mechanism.

9. A storage medium, characterized in that, It stores a computer program for generating guided three-dimensional virtual fixtures for dynamic surgical path guidance, wherein the computer program causes a computer to execute the guided three-dimensional virtual fixture generation method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the guided three-dimensional virtual fixture generation method as described in any one of claims 1 to 7.

Images