In-vivo safe distance identification system for minimally invasive dissection path constraints

Abstract

The application provides a kind of in-vivo safety distance marking system for minimally invasive dissection path constraint, and relates to the technical field of minimally invasive surgery.The application is based on intraoperative assisted safety area marking algorithm, obtains real three-dimensional information of surgical site through precisely reconstructed tissue point cloud model, receives dissection point and dissection direction provided by the doctor, and sets safety distance, calculates safety distance edge point and displays safety distance line, prompts the direction and distance of the operation of the doctor in the surgical site, and simultaneously has the accuracy of direction and distance.It greatly reduces the possible surgical errors caused by the doctor's intraoperative judgment relying on his own experience.In addition, the introduced online self-supervised learning depth estimation method based on binocular endoscope has the ability of fast over-learning, can continuously adapt to new scenes using self-supervised information, and thus ensures the accuracy of calculating the real length between the dissection starting point and the safety distance edge point in the complex and variable environment.

Description

Technical Field

[0001] This invention relates to the field of minimally invasive surgery technology, and more specifically to an in vivo safety distance marking system for minimally invasive anatomical path constraints. Background Technology

[0002] Compared to traditional open surgery, minimally invasive surgery (such as endoscopic surgery) has advantages such as smaller incisions, less bleeding, and faster recovery, and is gradually being widely adopted.

[0003] Existing surgical navigation systems' area identification assistance functions can only guide surgeons to operate in the correct direction. For example, Chinese patent CN112807083A discloses a "method and system for establishing a cranial puncture path for neurosurgical navigation," which includes: generating a three-dimensional cranial model based on medical imaging data of the cranium; identifying several feature points in the cranial model and requesting confirmation or manual modification of the geometric features of the feature points in the cranial model; constructing a three-dimensional coordinate system of the cranial model based on the confirmed or manually modified geometric features of the feature points; generating danger zones and punctureable zones in the cranial model; and establishing a puncture path based on the danger zones and punctureable zones.

[0004] However, operating distance is also an essential control factor in surgical navigation, and the aforementioned methods rely solely on the surgeon's experience and observation of the screen for judgment. During surgery, complex anatomical structures, dynamically moving tissues, and reflective smoke can mislead the surgeon's judgment, increasing surgical stress, reducing operational accuracy, and decreasing surgical efficiency. Therefore, it is necessary to provide a new technical solution that can not only correctly guide the direction during surgery but also assist the surgeon in quantitative distance control. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides an in vivo safety distance marking system for minimally invasive anatomical path constraints, which solves the technical problem of not being able to correctly guide the direction during surgery while assisting doctors in quantitative distance control.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] An in vivo safety distance marking system for minimally invasive anatomical pathway constraints includes:

[0010] The modeling module is used to obtain intraoperative tissue point cloud models based on the depth values ​​of specified binocular endoscopic image frames;

[0011] The receiving module is used to receive the anatomical starting point, anatomical direction, and set safety distance marked by the doctor on the region of interest of the specified binocular endoscopic image frame;

[0012] The identification module is used to obtain the edge points of the safety distance based on the intraoperative tissue point cloud model, the anatomical starting point, the anatomical direction and the safety distance, identify the line connecting the anatomical starting point and the edge points of the safety distance, and use it as the safety distance line on the specified binocular endoscope image frame.

[0013] The tracking module is used to track the anatomical starting point and the safety distance edge point for each binocular endoscope image frame after the specified binocular endoscope image frame, and to display the safety distance line in real time on each binocular endoscope image frame.

[0014] Preferably, the modeling module is used for:

[0015] A depth estimation algorithm is used to estimate the depth values ​​of pixels in a specified binocular endoscope image frame. The spatial coordinates of the pixels in the camera coordinate system are calculated using a pinhole camera model. The formula is as follows:

[0016]

[0017]

[0018] z = D

[0019] Where D is the depth estimate of the pixel; x, y, and z represent the x, y, and z coordinates in the camera coordinate system, respectively; c x ,c y ,f x ,f y The intrinsic parameter matrix between the left or right eye endoscope and the camera in a binocular endoscope. The corresponding parameters;

[0020] The specified binocular endoscopic image frame is converted into an intraoperative tissue point cloud model V. in ={v in,a |a=1,2,…N1},v in,a This represents the spatial coordinates of the a-th pixel. The model has a total of N1 pixels.

[0021] Preferably, the identification module includes:

[0022] The selection unit is used to arbitrarily select a first pixel point that does not coincide with the anatomical starting point along the anatomical direction on the specified binocular endoscope image frame.

[0023] The first summing unit is used to uniformly sample along the two-dimensional line connecting the anatomical starting point and the first pixel to obtain a number of sampling points; and based on the intraoperative tissue point cloud model, to approximate the spatial curve as the sum of line segments between each sampling point, and to obtain the true length L of the two-dimensional line b1 between the anatomical starting point and the first pixel projected into three-dimensional space using an accumulation summation method. b1 ;

[0024] The first judgment unit is used to judge L. b1 The relationship between L and the safe distance, if L b1 >L and |L b1 -L|>ε safe The midpoint of the two-dimensional line segment connecting the dissection starting point and the first pixel point is obtained using the binary search method and denoted as the second pixel point. The process then proceeds to the second summation unit. If L... b1 <and|L b1 -|> safe Return to the selected unit; if |L b1 -|≤ε safe Then, proceed to the termination unit; where ε safe To set a threshold;

[0025] The second summation unit is used to uniformly sample along the two-dimensional line connecting the anatomical starting point and the second pixel to obtain several sampling points; and based on the intraoperative tissue point cloud model, to approximate the spatial curve as the sum of line segments between each sampling point, and to obtain the true length L of the two-dimensional line b2 between the anatomical starting point and the second pixel projected into three-dimensional space using an accumulation summation method. b2 ;

[0026] The second judgment unit is used to judge L. b2 The relationship between the magnitude of L and the safety distance L, if L b2 >and|L b2 -|> safe The midpoint of the two-dimensional line segment connecting the dissection starting point and the second pixel is obtained using a binary search method. The second pixel is then updated to this midpoint, and the result is returned to the second summation unit. If L b2 <and|L b2 -|> safe The midpoint of the two-dimensional line segment connecting the first pixel and the second pixel is obtained using a binary search method. The second pixel is then updated to this midpoint, and the result is returned to the second summation unit. If |L b2 -|≤ε safe Proceed to the termination unit;

[0027] The termination unit is used to terminate the loop process and designate the first pixel or the second pixel as a safe distance edge point.

[0028] Preferably, the first summing unit is specifically used for:

[0029] N sampling points are obtained by uniformly sampling along the two-dimensional line between the dissection starting point and the first pixel point;

[0030] For sampling points whose planar coordinates are integers in the image pixel coordinate system, the spatial coordinates of these sampling points in the camera coordinate system are directly obtained according to the intraoperative tissue point cloud model. Otherwise, bilinear interpolation is performed to indirectly obtain the spatial coordinates of another part of the sampling points in the camera coordinate system.

[0031] The spatial curve is approximated as the sum of line segments between sampling points, and the true length L of the two-dimensional line segment between the anatomical starting point and the first pixel point is obtained and projected into three-dimensional space. b1 ;

[0032]

[0033] in, This represents the set of spatial coordinates of the anatomical starting point, the first pixel, and all sampling points on the two-dimensional line segment connected to it, in camera coordinates. These represent the spatial coordinates of the a-th and a-1-th points on the two-dimensional line segment b1 in the camera coordinate system, respectively.

[0034] Preferably, the bilinear interpolation operation specifically refers to:

[0035] Define the planar coordinates of sampling points whose pixel coordinates are not integers in the image pixel coordinate system as follows:

[0036] Select its four nearest integer pixels Based on the intraoperative tissue point cloud model, the corresponding depth values ​​were obtained as follows:

[0037] Perform the following operation to obtain the depth values ​​corresponding to sampling points whose pixel coordinates are not integers.

[0038]

[0039]

[0040] Similarly, obtain the sampling points whose pixel coordinates are not integers in the camera coordinate system. and

[0041] Preferably, the modeling module employs an online self-supervised learning depth estimation method based on binocular endoscope to obtain the depth value of the specified binocular endoscope image frame; the binocular depth estimation network used in the online self-supervised learning depth estimation method has the ability to quickly overlearn and can continuously adapt to new scenes using self-supervised information;

[0042] In real-time reconstruction mode, the modeling module is specifically used to overfit continuous video frames to obtain the depth value of a specified binocular endoscopic image frame, including:

[0043] The extraction unit is used to acquire binocular endoscope images and uses the encoder network of the current binocular depth estimation network to extract multi-scale features of the current frame image.

[0044] The fusion unit is used to fuse multi-scale features using the decoder network of the current binocular depth estimation network to obtain the disparity of each pixel in the current frame image;

[0045] The conversion unit is used to convert parallax into depth based on camera intrinsic and extrinsic parameters and output it as the result of the current frame image.

[0046] The first estimation unit is used to update the parameters of the current stereo depth estimation network using self-supervised loss without introducing external ground truth, for depth estimation of the next frame image.

[0047] Preferably, in the precise measurement mode, the modeling module is specifically used to overfit key image video frames, including:

[0048] The second estimation unit, without introducing external ground truth, uses the binocular depth estimation network obtained in real-time reconstruction mode based on the previous frame of the specified binocular endoscope image frame. It then updates the parameters of the aforementioned binocular depth estimation network using the self-supervised loss corresponding to the specified binocular endoscope image frame until convergence. The converged binocular depth estimation network is then used to accurately estimate the depth of the specified binocular endoscope image frame, thereby obtaining the depth value of the specified binocular endoscope image frame.

[0049] (III) Beneficial Effects

[0050] This invention provides an in vivo safety distance marking system for minimally invasive anatomical pathway constraints. Compared with existing technologies, it has the following advantages:

[0051] This invention is based on an intraoperative safety zone identification algorithm. Through a precisely reconstructed tissue point cloud model during surgery, it acquires true 3D information of the surgical site. It receives anatomical points, anatomical directions, and set safety distances provided by the surgeon, calculates the edge points of the safety distances, and displays the safety distance lines, indicating the direction and distance for the surgeon's operations. It also ensures accuracy in both direction and distance. This significantly reduces surgical errors that may arise from surgeons relying on their own experience, allowing for more accurate surgical procedures and improving the safety and success rate of the surgery. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A structural block diagram of an in vivo safety distance marking system for minimally invasive anatomical path constraints provided in this embodiment of the invention;

[0054] Figure 2 This is a schematic diagram of safety distance edge point calculation provided in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram illustrating the technical framework of an online self-supervised learning depth estimation method based on binocular endoscopy, provided in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] This application provides an in vivo safety distance marking system for minimally invasive anatomical path constraints, which solves the technical problem of not being able to correctly guide the direction during surgery while assisting doctors in quantitative distance control.

[0058] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0059] This invention is primarily applied to, but not limited to, surgical endoscopic scenarios such as laparoscopic surgery. Specifically, this invention is based on an intraoperatively assisted safety area identification algorithm. Through a precisely reconstructed tissue point cloud model during surgery, it acquires true three-dimensional information of the surgical site, receives anatomical points and directions provided by the surgeon, and sets safety distances. It calculates the edge points of the safety distances and displays the safety distance lines, indicating the direction and distance of the surgical site manipulation to the surgeon, while maintaining accuracy in both direction and distance. This significantly reduces surgical errors that may arise from surgeons relying on their own experience, allowing for more accurate surgical operations and improving the safety and success rate of the surgery.

[0060] Furthermore, this embodiment of the invention employs an online self-supervised learning depth estimation method based on binocular endoscopy to obtain the depth value of the specified binocular endoscopy image frame. The binocular depth estimation network used in the online self-supervised learning depth estimation method has the ability to quickly overlearn and can continuously adapt to new scenes using self-supervised information. Moreover, the online self-supervised learning depth estimation method also provides two modes: a real-time reconstruction mode and a precise measurement mode, for determining the depth value of the specified binocular endoscopy image frame.

[0061] The dual-mode switching depth estimation can provide real-time point clouds of intraoperative anatomical structures to help doctors intuitively understand the intraoperative three-dimensional structure. It can also achieve high-precision reconstruction of the binocular endoscopic image frames specified by the doctor based on single-frame overfitting, providing a basis for subsequent processing and balancing speed and accuracy in application.

[0062] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0063] Example:

[0064] like Figure 1 As shown, this embodiment of the invention provides an in vivo safety distance marking system for minimally invasive anatomical pathway constraints, comprising:

[0065] The modeling module is used to obtain intraoperative tissue point cloud models based on the depth values ​​of specified binocular endoscopic image frames;

[0066] The receiving module is used to receive the anatomical starting point, anatomical direction, and set safety distance marked by the doctor on the region of interest of the specified binocular endoscopic image frame;

[0067] The identification module is used to obtain the edge points of the safety distance based on the intraoperative tissue point cloud model, the anatomical starting point, the anatomical direction and the safety distance, identify the line connecting the anatomical starting point and the edge points of the safety distance, and use it as the safety distance line on the specified binocular endoscope image frame.

[0068] The tracking module is used to track the anatomical starting point and the safety distance edge point for each binocular endoscope image frame after the specified binocular endoscope image frame, and to display the safety distance line in real time on each binocular endoscope image frame.

[0069] This invention, based on an intraoperative safety zone identification algorithm, acquires true 3D information of the surgical site through a precisely reconstructed tissue point cloud model. It receives anatomical points, directions, and set safety distances provided by the surgeon, calculates the edge points of the safety distances, and displays the safety distance lines, indicating the direction and distance for the surgeon's operations. This approach ensures both precise direction and distance measurement. It significantly reduces surgical errors that might arise from surgeons relying on their own experience, allowing for more accurate surgical procedures and improving the safety and success rate of the surgery.

[0070] The following section will detail each component module of the above technical solution:

[0071] The modeling module is used to obtain intraoperative tissue point cloud models based on the depth values ​​of specified binocular endoscopic image frames.

[0072] For example, this module uses a depth estimation algorithm to estimate the depth value of a pixel on a specified binocular endoscope image frame, and calculates the spatial coordinates of the pixel in the camera coordinate system using a pinhole camera model. The formula is as follows:

[0073]

[0074]

[0075] z = D

[0076] Where D is the depth estimate of the pixel; x, y, and z represent the x, y, and z coordinates in the camera coordinate system, respectively; c x ,c y ,f x ,f y The intrinsic parameter matrix between the left or right eye endoscope and the camera in a binocular endoscope. The corresponding parameters;

[0077] The specified binocular endoscopic image frame is converted into an intraoperative tissue point cloud model V. in ={v in,a |a=1,2,…N1},v in,a This represents the spatial coordinates of the a-th pixel. The model has a total of N1 pixels.

[0078] The receiving module is used to receive the anatomical starting point, anatomical direction, and set safety distance marked by the doctor on the region of interest of the specified binocular endoscopic image frame.

[0079] This module is used to receive preoperative information, which can assist doctors in keeping their intraoperative operations within safe distance lines in subsequent modules.

[0080] For the identification module, such as Figure 2 As shown, it is used to obtain the edge points of the safety distance based on the intraoperative tissue point cloud model, the anatomical starting point, the anatomical direction and the safety distance, identify the line connecting the anatomical starting point and the edge points of the safety distance, and use it as the safety distance line on the specified binocular endoscopic image frame.

[0081] Specifically, the identification module includes a selection unit, a first summation unit, a first judgment unit, a second summation unit, a second judgment unit, and a termination unit. Wherein:

[0082] The selection unit is used to arbitrarily select a first pixel point that does not coincide with the anatomical starting point along the anatomical direction on the specified binocular endoscope image frame.

[0083] The first summing unit is used to uniformly sample along the two-dimensional line connecting the anatomical starting point and the first pixel to obtain a number of sampling points; and based on the intraoperative tissue point cloud model, to approximate the spatial curve as the sum of line segments between each sampling point, and to obtain the true length L of the two-dimensional line b1 between the anatomical starting point and the first pixel projected into three-dimensional space using an accumulation summation method. b1 .

[0084] The first summation unit is specifically used for:

[0085] N sampling points are obtained by uniformly sampling along the two-dimensional line between the dissection starting point and the first pixel point;

[0086] For sampling points whose planar coordinates are integers in the image pixel coordinate system, the spatial coordinates of these sampling points in the camera coordinate system are directly obtained according to the intraoperative tissue point cloud model. Otherwise, bilinear interpolation is performed to indirectly obtain the spatial coordinates of another part of the sampling points in the camera coordinate system.

[0087] Specifically, the bilinear interpolation operation refers to:

[0088] Define the planar coordinates of sampling points whose pixel coordinates are not integers in the image pixel coordinate system as follows:

[0089] Select its four nearest integer pixels Based on the intraoperative tissue point cloud model, the corresponding depth values ​​were obtained as follows:

[0090] Perform the following operation to obtain the depth values ​​corresponding to sampling points whose pixel coordinates are not integers.

[0091]

[0092] Similarly, obtain the sampling points whose pixel coordinates are not integers in the camera coordinate system. and

[0093] The spatial curve is approximated as the sum of line segments between sampling points, and the true length L of the two-dimensional line segment between the anatomical starting point and the first pixel point is obtained and projected into three-dimensional space. b1 ;

[0094]

[0095] in, This represents the set of spatial coordinates of the anatomical starting point, the first pixel, and all sampling points on the two-dimensional line segment connected to it, in camera coordinates. These represent the spatial coordinates of the a-th and a-1-th points on the two-dimensional line segment b1 in the camera coordinate system, respectively.

[0096] The first judgment unit is used to judge L. b1 The relationship between L and the safe distance, if L b1 >and|L b1 -|> safe The midpoint of the two-dimensional line segment connecting the dissection starting point and the first pixel point is obtained using the binary search method and denoted as the second pixel point. The process then proceeds to the second summation unit. If L... b1 <and|L b1 -|> safe Return to the selected unit; if |L b1 -|≤ε safe Then, proceed to the termination unit; where ε safe To set a threshold.

[0097] The second summation unit is used to uniformly sample along the two-dimensional line connecting the anatomical starting point and the second pixel to obtain several sampling points; and based on the intraoperative tissue point cloud model, to approximate the spatial curve as the sum of line segments between each sampling point, and to obtain the true length L of the two-dimensional line b2 between the anatomical starting point and the second pixel projected into three-dimensional space using an accumulation summation method. b2 .

[0098] It is not difficult to understand that in order to obtain L b2 The processing procedure performed by the second summing unit is similar to that of the first summing unit, and will not be described again here.

[0099] The second judgment unit is used to judge L. b2 The relationship between the magnitude of L and the safety distance L, if L b2>and|L b2 -|> safe The midpoint of the two-dimensional line segment connecting the dissection starting point and the second pixel is obtained using a binary search method. The second pixel is then updated to this midpoint, and the result is returned to the second summation unit. If L b2 <and|L b2 -|> safe The midpoint of the two-dimensional line segment connecting the first pixel and the second pixel is obtained using a binary search method. The second pixel is then updated to this midpoint, and the result is returned to the second summation unit. If |L b2 -|≤ε safe Proceed to the termination unit.

[0100] The termination unit is used to terminate the loop process and designate the first pixel or the second pixel as a safe distance edge point.

[0101] The identification module is used to obtain the edge points of the safety distance based on the intraoperative tissue point cloud model, the anatomical starting point, the anatomical direction and the safety distance, identify the line connecting the anatomical starting point and the edge points of the safety distance, and use it as the safety distance line on the specified binocular endoscopic image frame.

[0102] The tracking module is used to track the anatomical starting point and the safety distance edge point for each binocular endoscope image frame after the specified binocular endoscope image frame, and to display the safety distance line in real time on each binocular endoscope image frame.

[0103] For example, this module uses a weight-sharing neural network to extract feature descriptors from two adjacent binocular endoscope image frames. The feature descriptors are one-dimensional vectors used to describe the features of pixels.

[0104] In this method, the preceding binocular endoscopic image frame is defined as the source image, and the following binocular endoscopic image frame is defined as the target image. The feature descriptors of the anatomical starting point and the safe distance edge point in the source image are used as a one-dimensional convolution kernel, and a two-dimensional convolution is performed with the feature descriptors of the target image to generate a similarity distribution map. The position with the highest similarity is selected as the tracked pixel and displayed on the target image.

[0105] Furthermore, considering the characteristics of the flexible and dynamic environment within the body, and the influence of features such as indistinct internal environment characteristics and mirror-like areas, the embodiments of the present invention can accurately estimate depth information using intraoperative images, ensuring the accuracy of calculating the true length between the anatomical starting point and the edge point of the safe distance in complex and ever-changing environments.

[0106] As mentioned above, the modeling module uses an online self-supervised learning depth estimation method based on binocular endoscope to obtain the depth value of the specified binocular endoscope image frame; the binocular depth estimation network used by the online self-supervised learning depth estimation method has the ability to quickly overlearn and can continuously adapt to new scenes using self-supervised information;

[0107] In real-time reconstruction mode, the modeling module is specifically used to overfit continuous video frames to obtain the depth value of a specified binocular endoscopic image frame, including:

[0108] The extraction unit is used to acquire binocular endoscope images and uses the encoder network of the current binocular depth estimation network to extract multi-scale features of the current frame image.

[0109] The fusion unit is used to fuse multi-scale features using the decoder network of the current binocular depth estimation network to obtain the disparity of each pixel in the current frame image;

[0110] The conversion unit is used to convert parallax into depth based on camera intrinsic and extrinsic parameters and output it as the result of the current frame image.

[0111] The first estimation unit is used to update the parameters of the current stereo depth estimation network using self-supervised loss without introducing external ground truth, for depth estimation of the next frame image.

[0112] This depth estimation scheme utilizes the similarity of consecutive frames to extend the overfitting idea from a pair of binocular images to overfitting over time series. By continuously updating the model parameters through online learning, it can obtain high-precision tissue depth in various binocular endoscopic surgical environments.

[0113] The pre-training stage of the stereo depth estimation network abandons the traditional training mode and adopts the idea of ​​meta-learning, which allows the network to learn the depth of one image to predict the depth of another image, thereby calculating the loss and updating the network. This can effectively promote the network's generalization to new scenes and improve its robustness to low-texture complex lighting, while significantly reducing the time required for subsequent overfitting.

[0114] like Figure 3 As shown in section b, the initial model parameters corresponding to the stereo depth estimation network are obtained through meta-learning training, specifically including:

[0115] S100, Randomly select an even number of pairs of stereo images {e1,e2,…,e 2K} and equally divided into support sets and query set and Images are randomly paired to form K tasks

[0116] S200, Inner Circulation Training: Based on The loss is calculated from the support set image to perform a parameter update;

[0117]

[0118] in, This represents the network parameters after the inner loop update; Let α represent the derivative, where α is the learning rate of the inner loop. For the support set image of the k-th task, It is based on the initial parameters φ of the model m The calculated loss; f represents the stereo depth estimation network;

[0119] S300, External Circulation Training: Based on The query set image is used to calculate the meta-learning loss using the updated model, and the initial parameters φ of the model are directly updated. m For φ m+1 ;

[0120]

[0121] Where β is the learning rate of the outer loop; This is the query set image for the k-th task. This is the learning loss of the meta-learning.

[0122] The following is a detailed description of each unit included in the modeling module:

[0123] For extraction units, such as Figure 3 As shown in section a, it acquires binocular endoscopic images and uses the encoder network of the current binocular depth estimation network to extract multi-scale features of the current frame image.

[0124] For example, the encoder of the stereo depth estimation network in this unit uses a ResNet18 network to extract feature maps at five scales for the current frame image (left and right eyes).

[0125] For fusion units, such as Figure 3 As shown in section a, it employs the decoder network of the current binocular depth estimation network to fuse multi-scale features and obtain the disparity of each pixel in the current frame image; specifically, it includes:

[0126] The decoder network described above processes the coarse-scale feature map through convolutional blocks and upsampling, concatenates it with the fine-scale feature map, and then performs feature fusion through convolutional blocks again. The convolutional blocks are constructed by combining reflection padding, convolutional layers, and nonlinear activation units (ELUs).

[0127] Calculate the disparity directly based on the output with the highest network resolution:

[0128] d = k·((conv(Y))-H)

[0129] Where d represents the disparity estimate of a pixel; k is the preset maximum disparity range; Y is the highest resolution output; TH represents a parameter related to the type of binocular endoscope, which is 0.5 when the endoscope image has negative disparity and 0 when all endoscope images have positive disparity; conv is a convolutional layer; and sigmoid performs range normalization.

[0130] The conversion unit converts parallax into depth based on camera intrinsic and extrinsic parameters and outputs it as the result of the current frame image.

[0131] In this unit, converting parallax to depth means:

[0132]

[0133] Among them, c x1 , These are the intrinsic parameter matrices of the left and right eye endoscopes and cameras in a binocular endoscope. The corresponding parameter in; if f x Take the corresponding internal parameters of the left eye camera When f is the left-eye pixel, then d takes the disparity estimate of the left-eye pixel, and D is the depth estimate of the left-eye pixel; if f x Take the corresponding internal parameters of the right eye camera Then d is the disparity estimate of the right eye pixel, and D is the depth estimate of the right eye pixel; b is the baseline length, i.e. the extrinsic parameter of the binocular camera.

[0134] For the first estimation module, such as Figure 3 As shown in section b, it uses self-supervised loss to update the parameters of the current stereo depth estimation network without introducing external ground truth, for depth estimation of the next frame image.

[0135] It is easy to understand that the "external truth value" mentioned in the embodiments of the present invention is the label (or "supervision information"), which is a well-known expression in the art.

[0136] In this unit, such as Figure 3 As shown in part b, the self-supervised loss is expressed as:

[0137]

[0138] Among them, L self The value represents the self-supervised loss; α1, α2, α3, and α4 are all hyperparameters, l corresponds to the left figure, and r corresponds to the right figure.

[0139] Since both eyes are observing the same scene, the values ​​of corresponding pixels on the left and right depth maps should be equal when transformed to the same coordinate system. Therefore, we introduce... and

[0140] (1) The geometric consistency loss is represented by the left figure:

[0141]

[0142] Wherein, P1 represents the first set of valid pixels (i.e., the valid pixels of the right eye); The effective pixel p represents the left-eye depth obtained from the right-eye depth map after camera pose transformation, and D represents the left-eye depth. l ′(p) represents the effective pixel point p using the predicted right-side disparity Dis. R The left eye depth is obtained by sampling on the left eye depth map.

[0143] (2) The geometric consistency loss is shown in the right figure:

[0144]

[0145] Wherein, P2 represents the second set of valid pixels (i.e., the valid pixels of the left eye); The effective pixel p represents the right-eye depth obtained from the left-eye depth map after camera pose transformation, and D represents the right-eye depth. r ′(p) represents the effective pixel point p using the predicted left image disparity Dis. L The right eye depth is obtained by sampling on the right eye depth map.

[0146] By incorporating geometric consistency constraints into the training loss, the network's general applicability to hardware is ensured, enabling it to autonomously adapt to unconventional binocular images such as surgical endoscopes.

[0147] Assuming constant brightness and spatial smoothness during endoscopic surgery, reprojection between left and right eye images can achieve reconstruction of another objective. However, this introduces structural similarity loss. The brightness, contrast, and structure of the two images are normalized and compared, and then... and

[0148] (3) The left image shows the luminous loss:

[0149]

[0150] Among them, I L (p) represents the left figure, I L ′(p) represents the parallax between the right image and the predicted left image Dis.L (p) Reconstructed image from the left eye endoscope, λ i and λ s To balance the parameters, SSIM LL′ (p) represents I L (p) and I L Image structural similarity of ′(p);

[0151] (4) The image on the right shows the luminous loss:

[0152]

[0153] Among them, I R (p) represents the right figure, I′ R (p) indicates the use of the disparity between the left image and the predicted right image. R (p) Generated reconstructed image from the right eye endoscope, SSIM RR′ (p) represents I R (p) and I′ R Image structural similarity (p).

[0154] In low-texture and monochromatic organizational regions, smoothing priors are used to aid inference, and depth regularization is applied, introducing... and

[0155] (5) The smoothing loss is shown in the left graph:

[0156]

[0157] in, This represents the normalized left eye depth map. and This represents the first derivative along the horizontal and vertical directions of the image;

[0158] (6) The smoothing loss is shown in the right figure:

[0159]

[0160] in, This represents the normalized depth map of the right eye. and This represents the first derivative along the horizontal and vertical directions of the image.

[0161] Specifically, the process of obtaining the first set of valid pixels P1 and the second set of valid pixels P2 is as follows:

[0162] Define the left eye disparity predicted by the current binocular depth estimation network as: Right eye parallax is The formulaic expression for the cross-validation mask for the left and right eyes is as follows:

[0163]

[0164]

[0165] in, These are used to determine whether the pixel at position (,j) in the left and right eye images is within the stereo matching range; i takes the value of any integer between [1,W]; j takes the value of any integer between [1,H]; W represents the image width, and H represents the image height;

[0166] Let c take the value L or R, when If the value is within the stereo matching range under the current calculation method, it means that the pixel at position (j) is within the stereo matching range; otherwise, it is not within the stereo matching range.

[0167] By using a pinhole camera model, binocular pose transformation, and predicted depth for projection, an effective region mask based on 3D points is obtained. Take 0 or 1, when If the value is within the stereo matching range under the current calculation method, it means that the pixel at position (j) is within the stereo matching range; otherwise, it is not within the stereo matching range.

[0168] Obtain the final valid region mask

[0169]

[0170] If pixel p satisfies When c is R, the first set of valid pixels P1 is obtained; when c is L, the second set of valid pixels P2 is obtained.

[0171] In the corrected stereo image, additional regions caused by viewpoint shift cannot find matching pixels. However, this embodiment of the invention considers that low texture and uneven illumination of in vivo tissues can make local features less obvious, and pixels in these invalid regions often find similar pixels in neighboring regions. Therefore, as mentioned above, this embodiment of the invention proposes a cross-validation-based binocular effective region recognition algorithm, which eliminates the misleading effect of self-supervised loss of invalid region pixels on network learning and improves the accuracy of depth estimation.

[0172] In addition, to avoid insufficient robustness of depth estimation in pure texture or low-light scenes, a method is also introduced.

[0173] (7) Indicates the loss of sparse optical flow:

[0174]

[0175] wherein, Dis L (p) represents the predicted left-view disparity map, OF L (p) represents the left-view sparse disparity map, Dis R (p) represents the predicted right-view disparity map, OF R (p) represents the right-view sparse disparity map; P3 represents the third valid pixel set in the left-view sparse disparity map OF L (p); P4 represents the fourth valid pixel set in the right-view sparse disparity map OF R (p); γ1 and γ2 are balancing parameters, both are non-negative numbers and do not take 0 at the same time.

[0176] Specifically, the acquisition process of the third valid pixel set P3 and the fourth valid pixel set P4 is as follows:

[0177] By using the LK (Lucas-Kanade) optical flow solving algorithm, sparse optical flow (Δx, Δy) is calculated every n pixels in the row and column directions, wherein Δx represents the horizontal offset of the pixel, and Δy represents the vertical offset of the pixel;

[0178] When solving the optical flow from the left image to the right image, only when and Δx>thd1, the disparity at this pixel position is retained as Δx, wherein KT and thd1 are corresponding preset thresholds, the disparity at positions that do not meet the above conditions or where sparse optical flow is not calculated is set to 0, so as to obtain the final sparse disparity map OF L (), pixels with OF L (p)≠0 constitute the third valid pixel set P3;

[0179] When solving the optical flow from the right image to the left image, only when and Δx<thd2, the disparity at this pixel position is retained as Δx, wherein thd2 is a corresponding preset threshold, the disparity at positions that do not meet the above conditions or where sparse optical flow is not calculated is set to 0, so as to obtain the final sparse disparity map OF R (), pixels with OF R (p)≠0 constitute the fourth valid pixel set P4.

[0180] As mentioned above, the embodiment of the present invention introduces the traditional Lucas-Kanade optical flow to derive the sparse disparity between binocular images, provides the network with a reasonable learning direction, improves the rapid learning ability and reduces the probability of falling into local optimum.

[0181] It is particularly emphasized that, in addition to the real-time reconstruction mode, the online self-supervised learning depth estimation method adopted by the modeling module in the embodiment of the present invention is further provided with a precision measurement mode. As Figure 3As shown in section b, in the precise measurement mode, the modeling module is specifically used to overfit key image video frames, including:

[0182] The second estimation unit, without introducing external ground truth, uses the binocular depth estimation network obtained in real-time reconstruction mode based on the previous frame of the specified binocular endoscope image frame. It then updates the parameters of the aforementioned binocular depth estimation network using the self-supervised loss corresponding to the specified binocular endoscope image frame until convergence. The converged binocular depth estimation network is then used to accurately estimate the depth of the specified binocular endoscope image frame, thereby obtaining the depth value of the specified binocular endoscope image frame.

[0183] It is worth noting that the technical details of the depth estimation network, self-supervised loss function, effective region mask calculation, and meta-learning pre-training method in the accurate measurement mode are consistent with the technical details extended in the real-time reconstruction mode, and will not be repeated here.

[0184] In summary, compared with existing technologies, it has the following beneficial effects:

[0185] 1. This invention, based on an intraoperative safety zone identification algorithm, acquires true 3D information of the surgical site through a precisely reconstructed tissue point cloud model during surgery. It receives anatomical points, anatomical directions, and set safety distances provided by the surgeon, calculates the edge points of the safety distances, and displays the safety distance lines, indicating the direction and distance of the surgical site manipulation to the surgeon. It also ensures accuracy in both direction and distance. This significantly reduces surgical errors that may arise from surgeons relying on their own experience, allowing for more accurate surgical procedures and improving the safety and success rate of the surgery.

[0186] 2. This invention provides an online self-supervised learning depth estimation method based on binocular endoscopy, the beneficial effects of which include at least:

[0187] 2.1 The switching depth estimation can provide real-time point cloud of intraoperative anatomical structure to help doctors intuitively understand the intraoperative three-dimensional structure. It can also realize high-precision reconstruction of key frames selected by doctors based on single-frame overfitting, providing a basis for subsequent measurement, so that speed and accuracy can be balanced in application.

[0188] 2.2 By utilizing the similarity of consecutive frames, the overfitting concept on a pair of binocular images is extended to overfitting on time series. Through online learning, the model parameters are continuously updated, enabling high-precision tissue depth measurements to be obtained in various binocular endoscopic surgical environments.

[0189] 2.3 The pre-training stage of the network model abandons the traditional training mode and adopts the idea of ​​meta-learning, which allows the network to learn the depth of one image to predict the depth of another image, thereby calculating the loss and updating the network. This can effectively promote the generalization of the network to new scenes and improve its robustness to low-texture complex lighting, while significantly reducing the time required for subsequent overfitting.

[0190] 2.4. By incorporating geometric consistency constraints into the training loss, the network's general applicability to hardware is ensured, enabling autonomous adaptation to unconventional binocular images such as surgical endoscopes.

[0191] 2.5. Depth estimation of each frame of stereo image is treated as an independent task, and high-precision models suitable for the current frame are obtained through real-time overfitting; and new scenes can be learned quickly through online learning to obtain high-precision depth estimation results.

[0192] 2.6 The cross-validation binocular effective region recognition algorithm eliminates the misleading effect of the self-supervised loss of invalid region pixels on network learning and improves the accuracy of depth estimation.

[0193] 2.7. Introducing the traditional Lucas-Kanade optical flow to derive the sparse parallax between binocular images provides the network with a reasonable learning direction, improves its rapid learning ability, and reduces the probability of getting trapped in local optima.

[0194] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0195] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An in vivo safety distance marking system for minimally invasive anatomical pathway constraints, characterized in that, include: The modeling module is used to obtain intraoperative tissue point cloud models based on the depth values ​​of specified binocular endoscopic image frames; The receiving module is used to receive the anatomical starting point, anatomical direction, and set safety distance marked by the doctor on the region of interest of the specified binocular endoscopic image frame; The identification module is used to obtain the edge points of the safety distance based on the intraoperative tissue point cloud model, the anatomical starting point, the anatomical direction and the safety distance, identify the line connecting the anatomical starting point and the edge points of the safety distance, and use it as the safety distance line on the specified binocular endoscope image frame. The tracking module is used to track the anatomical starting point and the safety distance edge point for each binocular endoscope image frame after the specified binocular endoscope image frame, and to display the safety distance line on each binocular endoscope image frame in real time. The modeling module employs an online self-supervised learning depth estimation method based on binocular endoscopes to obtain the depth values ​​of the specified binocular endoscope image frames. The binocular depth estimation network used in the online self-supervised learning depth estimation method has the ability to quickly overlearn and can continuously adapt to new scenes using self-supervised information. In real-time reconstruction mode, the modeling module is specifically used to overfit continuous video frames to obtain the depth value of a specified binocular endoscopic image frame, including: The extraction unit is used to acquire binocular endoscope images and uses the encoder network of the current binocular depth estimation network to extract multi-scale features of the current frame image. The fusion unit is used to fuse multi-scale features using the decoder network of the current binocular depth estimation network to obtain the disparity of each pixel in the current frame image; The conversion unit is used to convert parallax into depth based on camera intrinsic and extrinsic parameters and output it as the result of the current frame image. The first estimation unit is used to update the parameters of the current stereo depth estimation network using self-supervised loss without introducing external ground truth, for depth estimation of the next frame image. In precise measurement mode, the modeling module is specifically used to overfit key image video frames, including: The second estimation unit, without introducing external ground truth, uses the binocular depth estimation network obtained in real-time reconstruction mode based on the previous frame of the specified binocular endoscope image frame. It then updates the parameters of the aforementioned binocular depth estimation network using the self-supervised loss corresponding to the specified binocular endoscope image frame until convergence. The converged binocular depth estimation network is then used to accurately estimate the depth of the specified binocular endoscope image frame, thereby obtaining the depth value of the specified binocular endoscope image frame.

2. The in-vivo safety distance marking system as described in claim 1, wherein the modeling module is used for: The formula for calculating the spatial coordinates of a pixel in the camera coordinate system using a pinhole camera model is as follows: in, This is the depth estimate of a pixel; , , Representing the camera coordinate system coordinate 、 coordinate 、 coordinate; The intrinsic parameter matrix between the left or right eye endoscope and the camera in a binocular endoscope. The corresponding parameters; The specified binocular endoscopic image frame is converted into an intraoperative tissue point cloud model. , Indicates the first The model has the spatial coordinates of pixels, and a total of pixels. Each pixel.

3. The in-body safe distance marking system as described in claim 1, characterized in that, The identification module includes: The selection unit is used to arbitrarily select a first pixel point that does not coincide with the anatomical starting point along the anatomical direction on the specified binocular endoscope image frame. The first summation unit is used to uniformly sample along the two-dimensional line connecting the anatomical starting point and the first pixel to obtain a number of sampling points; and based on the intraoperative tissue point cloud model, to approximate the spatial curve as the sum of line segments between each sampling point, and to obtain the two-dimensional line connecting the anatomical starting point and the first pixel using an accumulation summation method. The true length projected into three-dimensional space ; The first judgment unit is used to make judgments. and safe distance The size relationship, if and The midpoint of the two-dimensional line segment connecting the dissection starting point and the first pixel point is obtained using the binary search method and denoted as the second pixel point. The process then proceeds to the second summation unit. and Return to the selected unit; if Proceed to the termination unit; where, To set a threshold; The second summation unit is used to uniformly sample along the two-dimensional line connecting the anatomical starting point and the second pixel point to obtain a number of sampling points; and based on the intraoperative tissue point cloud model, to approximate the spatial curve as the sum of line segments between each sampling point, and to obtain the two-dimensional line connecting the anatomical starting point and the second pixel point using an accumulation summation method.

2. The actual length projected into three-dimensional space ; The second judgment unit is used to judge. and safe distance The size relationship, if and The midpoint of the two-dimensional line segment connecting the dissection starting point and the second pixel is obtained using a binary search method. The second pixel is then updated to this midpoint, and the result is returned to the second summation unit. and The midpoint of the two-dimensional line segment connecting the first pixel and the second pixel is obtained using a binary search method. The second pixel is then updated to this midpoint, and the result is returned to the second summation unit. Proceed to the termination unit; The termination unit is used to terminate the loop process and designate the first pixel or the second pixel as a safe distance edge point.

4. The in-body safe distance marking system as described in claim 3, characterized in that, The first summation unit is specifically used for: Uniform sampling is performed along the two-dimensional line connecting the dissection starting point and the first pixel to obtain... Sampling points; For sampling points whose planar coordinates are integers in the image pixel coordinate system, the spatial coordinates of these sampling points in the camera coordinate system are directly obtained according to the intraoperative tissue point cloud model. Otherwise, bilinear interpolation is performed to indirectly obtain the spatial coordinates of another part of the sampling points in the camera coordinate system. The spatial curve is approximated as the sum of line segments between sampling points, and the true length of the two-dimensional line segment between the anatomical starting point and the first pixel point is obtained and projected into three-dimensional space. ; in, This represents the set of spatial coordinates of the anatomical starting point, the first pixel, and all sampling points on the two-dimensional line segment connected to it, in camera coordinates. , Representing two-dimensional line segments respectively The first , The spatial coordinates of a point in camera coordinates.

5. The in-body safe distance marking system as described in claim 4, characterized in that, The bilinear interpolation operation specifically refers to: Define the planar coordinates of sampling points whose pixel coordinates are not integers in the image pixel coordinate system as follows: ; Select its four nearest integer pixels , , , Based on the intraoperative tissue point cloud model, the corresponding depth values ​​were obtained as follows: , , , ; Perform the following operation to obtain the depth values ​​corresponding to sampling points whose pixel coordinates are not integers. ; Similarly, obtain the sampling points whose pixel coordinates are not integers in the camera coordinate system. and .

Images