A binocular camera-based graph convolution enhanced real-time tracking method for abdominal organs

Abstract

The application discloses a kind of based on binocular camera's graph convolution enhancement abdominal organ real-time tracking method, comprising the following steps: collecting data and being marked, preoperative model is constructed and is trained, reconstructs three-dimensional model and is optimized, adopts co-rotating finite element modeling, constructs biomechanical model, carries out preoperative initial registration, eliminates the change between preoperative and intraoperative imaging of patient, extracts a variety of abdominal organ features and matches feature points, carries out intraoperative deformation registration and dynamically tracks feature points, and verifies experimental results;The application efficiently solves the problem of complex deformation of soft tissue, accurately selects key feature points through deep learning algorithm, and creates displacement vector field, provides boundary conditions for biomechanical model in time, can effectively simulate various complex deformations, and can adaptively track the dynamic changes of soft tissue during operation, significantly improve the capture and modeling ability of complex deformation behavior of soft tissue.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and intraoperative navigation, in particular to a graph convolution enhanced abdominal organ real-time tracking method based on a binocular camera. BACKGROUND

[0002] In the modern medical field, image-guided surgery is a key technology that can improve the accuracy and safety of surgery using advanced imaging techniques. Preoperative imaging techniques such as computed tomography or magnetic resonance imaging can provide detailed information about the internal structure of abdominal organs, including the spatial location of tumors and blood vessels. This information can help surgeons understand the complexity of the surgical area and predict possible complications, so it is important for the planning and execution of surgery. However, during surgery, abdominal organs and other soft tissues may undergo significant deformation due to surgical operations, organ displacement, cutting and suturing, etc. However, these preoperative images cannot reflect the changes in organ tissue in real time during surgery, increasing the risk and complexity of surgery. Therefore, they cannot be directly used for preoperative navigation. Therefore, a technology is needed that can correct this deformation in real time and update the preoperative image to ensure the accuracy of surgical navigation.

[0003] Currently, there are three main methods to accurately align preoperative organ models with intraoperative anatomical structures: model-based methods, data-driven methods, and hybrid methods.

[0004] Among them, the model-based method mainly relies on physical laws and biological principles to simulate organ deformation, including biomechanical models, physics-based modeling methods, and geometric alignment methods. Yang et al. proposed a finite element model that does not require pre-defined boundary conditions. This model simulates the elastic deformation of organs by solving partial differential equations related to soft tissue deformation. This method does not rely on pre-defined zero displacement and force positions, reducing the difficulty of obtaining boundary conditions during surgery. Experiments show that this method achieves a mean TRE of 3.14±1.34mm. Suwelack et al. compared the non-rigid deformation problem to an electrostatic-elastic scenario, where the preoperative liver model is treated as a charged object interacting with the opposite charge on the intraoperative liver surface. The electrostatic force promotes the alignment of the preoperative liver model with the intraoperative surface, while the elastic force is used for regularization. Experiments show that this method has good alignment results and can accurately simulate the deformation of the liver during surgery to some extent.

[0005] Data-driven methods mainly utilize deep learning techniques to recover the true deformation of organs from a large amount of data, including Pfeiffer et al. using deep learning methods to predict the entire displacement field, reconstructing the deformed part of the surface directly from intraoperative laparoscopic video streams, which uses a 3D convolutional network to recover the displacement field from the digitized organ surface during surgery, and then uses it to infer the node displacement of the entire organ. Experiments show that this method achieves a mean TRE of 5.70 mm, demonstrating its accuracy in intraoperative liver deformation prediction;

[0006] Hybrid methods combine the advantages of model-based and data-driven methods, including neural network-driven biomechanical models, physically guided deep learning, and multi-stage deformation modeling methods, including Tagliabue et al. using BANet network to estimate boundary conditions from intraoperative point cloud data, and guiding the deformation of preoperative biomechanical organ models to accurately represent intraoperative organ deformation. This method combines biomechanical models and deep learning methods to estimate boundary conditions through neural networks to drive biomechanical models;

[0007] However, the above three methods still have the following many shortcomings:

[0008] High complexity of tissue deformation: Soft tissues such as the liver may undergo complex deformation during surgery, which may exceed the scope of simple rigid transformation, covering compression, stretching, shear, and other modes, and the degree of deformation may be quite significant, such as liver deformation during laparoscopic surgery may exceed 10 mm. This significant and complex deformation poses great challenges to mathematical modeling and machine learning, making it difficult to build accurate and universal deformation prediction learning models;

[0009] It is difficult to obtain boundary conditions: To build an accurate biomechanical model, accurate boundary conditions such as displacement or force are needed to simulate organ deformation. In actual surgery, there is a lack of effective force sensing and displacement tracking means, making it difficult to accurately measure the mechanical interaction during surgical operations in real time, resulting in missing or inaccurate model input data, which will affect the authenticity and reliability of the model simulation;

[0010] Low computational efficiency, difficult to meet the real-time processing requirements in surgery: Biomechanical models usually need to solve large, sparse linear equations, and traditional numerical calculation methods are inefficient. In addition, in the surgical environment, the registration process requires real-time or near real-time response capability to ensure that the surgeon can obtain accurate navigation information in a timely manner. This requires the registration algorithm to not only be accurate but also efficient to meet the real-time processing requirements. Directly solving the finite element-based biomechanical model takes a long time, making it difficult to achieve real-time or near real-time feedback, and unable to meet the stringent requirements of computational efficiency in surgical scenarios, limiting its application and promotion in clinical practice;

[0011] Insufficient accuracy of organ surface feature extraction: Existing feature extraction methods cannot accurately capture the subtle features of the organ surface, such as the corner points or edges of the liver, which makes it difficult to accurately track and match the surface during surgery. At the same time, existing feature extraction cannot accurately capture key points in the image, such as the branch points of blood vessels, which makes it difficult for the model to understand the spatial relationship between points, affecting the accuracy and reliability of organ deformation modeling. SUMMARY

[0012] The purpose of the present application is to provide a graph convolution enhanced abdominal organ real-time tracking method based on a binocular camera to solve the problems of high complexity of tissue deformation, difficulty in obtaining boundary conditions, low computational efficiency, difficulty in meeting real-time processing requirements during surgery, and insufficient accuracy of organ surface feature extraction as proposed in the background art.

[0013] To this end, the present application provides a graph convolution enhanced abdominal organ real-time tracking method based on a binocular camera, comprising the following steps:

[0014] A graph convolution enhanced abdominal organ real-time tracking method based on a binocular camera, comprising the following steps:

[0015] S1: Obtain the preoperative abdominal CT scan image of the patient and the laparoscopic intraoperative image data collected by the binocular camera, and label the target abdominal organ in the preoperative CT image and the intraoperative laparoscopic image, respectively;

[0016] S2: Based on the preoperative CT image, a three-dimensional surface model of the abdominal organ is constructed by a 3D organ segmentation network, and a biomechanical model of the abdominal organ is established by a co-rotation finite element modeling method;

[0017] S3: According to the intraoperative laparoscopic image, the three-dimensional point cloud of the abdominal organ in the initial frame is extracted, and it is rigidly registered with the preoperative three-dimensional surface model to estimate the rotation and translation matrix;

[0018] S4: For the three-dimensional point cloud of the initial frame after rigid registration, the feature point set of the organ is extracted, and the deformation field of the feature point cloud is estimated in combination with the elastic constraint of the biomechanical model to realize elastic registration to compensate for the non-rigid deformation of the tissue, so as to capture the actual shape and displacement of the organ;

[0019] S5: Based on the feature point set of the initial frame, the feature points in the subsequent frames are matched with the initial frame to realize dynamic tracking of the abdominal organ during the entire surgery;

[0020] S6: Quantitative and qualitative evaluation is performed on the dynamic tracking results to verify the accuracy and robustness of the abdominal organ real-time tracking method.

[0021] Preferably, S1 specifically comprises:

[0022] S11: Collect CT image data, label abdominal organ contours using Mimics software, obtain 3D segmentation mask corresponding to CT image, and provide required data set for subsequent 3D organ segmentation network training;

[0023] S12: Collect laparoscope video collected by intraoperative binocular camera, label abdominal organ region by Label-me software, obtain time sequence segmentation mask corresponding to video image, and provide required data set for subsequent intraoperative video organ registration network training.

[0024] Preferably, the S2 specifically comprises:

[0025] S21: Preprocessing the obtained preoperative CT image, the preprocessing including image data size 256*256*256 image block cropping and normalization processing;

[0026] S22: Constructing 3D organ segmentation network of preoperative image based on nnUNet framework, extracting mask of abdominal organ region from preprocessed preoperative CT image, using Dice loss function for network training, AdamW for optimizer, initial value of learning rate set to 0.001, iteration number set to 10000-15000 times, using data augmentation strategy including random scaling, Gaussian noise, contrast enhancement, Gaussian noise and random mirror flipping for CT image data;

[0027] S23: Inputting the three-dimensional mask of the abdominal organ into Marching Cubes algorithm, first preprocessing the original data and reading into the specified array, second extracting a unit body and obtaining the values and coordinate positions of its 8 vertices, then comparing the function values of the 8 vertices of the unit body with the given isosurface value to obtain the state table of the unit body, then finding out the unit body edges intersecting with the isosurface according to the state table index, and using linear interpolation to calculate the intersection position, then using central difference method to obtain the normal vector of the 8 vertices of the unit body, using linear interpolation to obtain the normal vector of each vertex of the triangular facet, and finally drawing the isosurface image according to the coordinates and normal vectors of the triangular facet vertices, and reconstructing a three-dimensional surface model composed of 5000 triangular facets for expressing the organ structure;

[0028] S24: Optimizing the organ surface model, using quadratic error measurement algorithm to reduce the number of triangular facets to 1000, and improving the uniformity of the surface model by Laplacian smoothing and regridding processing;

[0029] S25: Combining the optimized surface model, using co-rotational finite element modeling method to construct biomechanical model.

[0030] Preferably, the S25 specifically comprises:

[0031] S251: using a quasi-static integral scheme, assuming that the inertial force component is small compared to the elastic component, regarding the soft tissue as an elastic body, and based on Newton's second law, focusing on static equilibrium, ignoring the inertial force, resulting in an overall biomechanical model where M is the mass, f represents the sum of internal and external forces corresponding to the deformed state and ;

[0032] S252: selecting a tetrahedron as the basic unit of the organ biomechanical model, for each tetrahedral element e, constructing its deformation equation as where is the displacement vector of the vertex, is the force acting on the element, is the stiffness matrix of the element, expressed as where V is the total volume, is the volume of an element, is the strain-displacement matrix, is the stress-strain matrix, denotes the rotation matrix of the current state of the tetrahedron relative to the initial state; the stress-strain matrix is then derived from the Young's modulus and Poisson's ratio where , and ν represent the Young's modulus and Poisson's ratio, respectively.

[0033] Preferably, the S3 specifically comprises:

[0034] S31: Before the start of the operation, use the binocular camera to take multiple angle calibration board shots, use the OpenCV function cv: findChessboardCorners to detect the checkerboard corner points in each image, then for each detected corner point, extract its three-dimensional position in the world coordinate system, to establish the mapping relationship between two-dimensional image coordinates and three-dimensional world coordinates; use the cv: calibrateCamera function, pass all the collected three-dimensional object points and corresponding two-dimensional image points to the function to calculate the internal parameters and distortion coefficients of the camera; then, perform distortion correction, use the cv: undistort function to correct the distortion of the original image according to the camera intrinsic matrix and distortion coefficients obtained in the previous step; finally, evaluate the calibration by calculating the reprojection error to evaluate the accuracy of the calibration, complete the general calibration of the binocular camera;

[0035] S32: Based on the initial frame of the intraoperative endoscope image, a semi-global block matching algorithm is used to match the binocular left and right images. First, the horizontal Sobel operator is used to perform edge detection on the left and right images captured by the calibrated binocular camera to obtain gradient images. The matching cost is calculated. For each pixel, the matching cost with the corresponding pixel under different disparities is calculated. The absolute difference is used as the cost function. The cost is composed of two parts: the BT cost (Birchfield-Tomasi cost) calculated from the preprocessed gradient image and the BT cost calculated from the original image without preprocessing. Then, the two are added together to minimize the energy function. The dynamic programming method is used to calculate the cumulative cost along the 8 directions, and the minimum value is taken as the final cost. The disparity map is generated, and the post-processing method is used to repair the abnormal values or holes in the disparity map. After median filtering, the initial point cloud of three-dimensional reconstruction is obtained;

[0036] S33: The depth information of each pixel point in the disparity map is converted into three-dimensional space coordinates to generate the three-dimensional point cloud of the abdominal organ corresponding to the initial frame;

[0037] S34: A density adaptive registration algorithm based on Dirichlet process Gaussian mixture model is used to solve the registration problem using a variational Bayesian inference framework. The point cloud data and the model are expanded in a coarse-to-fine manner to improve the accuracy and efficiency of registration. The scanning weight of each point is estimated to obtain the initial rotation matrix and translation vector , thereby realizing adaptive rigid registration between the preoperative three-dimensional surface model and the intraoperative three-dimensional point cloud.

[0038] Preferably, the S4 specifically comprises:

[0039] S41: Based on the intraoperative endoscope image, the SiamMask target tracking algorithm is used to delineate the target region. The binocular lens video data collected in the S1 step and the abdominal organ region annotation fine-tuning model weight are used. The preselected initial bounding box is used as input to quickly extract the mask of the abdominal organ region in the intraoperative image.

[0040] S42: A feature point extraction network is constructed to extract features from the abdominal organ region to obtain feature point set coordinates and description vectors.

[0041] S43: The tetrahedral vertices of the biomechanical model in S2 are used as control points for non-rigid matching of the feature point set to realize modeling and correction of organ elastic deformation.

[0042] Preferably, the feature point extraction network specifically comprises:

[0043] A graph convolutional neural network with self-attention mechanism (SA-GCNPoint) is constructed for efficiently extracting feature points of abdominal organs; the overall architecture and training method of the network refer to the SuperPoint network; a single shared encoder is included for processing and reducing the dimensionality of the input image, and the backend is decoupled into two decoder "heads" that perform interest point detection and interest point description, respectively; the encoder is improved and optimized, and the original VGG encoder is replaced with a graph convolutional network with self-attention mechanism (i.e., 4 groups of Transformer and 4 groups of graph convolutional network), which extracts features of abdominal organs, respectively, while the configuration of the decoder remains unchanged; by fusing the self-attention mechanism with the graph convolution, the encoder can capture the complex relationships between all nodes in the organ graph at once, obtain the global context, and efficiently solve the long-distance dependency problem, thereby accurately modeling the interaction relationship between organs in a large receptive field; the network uses a self-supervised learning method to automatically learn the feature points of abdominal organs without manual annotation of data;

[0044] Preferably, the non-rigid matching specifically includes:

[0045] A biomechanics control point set Each control point is matched according to the minimum Hamming distance and a feature point set to determine the corresponding points , and displacement constraints are established between each pair of matching points to construct the following cost function: ; wherein, is the stiffness coefficient of the organ model, which is of the same order of magnitude as the Young's modulus, is the number of control points, are the initial rotation matrix and translation vector obtained in S3, respectively, is a global stiffness matrix obtained based on the element stiffness matrix group set; by globally minimizing the cost function, the optimal displacement vector is solved to accurately estimate and match the non-rigid deformation field.

[0046] Preferably, S5 specifically includes:

[0047] S51: Using the target tracking algorithm and the feature point extraction network described in S4, extract feature points from each subsequent frame in the same way, denoted as set ;

[0048] S52: According to the coordinates and description vectors of each feature point, calculate the current frame feature point set and the feature points The Hamming distance between corresponding description vectors is obtained through nearest neighbor matching. For initial point pairs ;

[0049] S53: Implement the Lucas Kanade optical flow algorithm using the `cv.calcOpticalFlowPyrLK` function in OpenCV to estimate the displacement of each matched feature point pair. If the displacement is greater than the mean plus three times the standard deviation, it is considered an anomalous optical flow and removed. For the feature point set;

[0050] S54: The remaining... For each unmatched feature point, the non-rigid matching method described in S43 is used to match it with the organ biomechanical model, and the following cost function is constructed and solved. ,in The final cross-frame feature point displacement estimate is obtained through optimization, enabling dynamic tracking of abdominal organs.

[0051] Preferably, the optimization solution process employs Newton's acceleration method, which accelerates convergence by dynamically adjusting the step size and momentum. Specifically, it includes the following steps: selecting an initial point. And set the learning rate In each iteration The following steps are executed sequentially: Calculate the predicted location. Calculate the objective function in gradient at and update momentum Finally, based on the updated parameters ; until the convergence condition is met, thus accelerating the solution process for displacement estimation;

[0052] Preferably, S6 specifically includes:

[0053] Registration errors were evaluated on two different open-source clinical liver datasets, 3Dircadb (as shown in Table 1) and DePoLL (as shown in Table 2). Root mean square error (RMSE) measures the average Euclidean distance between corresponding point sets after registration, and is more sensitive to larger errors, reflecting the overall registration accuracy. Mean absolute error (MAE) calculates the average absolute value of the distance between corresponding points, and is less affected by outliers, reflecting the stability and robustness of the registration results. Maximum error (Error) is the maximum distance among all corresponding point pairs, directly reflecting the registration deviation in the worst case, and is a key indicator for evaluating the robustness of the algorithm. The effectiveness of the proposed algorithm was evaluated by calculating the root mean square error, mean absolute error, and maximum error between the point cloud and the 3D model. Experiments compared six methods: ICP, DCP, RPM-Net, ROPNet, PREDATOR, and UTOPIC.

[0054] The beneficial effects of the two-eye camera-based graph convolution enhanced abdominal organ real-time tracking method are:

[0055] The present application solves the problem of complex deformation of soft tissue efficiently based on the feature extraction of deep learning and the biomechanical model based on the finite element method. In the intraoperative registration stage, the deformation of the soft tissue needs to be tracked in real time. The key feature points are accurately selected through the deep learning algorithm, and the displacement vector field is created. These feature points provide boundary conditions for the biomechanical model in time, which can effectively simulate complex deformations including compression, stretching and shearing, and can adaptively track the dynamic changes of soft tissue in surgery, significantly improving the ability to capture and model the complex deformation behavior of soft tissue.

[0056] The present application continuously provides time boundary conditions through sparse feature point displacement vector field, and obtains effective quality. These boundary conditions are used for biomechanical model. Through the cooperative operation of stereo matching and neural network extracted feature points and stereo reconstruction technology, the accuracy and efficiency of boundary condition acquisition are improved. The deep learning network can effectively deal with complex situations such as tissue occlusion, bleeding and light change in the operation process due to its adaptive feature extraction capability. In addition, through the combination of stereo matching and stereo reconstruction technology, high precision can be ensured, and massive image data can be efficiently processed, providing rich and real-time updated boundary condition information for the biomechanical model.

[0057] The present application optimizes the algorithm solving efficiency and accuracy, adopts Nesterov accelerated gradient algorithm for solving optimization, and further improves the efficiency of the algorithm by proposing a new method for determining the optimal step size. This method dynamically adjusts the step size in each iteration to adapt to the current optimization state, thereby speeding up the convergence speed and improving the accuracy of registration. As a momentum optimizer, this algorithm adds the momentum of previous iterations in the current iteration, forming an effect of accelerating the movement towards the minimum point, which can effectively solve the problem of low efficiency of traditional numerical solution.

[0058] The present application enhances the organ deformation modeling and prediction ability, uses graph convolutional neural network for feature extraction, can fully capture the structure information and spatial relationship of abdominal organ grid, so as to realize more accurate modeling and prediction of abdominal organ deformation. GCN can process irregular structured data, learn the feature representation of liver surface points, blood vessel branch points and other nodes with the help of graph topology information. The self-attention mechanism is introduced into GCN, which allows the model to dynamically adjust the attention degree of neighbor nodes when processing each node, so as to capture more fine abdominal organ deformation features and blood vessel branch point features. Not only the model's extraction ability of soft tissue deformation features is strengthened, but also the model's adaptability to different patient tissue structures is enhanced, and the application range is expanded. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 A flow chart of a graph convolution enhanced abdominal organ real-time tracking method based on a binocular camera according to the present application;

[0060] Figure 2 A framework diagram of a liver real-time tracking algorithm according to the present application. DETAILED DESCRIPTION

[0061] In the following, the technical solutions of the present application will be described in detail through specific embodiments;

[0062] Please refer to Figures 1-2 The present application provides a graph convolution enhanced abdominal organ real-time tracking method based on a binocular camera, comprising the following steps:

[0063] S1: obtaining preoperative abdominal CT scan images of a patient and laparoscopic intraoperative image data collected by a binocular camera, and labeling target abdominal organs in the preoperative CT images and the intraoperative laparoscopic images, respectively;

[0064] S2: based on the preoperative CT images, constructing a three-dimensional surface model of the abdominal organs through a 3D organ segmentation network, and establishing a biomechanical model of the abdominal organs by using a co-rotational finite element modeling method;

[0065] S3: according to the intraoperative laparoscopic images, extracting abdominal three-dimensional point clouds of an initial frame, and performing rigid registration on the three-dimensional point clouds and the preoperative three-dimensional surface model to estimate a rotation and translation matrix;

[0066] S4: extracting an organ feature point set from the initial frame three-dimensional point clouds after rigid registration, combining the elastic constraint of the biomechanical model, estimating the deformation field of the feature point cloud, realizing elastic registration to compensate for the non-rigid deformation of the tissue, and capturing the actual shape and displacement of the organ;

[0067] S5: based on the feature point set of the initial frame, matching the feature points in the subsequent frames with the initial frame to realize dynamic tracking of the abdominal organs during the whole operation process;

[0068] S6: quantitatively and qualitatively evaluating the dynamic tracking results to verify the accuracy and robustness of the abdominal organ real-time tracking method.

[0069] S1 specifically includes:

[0070] S11: collecting CT image data, labeling abdominal organ contours using Mimics software, obtaining 3D segmentation masks corresponding to the CT images, and providing a training data set required for the subsequent 3D organ segmentation network;

[0071] S22: Collect the laparoscope video collected by the intraoperative binocular camera, label the abdominal organ region through the Label-me software, obtain the time sequence segmentation mask corresponding to the video image, and provide the required dataset for training the subsequent video organ segmentation network;

[0072] The data collection and labeling process of this step lays a foundation for subsequent model training through multi-modal data and multi-task labeling. The preoperative CT scan data of the liver and gallbladder is used to train a 3D segmentation network to extract the three-dimensional structure of abdominal organs from the CT. The laparoscope video collected by the intraoperative binocular camera is used to train a video segmentation network to segment the intraoperative organ region in real time.

[0073] S2 specifically includes:

[0074] S21: Preprocessing the obtained preoperative CT image, the preprocessing including image data size 256x256x256 image block cropping and normalization processing;

[0075] S22: Constructing a 3D organ segmentation network of the preoperative image based on the nnUNet framework, extracting the mask of the abdominal organ region from the preprocessed preoperative CT image, the network training using the Dice loss function, the optimizer being AdamW, the initial learning rate being set to 0.001, the iteration number being set to 10000-15000, and the CT image data using the data augmentation strategy including random scaling, Gaussian noise, contrast enhancement, Gaussian noise and random mirror flipping. This step trains the key model, the 3D segmentation network, for organ region identification, which uses the general medical image segmentation framework published in the top natural medical journal to realize full-automatic three-dimensional organ precise modeling;

[0076] S23: Inputting the three-dimensional mask of the abdominal organ into the Marching Cubes algorithm, first preprocessing the original data and reading into the specified array, second extracting a unit body and obtaining the values and coordinate positions of its 8 vertices, then comparing the function values of the 8 vertices of the unit body with the given isosurface value to obtain the state table of the unit body, then finding out the unit body edges intersecting with the isosurface according to the state table index, and using linear interpolation to calculate the intersection position, then using the central difference method to obtain the normal vector of the 8 vertices of the unit body, using linear interpolation to obtain the normal vector of each vertex of the triangular facet, and finally drawing the isosurface image according to the coordinates and normal vectors of the triangular facet vertices, and reconstructing a three-dimensional surface model composed of 5000 triangular facets for expressing the organ structure;

[0077] S24: The organ surface model is optimized, the number of triangular facets is reduced to 1000 by using a quadratic error metric algorithm, and the uniformity of the surface model is improved by Laplacian smoothing and re-meshing processing; the reconstruction of the 3D surface model in this step is to extract the abdominal organ region by using the 3D segmentation network, reconstruct the 3D surface model by using the Marching Cubes algorithm, and then simplify the triangular mesh by using the quadratic error metric and optimize the uniformity of the model surface, so as to provide a lightweight and high-precision geometric basis for biomechanical modeling and registration;

[0078] S25: Combined with the optimized surface model, a co-rotational finite element modeling method is used to construct a biomechanical model.

[0079] S25 specifically includes:

[0080] S251: A quasi-static integral scheme is used, the soft tissue is regarded as an elastic body by assuming that the inertial force component is much smaller than the elastic component, and based on Newton's second law, the static balance is mainly considered and the inertial force is ignored, so that the overall biomechanical model where M is the mass, f represents the sum of internal and external forces corresponding to the deformed shape and .

[0081] S252: Tetrahedron is selected as the basic unit of the organ biomechanical model, and for each tetrahedral element e, the deformation equation is constructed as where is the displacement vector of the vertex, is the force acting on the element, is the stiffness matrix of the element, which is expressed as where V is the total volume, is the volume of an element, is the strain-displacement matrix, is the stress-strain matrix, represents the rotation matrix of the current state of the tetrahedron relative to the initial state; the stress-strain matrix is derived from the Young's modulus and Poisson's ratio where , and νrespectively represent Young's modulus and Poisson's ratio; this step adopts a co-rotational finite element method to construct a soft tissue biomechanical model, based on a quasi-static assumption, takes a tetrahedral element as a basic element, describes tissue mechanical properties through a stiffness matrix and a stress-strain matrix, introduces a rotation matrix to separate rigid rotation and pure deformation, effectively handles large deformation problems, ensures the objectivity of stress calculation, and supports the nonlinearity of materials such as the viscoelasticity of liver tissue, derives the element stiffness matrix through a strain energy density function, realizes accurate modeling of complex deformations such as compression, stretching and shearing, and provides a physical basis for accurate registration and real-time deformation prediction of organs during surgery.

[0082] S3 specifically comprises:

[0083] S31: Before the start of the operation, take pictures of the multi-angle calibration board for the binocular camera, use the OpenCV function cv: findChessboardCorners to detect the corner points of the checkerboard in each image, then for each detected corner point, extract its three-dimensional position in the world coordinate system, thereby establishing a mapping relationship between two-dimensional image coordinates and three-dimensional world coordinates; use the cv: calibrateCamera function to pass all collected three-dimensional object points and corresponding two-dimensional image points to the function to calculate the internal parameters and distortion coefficients of the camera; then, perform distortion correction using the cv: undistort function, and perform distortion correction on the original image according to the camera intrinsic parameter matrix and distortion coefficients obtained in the previous step; finally, evaluate the calibration by calculating the reprojection error to evaluate the accuracy of the calibration, and complete the general calibration of the binocular camera; this step eliminates lens distortion through OpenCV camera calibration, checkerboard corner point detection, and internal parameter / distortion coefficient calculation; enables the binocular camera to accurately capture the precise position of the organ, and establishes an accurate, stable and quantifiable three-dimensional visual reference for the entire surgical navigation system;

[0084] S32: Based on the initial frame of the intraoperative endoscope image, a semi-global block matching algorithm is used to match the left and right images of the binocular camera, first, the horizontal Sobel operator is used to detect the edges of the left and right images captured by the calibrated binocular camera, and the gradient image is obtained, the matching cost is calculated, for each pixel, the matching cost with the corresponding pixel under different disparities is calculated, the absolute difference is used as the cost function, the cost is composed of two parts: the BT cost (Birchfield-Tomasi cost) calculated from the gradient image after preprocessing and the BT cost calculated from the original image without preprocessing, then the two are added, the energy function is minimized, the dynamic programming method is used, the cumulative cost is calculated along the 8 directions, and the minimum value is taken as the final cost, the disparity map is generated, and the post-processing method is used to repair the abnormal values or holes in the disparity map, and the initial point cloud is reconstructed after median filtering;

[0085] S33: converting the depth information of each pixel point in the parallax map into three-dimensional space coordinates to generate an initial frame corresponding to the abdominal organ three-dimensional point cloud;

[0086] S34: using a density adaptive registration algorithm based on a Dirichlet process Gaussian mixture model to solve the registration problem using a variational Bayesian inference framework, expanding the point cloud data and the model in a coarse-to-fine manner to improve the accuracy and efficiency of registration, estimating the scanning weight of each point to obtain an initial rotation matrix and a translation vector , so as to realize adaptive rigid registration between the preoperative three-dimensional surface model and the intraoperative three-dimensional point cloud; this step uses a semi-global block matching algorithm to generate a parallax map and reconstruct an initial point cloud, and uses a Dirichlet process Gaussian mixture model to realize rigid mutual registration of the preoperative 3D model and the intraoperative point cloud, and to obtain a rotation matrix and a translation vector.

[0087] S4 specifically comprises:

[0088] S41: based on the intraoperative endoscopic image, using a SiamMask target tracking algorithm to delineate the target region, using the binocular lens video data collected in the S1 step and the abdominal organ region label fine-tuning model weight, using a preselected initial bounding box as input to quickly extract the mask of the abdominal organ region in the intraoperative image.

[0089] S42: constructing a feature point extraction network to extract features from the abdominal organ region to obtain feature point set coordinates and description vectors;

[0090] S43: non-rigid matching of the tetrahedral vertexes of the biomechanical model in S2 as control points with the feature point set to realize modeling and correction of organ elastic deformation; this step introduces control points through elastic compensation for organ deformation, marks the fixed constraint region, minimizes the cost function containing the stiffness matrix, compensates for the preoperative-intraoperative elastic deformation difference, and realizes preliminary alignment of the anatomical structure, laying a foundation for dynamic tracking.

[0091] The specific implementation process of the above feature point extraction network is as follows:

[0092] A graph convolutional neural network with self-attention mechanism (SA-GCNPoint) is constructed for efficiently extracting feature points of abdominal organs; the overall architecture and training method of the network refer to the SuperPoint network; a single shared encoder is included for processing and reducing the dimension of the input image, and the backend is decoupled into two decoder "heads" that perform interest point detection and interest point description, respectively; the encoder is improved and optimized, and the original VGG encoder is replaced with a graph convolutional network with self-attention mechanism (i.e. 4 groups of Transformer and 4 groups of graph convolutional network), which extracts features of abdominal organs respectively, while the configuration of the decoder remains unchanged; by fusing the self-attention mechanism with the graph convolution, the encoder can capture the complex relationships between all nodes in the organ graph at once, obtain the global context, and efficiently solve the long-distance dependency problem, thereby accurately modeling the interaction relationship between organs in a large receptive field; the network uses a self-supervised learning method, which can automatically learn the feature points of abdominal organs without manual annotation of data;

[0093] The specific calculation process of the above non-rigid matching is as follows:

[0094] The biomechanical control point set Each control point is matched according to the minimum Hamming distance and the feature point set to determine the corresponding point , and displacement constraints are established between each pair of matched points to construct the following cost function: ; wherein, is the stiffness coefficient of the organ model, which is of the same order of magnitude as the Young's modulus, is the number of control points, are the initial rotation matrix and translation vector obtained in S3, respectively, is the global stiffness matrix based on the element stiffness matrix group set; by globally minimizing the cost function, the optimal displacement vector is solved to accurately estimate and match the non-rigid deformation field.

[0095] The S5 specifically comprises:

[0096] S51: using the target tracking algorithm and the feature point extraction network described in S4, extract feature points from each subsequent frame in the same way, denoted as set ;

[0097] S52: according to the coordinates and description vectors of each feature point, calculate the current frame feature point set and the feature points Corresponding to the Hamming distance between the description vectors, get by nearest neighbor matching To the initial point pair ;

[0098] S53: Based on Lucas Kanade optical flow algorithm, estimate the displacement of each matching feature point pair by using cv.calcOpticalFlowPyrLK function in opencv. When the displacement is greater than the mean value + 3 times the standard deviation, it is considered as abnormal optical flow and removed to obtain To the feature point set

[0099] S54: Match the remaining unmatching feature points with the organ biomechanical model according to the non-rigid matching method described in S43, construct and solve the following cost function , wherein ; Through optimization, the final cross-frame feature point displacement estimation is obtained, and the dynamic tracking of the abdominal organ is realized; In this step, the SiamMask fine-tuning model is used to segment the intraoperative abdominal organ region in real time, the Lucas Kanade optical flow method is used to track the feature point displacement, the reliable feature point set is obtained by combining the distance threshold filtering, the triangular measurement method is used to reconstruct the 3D coordinates of the feature points, the distance field function is calculated as the displacement constraint of the biomechanical model, the node displacement vector is dynamically updated by minimizing the cost function containing the global stiffness matrix, the Newton acceleration method is used to optimize the cost function, the step size and momentum are adjusted dynamically to converge, the registration efficiency is improved, the real-time registration of the preoperative model and the intraoperative deformed organ is realized, and the dynamic tracking with sub-millimeter accuracy is supported; It continuously locks the dynamic changes of the key positions of the organ, provides real-time, accurate and robust navigation basis for doctors or surgical robots.

[0100] S6 specifically includes:

[0101] The registration error, root mean square error (RMSE) is used to measure the average Euclidean distance between the corresponding point sets after registration, which is more sensitive to larger errors and reflects the overall registration accuracy; the mean absolute error (MAE) calculates the average absolute value of the corresponding point distance, which is less affected by outliers, and reflects the stability and robustness of the registration result; the maximum error (Error) takes the maximum distance in all corresponding point pairs, which directly reflects the registration deviation in the worst case, and is a key indicator for evaluating the robustness of the algorithm; the effectiveness of the proposed algorithm is evaluated by calculating the root mean square error, mean absolute error and maximum error between the point cloud and the 3D model, and the six methods of ICP, DCP, RPM-Net, ROPNet, PREDATOR and UTOPIC are compared in the experiment; the results show that the proposed method has the best performance on the two data sets, and the point cloud registration effect is close to the true value, which verifies the effectiveness and robustness of the preoperative-intraoperative dynamic registration and real-time tracking of liver deformation.

[0102] Table 1 Experimental results of different initial registration algorithms of the present application on the 3Dircadb liver CT data set; error indicators are in millimeters, the lower the value, the better the performance; the method proposed in this patent achieves the lowest value in all indicators

[0103]

[0104] Table 2 Experimental results of different initial registration algorithms of the present application on the DePoLL laparoscopic liver surgery data set; error indicators are in millimeters, the lower the value, the better the performance; the method proposed in this patent achieves the lowest value in all indicators

[0105]

[0106] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any skilled person in the art can make equivalent replacement or change according to the technical solution and inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for real-time tracking of abdominal organs using image convolution enhancement based on binocular cameras, comprising the following steps: S1: Acquire the patient's preoperative abdominal CT scan images and intraoperative laparoscopic images captured by a binocular camera, and label the target abdominal organs in the preoperative CT images and intraoperative laparoscopic images respectively. S2: Based on preoperative CT images, a three-dimensional surface model of the abdominal organs is constructed using a 3D organ segmentation network, and a biomechanical model of the abdominal organs is established using the co-rotation finite element modeling method. S3: Based on the intraoperative laparoscopic images, extract the abdominal 3D point cloud from the initial frame, rigidly register it with the preoperative 3D surface model, and estimate the rotation and translation matrices. S4: For the initial frame 3D point cloud after rigid registration, extract the organ feature point set, combine it with the elastic constraints of the biomechanical model, estimate the deformation field of the feature point cloud, and realize elastic compensation to capture the actual shape and displacement of the organ: S41: Based on intraoperative laparoscopic images, the SiamMask target tracking algorithm is used to obtain the mask of the abdominal organ region in the intraoperative images; S42: Construct a feature point extraction network to extract features from laparoscopic images in the abdominal organ region and obtain feature point coordinates and description vectors; S43: Use the vertices of the tetrahedron in the biomechanical model described in S2 as control points. The feature point set Non-rigid matching is used to model and correct the elastic deformation of organs; specifically, this includes: setting up a biomechanical control point set. Each control point in Based on the minimum Hamming distance and the set of feature points Perform matching to determine the corresponding feature points. And establish displacement constraints between each pair of matching points, thereby constructing the following cost function: ;in, It is the stiffness coefficient of the organ model, and its value is on the same order of magnitude as Young's modulus. It refers to the number of control points. Let S1 and S2 be the initial rotation matrix and translation vector obtained in S3, respectively. For element stiffness matrix The global stiffness matrix is ​​obtained by assembly; the optimal displacement vector is solved by globally minimizing the cost function. This enables accurate estimation and matching of non-rigid deformation fields; S5: Based on the feature point set of the initial frame, the feature points in subsequent frames are matched with the initial frame across frames to achieve dynamic tracking of abdominal organs throughout the entire surgical process. S6: Perform quantitative and qualitative evaluations on the dynamic tracking results to verify the accuracy and robustness of the real-time abdominal organ tracking method.

2. The method for real-time tracking of abdominal organs based on image convolution enhancement using a binocular camera according to claim 1, wherein step S2 specifically includes: S21: Preprocess the acquired preoperative CT images, the preprocessing including cropping and normalizing the image data into 256×256×256 image blocks; S22: Construct a 3D organ segmentation network for the preoperative images based on the nnUNet framework, and extract the mask of the abdominal organ region from the preprocessed preoperative CT images. S23: Input the three-dimensional mask of the abdominal organ into the Marching Cubes algorithm to reconstruct a three-dimensional surface model composed of 5,000 triangular facets, which is used to express the organ structure; S24: Optimize the organ surface model by using a quadratic error metric algorithm to reduce the number of triangular facets to 1000, and improve the uniformity of the surface model through Laplacian smoothing and re-meshing. S25: Combining the optimized surface model, a biomechanical model is constructed using the co-rotational finite element modeling method.

3. The method for real-time tracking of abdominal organs based on image convolution enhancement using a binocular camera according to claim 2, wherein step S25 specifically includes: S251: Employing a quasi-static integral scheme based on Newton's second law, focusing on static equilibrium and neglecting inertial forces, resulting in an overall biomechanical model. Where M is mass and f represents the state corresponding to deformation. and The sum of internal and external forces; S252: Tetrahedrons are selected as the basic unit of the organ biomechanical model. For each tetrahedral element e, its deformation equation is constructed as follows: ,in It is the displacement vector of the vertex. It is a force acting on elements. The stiffness matrix of the element is represented as Where V is the total volume. It is the volume of an element. It is the strain-displacement matrix. It is the stress-strain matrix. The rotation matrix represents the current state of the tetrahedron relative to its initial state; The stress-strain matrix It is derived from Young's modulus and Poisson's ratio; S253: The unit and overall biomechanical model are constructed by dynamically solving the problem using a time-stepping linear method.

4. The method for real-time tracking of abdominal organs based on image convolution enhancement using a binocular camera according to claim 1, wherein step S3 specifically includes: S31: Before the surgery begins, the binocular camera is used to take pictures of the calibration plate from multiple angles. The mapping relationship between the two-dimensional image coordinates and the three-dimensional world coordinates is established through corner detection. The internal parameters and distortion coefficients of the camera are calculated to complete the general calibration of the binocular camera. S32: Based on the initial frame of the intraoperative laparoscopic image, a semi-global block matching algorithm is used to match the left and right binocular images to generate a disparity map; S33: Convert the depth information of each pixel in the disparity map into three-dimensional spatial coordinates to generate a three-dimensional point cloud of the abdominal organs corresponding to the initial frame; S34: Using a density adaptive registration algorithm based on the Dirichlet process Gaussian mixture model, the 3D point cloud is aligned with the preoperative 3D surface model described in S2 in terms of overall position, orientation, and scale, and the rotation matrix is ​​calculated. Translation vector This achieves rigid registration.

5. The method for real-time tracking of abdominal organs based on graph convolution enhancement using a binocular camera according to claim 1, wherein the feature point extraction network specifically includes an encoder and two decoders that perform feature point detection and feature point description respectively; wherein the encoder is composed of a graph convolutional network with a self-attention mechanism, and the decoders are composed of a convolutional neural network.

6. The method for real-time tracking of abdominal organs based on image convolution enhancement using a binocular camera according to claim 1, wherein step S5 specifically includes: S51: Using the target tracking algorithm and feature point extraction network described in S4, extract the target points from each subsequent frame in the same way. Let the set of feature points be _ ... ; S52: Calculate the feature point set of the current frame based on the coordinates and description vectors of each feature point. Feature points from the previous frame The Hamming distance between corresponding description vectors is obtained through nearest neighbor matching. For initial point pairs ; S53: Use optical flow to estimate the displacement of each matched feature point pair, and remove feature points whose displacement is greater than the mean plus three standard deviations to obtain... For the feature point set; S54: The remaining... For each unmatched feature point, the non-rigid matching method described in S43 is used to match it with the organ biomechanical model, and the following cost function is constructed and solved. ,in The final cross-frame feature point displacement estimate is obtained through optimization, enabling dynamic tracking of abdominal organs.

7. The method for real-time tracking of abdominal organs based on image convolution enhancement using a binocular camera according to claim 6, wherein the optimization solution process employs Newton's acceleration method, and accelerates convergence by dynamically adjusting the step size and momentum, specifically including the following steps: Select initial point And set the learning rate In each iteration The following steps are executed sequentially: Calculate the predicted location. Calculate the objective function in gradient at and update momentum Finally, based on the updated parameters ; until the convergence condition is met, the process of solving the displacement estimation is accelerated.

8. The method for real-time tracking of abdominal organs based on image convolution enhancement using a binocular camera according to claim 1, wherein step S6 specifically includes: The registration error was evaluated on two different open-source clinical liver datasets, 3Dircadb and DePoLL. The effectiveness of the proposed algorithm was assessed by calculating the root mean square error, mean absolute error, and maximum error between the point cloud and the biomechanical model. The experiment compared six methods: ICP, DCP, RPM-Net, ROPNet, PREDATOR, and UTOPIC.

Images