A three-dimensional reconstruction optical navigation method and device for electronic choledochoscope

Abstract

The application provides a three-dimensional reconstruction optical navigation method and device for an electronic choledochoscope, and the navigation method specifically comprises the following steps: S1, extracting MRCP images of a patient meeting a condition from a clinical case library for depth processing and data labeling; S2, importing the labeled data into a liver and gallbladder three-dimensional reconstruction system to reconstruct a liver, a bile duct tree and a stone; S3, taking a target stone bile duct as a terminal point to calculate a shortest and safest three-dimensional channel from a laparoscope bile duct incision or a needle insertion point on the skin to the bile duct; S4, inserting a marked choledochoscope into the bile duct; S5, entering a double image fusion navigation mode; S6, performing puncture until the tip of the choledochoscope accurately reaches a target bile duct stone position; and the like. The application can lock and mark an intrahepatic target bile duct and a stone position before surgery, and reconstruct a three-dimensional map, so that the choledochoscope can be guided by the three-dimensional map to quickly enter the target bile duct after entering the intrahepatic bile duct, and the surgery and stone removal efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of medical technology, specifically to a three-dimensional reconstruction optical navigation method and device for electronic cholangioscopes. Background Technology

[0002] Cholangioscopy techniques include rigid cholangioscopy and electronic cholangioscopy. Electronic cholangioscopy is widely used in the treatment of bile duct stones. Stones are removed through sinus tracts formed by laparoscopic bile duct incisions or percutaneous transhepatic cholangiocarcinoma, or through a T-tube placed after hepatobiliary surgery, entering the extrahepatic or intrahepatic bile ducts. This method is the main minimally invasive treatment for intrahepatic bile duct stones. However, because the intrahepatic bile ducts have many branches, often referred to as the "bile duct tree," the following problems frequently arise during stone removal: firstly, the cholangioscope often loses its way after entering the intrahepatic bile duct, making it difficult to reach the target bile duct; secondly, it is very easy to miss the target bile duct, leading to stone residue.

[0003] The liver is the largest solid organ in the human body, containing a complex bile duct system. Therefore, even experienced surgeons find it difficult to accurately determine the exact location and target direction of the "scope" within the bile duct during cholangioscopic stone removal. If bile duct stenosis or variations are encountered, the procedure becomes even more challenging, increasing the risk of missing bile duct stones, affecting the surgical outcome, and potentially leading to the failure of the minimally invasive procedure. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a three-dimensional reconstruction optical navigation method and device for electronic cholangioscopy. This method can preoperatively locate and mark the target bile duct and stone location within the liver, and reconstruct a three-dimensional map. After the cholangioscopy enters the intrahepatic bile duct, it is guided by the synchronous, real-time three-dimensional map to quickly enter the target bile duct, thereby improving the efficiency of surgery and stone removal.

[0005] To achieve the above technical solution, the present invention provides a three-dimensional reconstruction optical navigation method for an electronic cholangioscope, specifically including the following steps:

[0006] S1. Extract MRCP images of patients that meet the criteria from the clinical case database and perform depth processing and data annotation;

[0007] S2. Import the labeled data into the "Hepatobiliary 3D Reconstruction System" to reconstruct the liver, bile duct tree and stones, and distinguish bile ducts and stones with different colors. At the same time, record the three-dimensional coordinates of the bile duct where the stone is located.

[0008] S3. Using the target bile duct as the endpoint, calculate the shortest and safest three-dimensional channel from the laparoscopic bile duct incision or from the skin needle entry point to the bile duct.

[0009] S4. After successful laparoscopic incision of the bile duct or percutaneous liver puncture, the marked choledochoscope is inserted into the bile duct according to the calculated three-dimensional channel. The mark is uniformly illuminated by a near-infrared LED array, and the binocular camera simultaneously acquires left and right images at a frequency of 60Hz. Through sub-pixel center extraction, epipolar matching, triangulation and EPnP algorithm, the three-dimensional coordinates (x,y,z) and orientation (α,β,γ) of the endoscope tip are calculated in real time.

[0010] S5. Enter the dual-image fusion navigation mode. The left half screen displays real-time cholangioscopy video, while the right half screen overlays the preoperative 3D bile duct tree, planned path, real-time position of the endoscope tip, and the trajectory already traveled.

[0011] S6. Calculate the three-dimensional Euclidean distance between the endoscope tip and the center of the target bile duct stone at a frequency of 30Hz. Display the distance to the target at the top. When the endoscope tip deviates from the planned path by more than 2mm, the arrow will indicate the return direction in real time until the endoscope tip accurately reaches the location of the target bile duct stone.

[0012] Preferably, in step S2, the hepatobiliary three-dimensional reconstruction system reconstructs the liver, bile duct tree, and stones in the following manner:

[0013] S21. Read the MRCP image data after step S1 annotation into the specified array;

[0014] S22. Extract a single cell from the grid data volume to become the current cell volume, and obtain the values ​​and position information of the 8 vertices of this cell volume;

[0015] S23. Compare the function determined by the 8 vertices of the current element with the isosurface to obtain the state table of this element;

[0016] S24. Using the current state table index, determine the cell edges that intersect with the isosurface, and use linear interpolation to calculate the coordinates of each intersection point;

[0017] S25. Use the central difference method to calculate the normal vectors of the 8 vertices of the current cell, and use linear interpolation to obtain the normal direction of each vertex of the triangle facet.

[0018] S26. Draw isosurface images based on the vertex coordinates and normal vertex vectors of each triangular facet, and after rendering, obtain three-dimensional model images of the liver, bile duct tree, and stones.

[0019] Preferably, in step S1, the MRCP image is subjected to depth processing in the following manner:

[0020] S11. Convert the original DICOM format image of the MRCP image into PNG or JPEG two-dimensional image format;

[0021] S12. Extract shallow features (FS) of bile duct structures from a two-dimensional image through a convolutional layer;

[0022] S13. Extract deep features (FD) of bile duct structures from two-dimensional images using a six-layer RHAG and an SFB module;

[0023] S14. Use global residual connections to fuse shallow features (FS) and deep features (FD) of the bile duct structure.

[0024] S15. Upsampling is performed using two 3×3 convolutional layers and pixel shuffling to reconstruct a high-resolution image that highlights the bile duct structure by fusing the shallow features FS and deep features FD.

[0025] Preferably, in step S1, the depth-processed MRCP image is annotated in the following manner:

[0026] S16. Using LabelMe as the annotation tool, the region is delineated on the depth-processed MRCP 2D image, and a JSON-formatted annotation file is generated.

[0027] S17. After completing the initial drawing, the annotation personnel shall check the annotation boundaries frame by frame to avoid omissions and errors in annotation.

[0028] S18. The annotation results shall be independently reviewed by two or more experienced radiologists or gastroenterologists. If there is any disagreement, a third expert shall make the decision.

[0029] S19. Calculate the consistency index among annotators and conduct key reviews and corrections for areas with low consistency.

[0030] S20. Store the approved annotation files and the corresponding original images, and establish a unified naming and hierarchical system to form a standardized dataset.

[0031] Preferably, in step S16, the specific process of using LabelMe as a labeling tool to delineate regions in the two-dimensional image is as follows:

[0032] S161. Import the MRCP image to be labeled into LabelMe;

[0033] S162. Based on the anatomical location, select the outline area of ​​the bile duct in sequence, use the polygon tool, and adopt the one-click quick annotation method to obtain a complete depiction of the bile duct structure.

[0034] S163. After completing the drawing, name the area;

[0035] S164. Repeat the above operation for all slices until the entire bile duct system is labeled.

[0036] S165. Save the annotation results and generate the corresponding label PNG and JSON files.

[0037] Preferably, the specific process of the one-click rapid annotation method used in step S162 is as follows:

[0038] (1) The texture complexity of the image is calculated by calculating the gray-level co-occurrence matrix of the image;

[0039] (2) The concept of image texture complexity is introduced into the superpixel segmentation algorithm for adaptive calculation and selection, generating the most suitable superpixel segmentation result for each image;

[0040] (3) After generating the superpixel segmentation result, the result is combined with the original image. The operator clicks to select the candidate area and fills the superpixel area where the mouse clicked pixel is located through the seed fill algorithm to realize the mouse click annotation function.

[0041] (4) After the region selection is completed, the superpixel is filled by the filling algorithm. The cracks generated by the superpixel generation algorithm cannot be colored at the same time. The generated filling area is a fragmented area with gaps between each superpixel. A closing operation is performed on the image with cracks to clear the cracks and generate the required complete bile duct structure depiction.

[0042] Preferably, in step S3, the shortest and safest three-dimensional channel from the laparoscopic bile duct incision or the skin needle entry point to the bile duct is calculated in the following manner:

[0043] S31. First, calculate the spatial orientation of the puncture path formed by the line connecting the needle insertion point and the target point.

[0044] S32. Generate the trajectory to the needle insertion pose based on RRT-Connect and MoveIt trajectory post-processing;

[0045] S33. Perform linear interpolation on the puncture path and plan the linear puncture trajectory based on MoveIt's Cartesian space path planning computeCartesianPath interface.

[0046] Preferably, in step S3, after planning the straight puncture trajectory, a three-dimensional image is created and compared with the preoperative MRCP image to assess whether there is a blind spot in the bile duct tree.

[0047] Preferably, in step S3, if there are multiple bile duct stones, multiple numbered paths are generated sequentially, and their length, curvature, and risk score are given.

[0048] The present invention also discloses a three-dimensional reconstruction optical navigation device for an electronic cholangioscope, comprising: a navigation workstation and a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein the processor executes the above-described navigation method when running the program.

[0049] The beneficial effects of the three-dimensional reconstruction optical navigation method, system, and device for electronic cholangioscope provided by this invention are as follows:

[0050] 1) This invention overcomes the shortcomings of existing technologies where choledochoscopes cannot be positioned in real time and cannot accurately locate the bile duct where the stones are located. By planning the route before the operation and tracking the position of the choledochoscope in real time during the operation, the deviation between the bile duct route and the planned route is corrected in real time. The combination of real-time images and videos from the choledochoscope and virtual reality images inside the bile duct improves the efficiency, objectivity and safety of positioning, further reduces stone residue and recurrence, and improves the treatment effect of hepatobiliary stones.

[0051] 2) This invention provides a visualized "map" for the choledochoscope during its movement within the bile ducts. Through optical navigation technology, the bile duct bundle is transformed into a three-dimensional map. Firstly, by preoperatively locating and marking the target bile ducts and stone sites within the liver, the choledochoscope is guided by this synchronous, real-time three-dimensional map after entering the liver, improving surgical and stone removal efficiency. Secondly, the three-dimensional map generated during the choledochoscope's movement within the liver is compared with preoperative MRCP images, reducing blind spots during intraoperative exploration of the liver and lowering the rate of residual stones.

[0052] 3) By performing depth processing on MRCP two-dimensional images, this invention can more effectively obtain high-resolution MRCP images that highlight the bile duct structure and lesion location, providing a good image basis for subsequent data annotation of the anatomical structures of intrahepatic bile ducts, extrahepatic bile ducts, common bile duct, and pancreatic duct in MRCP images, and thus providing a good data basis for determining the target bile duct to be punctured.

[0053] 4) The one-click fast annotation method provided by this invention can effectively solve the problem of time-consuming and labor-intensive operation using traditional manual data annotation tools. This invention innovatively introduces superpixel segmentation algorithm into image annotation tasks. At the same time, it proposes to use gray-level co-occurrence matrix to determine the complexity of image in texture feature dimension, which can achieve one-click fast annotation with high annotation accuracy, greatly improving the efficiency and accuracy of bile duct structure depiction in MRCP images.

[0054] 5) The three-dimensional reconstruction method for the liver and gallbladder provided by this invention, combined with the MRCP image depth processing technology and data annotation technology provided by this invention, can quickly and accurately reconstruct the anatomical structure of intrahepatic bile ducts, extrahepatic bile ducts, common bile ducts and pancreatic ducts in three dimensions, and can highlight the location of lesions, thereby providing a good display basis for accurately locating the bile duct where the stones are located and for preoperative route planning.

[0055] 6) The present invention also provides a method to calculate the shortest and safest three-dimensional channel from the laparoscopic bile duct incision or from the skin needle entry point to the target bile duct in real time. If there are multiple bile duct stones, multiple numbered paths can be generated in sequence, and the length, curvature and risk score can be given, which can effectively enhance the efficiency and safety of the operation. Attached Figure Description

[0056] Figure 1 The flowchart of the overall steps provided for this invention.

[0057] Figure 2 This is the original image of MRCP used in this invention.

[0058] Figure 3 This is a flowchart of MRCP image labeling using the LabelMe tool in this invention.

[0059] Figure 4 This is a comparison diagram between the virtual three-dimensional bile tree formed by optical navigation during cholangioscopy in this invention and the MRCP annotation results. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0061] Example 1: A three-dimensional reconstruction optical navigation method for electronic cholangioscopes.

[0062] Reference Figures 1 to 4 As shown, a three-dimensional reconstruction optical navigation method for electronic cholangioscopes specifically includes the following steps:

[0063] (I) MRCP Image Processing and Annotation

[0064] Magnetic resonance cholangiopancreatography (MRCP) is a non-invasive imaging technique that clearly displays the biliary system and pancreatic duct structures. Compared to CT or ERCP, MRCP offers advantages such as no radiation exposure, high soft tissue resolution, and superior contrast. Therefore, it is widely used in clinical practice for the diagnosis and follow-up of biliary and pancreatic diseases, such as bile duct stones, bile duct strictures, bile duct dilatations, and biliary tumors.

[0065] High-quality annotation of MRCP data is a crucial step in combining medical imaging with artificial intelligence. A well-structured bile duct annotation dataset with clearly defined pathological features not only provides foundational data for the development and visualization of intelligent bile duct navigation systems and AI model development, but also promotes the intelligent application of MRCP in the diagnosis of biliary tract diseases. However, a search revealed that there are no publicly available methods specifically for annotating MRCP data, making it impossible to quickly and accurately construct a well-structured bile duct annotation dataset with accurate boundaries and clearly defined pathological features. Consequently, this hinders the provision of solid data support for subsequent bile duct structure modeling and disease-aided diagnosis.

[0066] Therefore, this invention first provides an MRCP image processing and annotation method, which mainly includes three steps: a data preparation stage, an annotation process stage, and a quality control stage.

[0067] 1. The data preparation stage mainly includes:

[0068] (1) Data collection: Extract eligible MRCP images from the clinical case database, ensuring that all data have been stripped of patient personal identification information and comply with medical ethics and privacy protection requirements.

[0069] (2) Data preprocessing: Convert the original DICOM format image into a common two-dimensional image format (such as PNG or JPEG), and adjust the window width and window level appropriately. Then perform depth processing on the two-dimensional image to highlight the bile duct structure while ensuring that the image resolution is not lower than the original acquisition accuracy.

[0070] The specific process of depth processing for two-dimensional images is as follows:

[0071] (21) Given an MRCP input image, extract shallow features FS of the bile duct structure through a convolutional layer; this step involves utilizing low-level image features such as edge and texture information.

[0072] (22) Deep features (FD) of bile duct structure are extracted through a six-layer RHAG and an SFB module. The RHAG contains Hybrid Attention Blocks (HAB), Overlapping Cross-Attention Blocks (OCAB), and the SFB module. These modules work together to extract and fuse multi-scale features and global information.

[0073] (23) Use global residual connections to fuse shallow features (FS) and deep features (FD) of bile duct structures; this operation helps to utilize information from all levels during reconstruction, thereby improving image quality.

[0074] (24) Upsampling is performed using two 3×3 convolutional layers and pixel shuffle to reconstruct a high-resolution image that highlights the bile duct structure by fusing the shallow features FS and deep features FD. This step can highlight the bile duct structure while obtaining a high-resolution image.

[0075] By performing depth processing on two-dimensional images in the above manner and performing super-resolution reconstruction on low-resolution images, the quality of MRCP images can be effectively improved, and the accuracy of semantic segmentation can be significantly improved. High-resolution images highlighting bile duct structures can be obtained, providing a good image foundation for subsequent data annotation of the anatomical structures of intrahepatic bile ducts, extrahepatic bile ducts, common bile ducts, and pancreatic ducts in MRCP images.

[0076] (3) Labeling tool selection: LabelMe was used as the labeling tool. This tool supports various labeling methods such as polygons, rectangles, and polylines, and can intuitively depict regions of two-dimensional images and automatically generate JSON format label files, which is convenient for subsequent data processing and model reading.

[0077] 2. The annotation process mainly includes:

[0078] Annotation targets: Primarily annotating anatomical structures such as intrahepatic bile ducts, extrahepatic bile ducts, common bile duct, and pancreatic duct in MRCP images. If pathological features are present (such as stones or strictures), additional annotations can be added as needed for the research.

[0079] Labeling principles:

[0080] (1) Layer-by-layer annotation: Each two-dimensional MRCP image is independently annotated to ensure the continuity of anatomical structures on different slices.

[0081] (2) Detailed drawing: Use the polygon tool to draw along the edge of the bile duct point by point to ensure that the envelope area matches the boundary of the anatomical structure, and avoid oversimplification or omission of small branches.

[0082] (3) Grading and labeling: Different bile duct branches are marked according to anatomical classification, such as left hepatic duct, right hepatic duct, common hepatic duct, etc., and different colors or labels are used to distinguish them.

[0083] (4) Consistency maintenance: The same anatomical structure is named consistently in different slices so that subsequent three-dimensional reconstruction or cross-slice data fusion can be carried out.

[0084] Annotation operation process:

[0085] (1) Import the MRCP image to be labeled into LabelMe.

[0086] (2) Based on the anatomical location, select the outline area of ​​the bile duct in sequence, use the polygon tool, and use the one-click quick annotation method to obtain a complete depiction of the bile duct structure.

[0087] In order to effectively solve the problem of time-consuming and labor-intensive operation using traditional manual data annotation tools, this invention innovatively adopts a one-click rapid annotation method, the specific process of which is as follows:

[0088] First, the texture complexity of the image is calculated by using its statistical properties after calculating the gray-level co-occurrence matrix.

[0089] Secondly, the concept of image texture complexity is introduced into the superpixel segmentation algorithm for adaptive calculation and selection, generating the most suitable superpixel segmentation result for each image;

[0090] Next, after generating the superpixel segmentation result, the result is combined with the original image. The operator clicks to select the candidate area, and the superpixel area where the mouse clicked pixel is located is filled by the seed fill algorithm to realize the mouse click annotation function.

[0091] Finally, after the region selection is completed, the superpixel interior is filled using a filling algorithm. Cracks generated by the superpixel generation algorithm cannot be colored simultaneously. The generated filling region is a fragmented region with gaps between each superpixel. A closing operation is performed on the image with cracks to remove the cracks and generate the required complete bile duct structure depiction.

[0092] (3) After completing the drawing, name the area, such as “common_bile_duct”, “left_hepatic_duct”, etc.

[0093] Repeat the above steps for all slices until the entire bile duct system is labeled.

[0094] (4) Save the annotation results and generate the corresponding label PNG and JSON files.

[0095] This invention innovatively employs a one-click rapid annotation method for region depiction in 2D images, effectively solving the problem of time-consuming and labor-intensive operations using traditional manual data annotation tools. This invention innovatively introduces a superpixel segmentation algorithm into image annotation tasks and proposes using the gray-level co-occurrence matrix to determine the complexity of the image in the texture feature dimension. This enables one-click rapid annotation with high accuracy, significantly improving the efficiency and precision of depicting bile duct structures in MRCP images.

[0096] 3. The quality control stage mainly includes:

[0097] To ensure the accuracy and reliability of the annotation results, multi-level quality audits are required:

[0098] (1) Self-check: After completing the initial drawing, the annotator will check the annotation boundaries frame by frame to avoid omissions and errors in annotation.

[0099] (2) Expert review: The annotation results shall be independently reviewed by two or more radiologists or gastroenterologists with extensive experience. If there is any disagreement, a third expert shall make the decision.

[0100] (3) Consistency assessment: Calculate consistency indicators (such as Dice coefficient, IoU) among annotators, and conduct key reviews and corrections for areas with low consistency.

[0101] (4) Data processing: Store the approved annotation files and the original images accordingly, and establish a unified naming and hierarchical system to form a standardized dataset.

[0102] This invention utilizes the LabelMe tool on two-dimensional MRCP slices, strictly adhering to annotation range and naming conventions, and supplemented by rigorous quality control. This enables the rapid and accurate construction of a bile duct annotation dataset with clear structure, accurate boundaries, and well-defined pathological features, thereby providing a solid data foundation for the three-dimensional reconstruction of the target bile duct to be punctured.

[0103] (II) Three-dimensional reconstruction of the liver and gallbladder

[0104] The labeled data is imported into the "Hepatobiliary 3D Reconstruction System" to reconstruct the liver, bile duct tree, and stones. Different colors are used to distinguish bile ducts (greenish-green and semi-transparent) from stones (orange-red and bright). At the same time, the 3D coordinates of the bile duct where the stone is located are recorded and displayed in the navigation display module.

[0105] The hepatobiliary 3D reconstruction system reconstructs the liver, bile duct tree, and stones in the following ways:

[0106] (1) Read the labeled MRCP image data into the specified array;

[0107] (2) Extract a single cell from the grid data volume to become the current cell volume, and obtain the values ​​and position information of the 8 vertices of this cell volume;

[0108] (3) Compare the function determined by the 8 vertices of the current unit with the isosurface to obtain the state table of this unit;

[0109] (4) Determine the cell edges that intersect with the isosurface using the current state table index, and calculate the coordinates of each intersection point using linear interpolation.

[0110] (5) Use the central difference method to calculate the normal vectors of the 8 vertices of the current cell, and use linear interpolation to obtain the normal direction of each vertex of the triangle face;

[0111] (6) Draw isosurface images based on the vertex coordinates and normal vertex vectors of each triangular facet, and obtain three-dimensional model diagrams of the liver, bile duct tree and stones after rendering.

[0112] This invention, through the aforementioned three-dimensional reconstruction method of the liver and gallbladder, combined with the MRCP image depth processing and data annotation technology provided by this invention, can quickly and accurately reconstruct the anatomical structures of intrahepatic bile ducts, extrahepatic bile ducts, common bile duct, and pancreatic duct in three dimensions, and can highlight the location of lesions, thereby providing a good visual basis for accurately locating the bile duct where stones are located and for preoperative route planning.

[0113] (III) Optical Navigation

[0114] During cholangioscopy, the initial entry point of the cholangioscope into the bile duct must be aligned with the corresponding location on the bile duct tree in the "Hepatobiliary Three-Dimensional Intelligent Reconstruction System." Then, optical navigation guides the cholangioscope to display the stone retrieval path in real time, allowing for entry into the target intrahepatic bile duct for stone removal. Optical navigation primarily includes the following steps:

[0115] (1) Using the target bile duct containing the stone as the endpoint, calculate the shortest and safest three-dimensional path from the laparoscopic bile duct incision site or the skin needle entry point to the bile duct. The specific calculation process is as follows:

[0116] a. Calculate the spatial orientation of the puncture path formed by the line connecting the needle insertion point and the target point;

[0117] b. Generate a trajectory to the needle insertion pose based on RRT-Connect and MoveIt trajectory post-processing;

[0118] c. Perform linear interpolation on the puncture path and plan the linear puncture trajectory based on MoveIt's Cartesian space path planning computeCartesianPath interface.

[0119] d. After planning the straight puncture trajectory, create a three-dimensional image and compare it with the preoperative MRCP image to assess whether there are blind spots in the bile duct tree, so as to reduce the blind spots in intrahepatic bile duct exploration during the operation and reduce the stone residue rate.

[0120] The above calculation method allows for real-time and rapid calculation of the shortest and safest three-dimensional path from the laparoscopic bile duct incision or the skin needle entry point to the target bile duct, improving the accuracy and safety of navigation. Furthermore, if multiple bile duct stones are present, multiple numbered paths can be generated sequentially, providing their length, curvature, and risk score, effectively enhancing surgical efficiency and safety.

[0121] (2) After successful laparoscopic incision of the bile duct or percutaneous liver puncture, the marked choledochoscope is inserted into the bile duct according to the calculated three-dimensional channel. The mark is uniformly illuminated by a near-infrared LED array, and a binocular camera simultaneously acquires left and right images at a frequency of 60Hz. Through subpixel center extraction, epipolar matching, triangulation, and the EPnP algorithm, the three-dimensional coordinates (x, y, z) and orientation (α, β, γ) of the endoscope tip are calculated in real time. By calculating the three-dimensional coordinates and orientation of the endoscope tip in real time, the position of the choledochoscope inserted into the bile duct can be accurately displayed in real time.

[0122] (3) Enter the dual-image fusion navigation mode. The left half screen displays real-time cholangioscopy video, while the right half screen overlays the preoperative 3D bile duct tree, planned path (green virtual guide rail), real-time position of the endoscope tip (red light dot) and the trajectory already traveled (blue trajectory line).

[0123] (4) Calculate the three-dimensional Euclidean distance between the lens tip and the center of the target bile duct stone at a frequency of 30Hz. Display the distance to the target at the top. When the lens tip deviates from the planned path by more than 2mm, the return direction is indicated by the arrow until the lens tip accurately reaches the target bile duct stone position, thereby achieving non-contact, radiation-free, and sub-millimeter precision near-infrared optical navigation throughout the entire process.

[0124] The three-dimensional reconstruction optical navigation method for electronic cholangioscopy provided by this invention overcomes the shortcomings of existing technologies, such as the inability of cholangioscopy to locate in real time and accurately pinpoint the location of stones in the bile duct. Through preoperative route planning, the method tracks the position of the cholangioscopy in real time during the procedure and corrects deviations from the planned route. The combination of real-time images and videos from the cholangioscopy with virtual reality images of the bile duct improves the efficiency, objectivity, and safety of positioning, further reducing stone residue and recurrence, and enhancing the treatment effect of hepatobiliary stones. Furthermore, this invention provides a visualized "map" for the cholangioscopy during its movement within the bile duct. By converting the bile duct bundle into a three-dimensional map using optical navigation technology, the method allows for preoperative locking and marking of the target bile duct and stone location. Once the cholangioscopy enters the intrahepatic bile duct, the synchronous, real-time three-dimensional map guides the cholangioscopy to quickly enter the target bile duct, improving surgical and stone removal efficiency.

[0125] Example 2: A three-dimensional reconstruction optical navigation device for an electronic cholangioscope.

[0126] A three-dimensional reconstruction optical navigation device for an electronic cholangioscope includes: a navigation workstation and a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein the processor executes the navigation method described in Embodiment 1 when running the program.

[0127] The above description is only a preferred embodiment of the present invention, but the present invention should not be limited to the content disclosed in the embodiments and drawings. Therefore, any equivalent or modified embodiments made without departing from the spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction optical navigation device for an electronic cholangioscope, characterized in that... include: A navigation workstation, a memory installed on the navigation workstation, a processor, and a program stored in the memory and capable of running on the processor, wherein the processor executes the following navigation method: S1. Extract MRCP images of patients that meet the criteria from the clinical case database and perform depth processing and data annotation; The MRCP images are processed using the following method: S11. Convert the original DICOM format image of the MRCP image into PNG or JPEG two-dimensional image format; S12. Extract shallow features (FS) of bile duct structures from a two-dimensional image through a convolutional layer; S13. Extract deep features (FD) of bile duct structures from two-dimensional images using a six-layer RHAG and an SFB module; S14. Use global residual connections to fuse shallow features (FS) and deep features (FD) of the bile duct structure. S15. Upsampling is performed using two 3×3 convolutional layers and pixel shuffling to reconstruct a high-resolution image that highlights the bile duct structure from the fused shallow features FS and deep features FD. S2. Import the labeled data into the "Hepatobiliary 3D Reconstruction System" to reconstruct the liver, bile duct tree and stones, and distinguish bile ducts and stones with different colors. At the same time, record the three-dimensional coordinates of the bile duct where the stone is located. S3. Using the target bile duct as the endpoint, calculate the shortest and safest three-dimensional channel from the laparoscopic bile duct incision or from the skin needle entry point to the bile duct. S4. After successful laparoscopic incision of the bile duct or percutaneous liver puncture, the marked choledochoscope is inserted into the bile duct according to the calculated three-dimensional channel. The mark is uniformly illuminated by a near-infrared LED array, and the binocular camera simultaneously acquires left and right images at a frequency of 60 Hz. Through sub-pixel center extraction, epipolar matching, triangulation and EPnP algorithm, the three-dimensional coordinates (x, y, z) and orientation (α, β, γ) of the endoscope tip are calculated in real time. S5. Enter the dual-image fusion navigation mode. The left half screen displays real-time cholangioscopy video, while the right half screen overlays the preoperative 3D bile duct tree, planned path, real-time position of the endoscope tip, and the trajectory already traveled. S6. Calculate the three-dimensional Euclidean distance between the endoscope tip and the center of the target bile duct stone at a frequency of 30 Hz. Display the distance to the target at the top. When the endoscope tip deviates from the planned path by more than 2 mm, the arrow will indicate the return direction in real time until the endoscope tip accurately reaches the location of the target bile duct stone.

2. The three-dimensional reconstruction optical navigation device for an electronic cholangioscope as described in claim 1, characterized in that, In step S2, the hepatobiliary three-dimensional reconstruction system reconstructs the liver, bile duct tree, and stones in the following manner: S21. Read the MRCP image data after step S1 annotation into the specified array; S22. Extract a single cell from the grid data volume to become the current cell volume, and obtain the values ​​and position information of the 8 vertices of this cell volume; S23. Compare the function determined by the 8 vertices of the current element with the isosurface to obtain the state table of this element; S24. Using the current state table index, determine the cell edges that intersect with the isosurface, and use linear interpolation to calculate the coordinates of each intersection point; S25. Use the central difference method to calculate the normal vectors of the 8 vertices of the current cell, and use linear interpolation to obtain the normal direction of each vertex of the triangle facet. S26. Draw isosurface images based on the vertex coordinates and normal vertex vectors of each triangular facet, and after rendering, obtain three-dimensional model images of the liver, bile duct tree, and stones.

3. The three-dimensional reconstruction optical navigation device for an electronic cholangioscope as described in claim 1, characterized in that, In step S1, the depth-processed MRCP image is annotated in the following way: S16. Using LabelMe as the annotation tool, the region of the MRCP image after depth processing is delineated, and a JSON-formatted annotation file is generated. S17. After completing the initial drawing, the annotation personnel shall check the annotation boundaries frame by frame to avoid omissions and errors in annotation. S18. The annotation results shall be independently reviewed by two or more experienced radiologists or gastroenterologists. If there is any disagreement, a third expert shall make the decision. S19. Calculate the consistency index among annotators and conduct key reviews and corrections for areas with low consistency. S20. Store the approved annotation files and the corresponding original images, and establish a unified naming and hierarchical system to form a standardized dataset.

4. The three-dimensional reconstruction optical navigation device for an electronic cholangioscope as described in claim 3, characterized in that, In step S16, the specific process of using LabelMe as a labeling tool to delineate regions in the depth-processed MRCP image is as follows: S161. Import the MRCP image to be labeled into LabelMe; S162. Based on the anatomical location, select the outline area of ​​the bile duct in sequence, use the polygon tool, and adopt the one-click quick annotation method to obtain a complete depiction of the bile duct structure. S163. After completing the drawing, name the area; S164. Repeat the above operation for all slices until the entire bile duct system is labeled. S165. Save the annotation results and generate the corresponding label PNG and JSON files.

5. The three-dimensional reconstruction optical navigation device for an electronic cholangioscope as described in claim 4, characterized in that, The specific process of the one-click rapid annotation method used in step S162 is as follows: (1) The texture complexity of the MRCP image is calculated by calculating the gray-level co-occurrence matrix; (2) The concept of texture complexity of MRCP images is introduced into the superpixel segmentation algorithm for adaptive calculation and selection, and the most suitable superpixel segmentation result is generated for each MRCP image; (3) After generating the superpixel segmentation result, the result is combined with the original image. The operator clicks to select the candidate area and fills the superpixel area where the mouse clicks the pixel through the seed fill algorithm to realize the mouse click annotation function. (4) After the region selection is completed, the superpixel is filled by the filling algorithm. The cracks generated by the superpixel generation algorithm cannot be colored at the same time. The generated filling area is a fragmented area with gaps between each superpixel. A closing operation is performed on the MRCP image with cracks to clear the cracks and generate the required complete bile duct structure depiction.

6. The three-dimensional reconstruction optical navigation device for an electronic cholangioscope as described in claim 1, characterized in that, In step S3, the shortest and safest three-dimensional channel from the laparoscopic bile duct incision or the skin needle entry point to the bile duct is calculated as follows: S31. First, calculate the spatial orientation of the puncture path formed by the line connecting the needle insertion point and the target point. S32. Generate the trajectory to the needle insertion pose based on RRT-Connect and MoveIt trajectory post-processing; S33. Perform linear interpolation on the puncture path and plan the linear puncture trajectory based on MoveIt's Cartesian space path planning computeCartesianPath interface.

7. The three-dimensional reconstruction optical navigation device for an electronic cholangioscope as described in claim 6, characterized in that, In step S3, after planning the straight puncture trajectory, a three-dimensional image is created and compared with the preoperative MRCP image to assess whether there is a blind spot in the bile duct tree.

8. The three-dimensional reconstruction optical navigation device for an electronic cholangioscope as described in claim 6, characterized in that, In step S3, if there are multiple bile duct stones, multiple numbered paths are generated sequentially, and their length, curvature, and risk score are given.

Images