Systems and methods for robotic endoscopes with integrated tools for tomosynthesis within lesions

Abstract

There is provided a method for navigating a robotic endoscopic device, comprising: (a) navigating the robotic endoscopic device to a target area inside a body part; (b) acquiring a fluoroscopic image using a fluoroscopic imager when the tool extends through the robotic endoscopic device to the target area, where the fluoroscopic image includes a plurality of slices stacked in a depth direction; and (c) identifying a first slice having first coordinates corresponding to the center of the target region in the depth direction and a second slice having second coordinates corresponding to the tool in the depth direction; and (d) determining whether the tool is within the target area based on a difference between the first coordinate and the second coordinate.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 347,203, filed on May 31, 2022, which is incorporated herein by reference in its entirety. Background Art

[0003] Early diagnosis of lung cancer is critical. Lung cancer remains the deadliest form of cancer, causing more than 150,000 deaths each year. Compared to CT-guided TTNA (CT-TTNA), navigational bronchoscopy has a better safety profile (lower risk of pneumothorax, life-threatening bleeding, and longer hospital stay) and is able to stage the mediastinum, but has a lower diagnostic yield. Endoscopy (e.g., bronchoscopy) can involve entering and visualizing the interior of a patient's lumen (e.g., airway) for diagnostic and / or therapeutic purposes. During the procedure, a flexible tubular tool (e.g., an endoscope) can be inserted into the patient's body, and instruments can be passed through the endoscope to identify tissue sites for diagnosis and / or treatment.

[0004] Robotic bronchoscopy systems have generated interest for biopsy of peripheral lung lesions. Robotic platforms have superior stability, distal articulation, and visualization compared to conventional pre-curved catheters. Some conventional robotic bronchoscopy systems utilize shape sensing technology (SS) for guidance. SS catheters have embedded fiber-optic sensors that measure the catheter shape hundreds of times per minute. Other conventional robotic bronchoscopy systems combine direct visualization, optical pattern recognition, and geolocation sensing (OPRGPS) for guidance. Both SS and OPRGPS systems utilize pre-planned CT scans to create electronically generated virtual targets. However, SS and OPRGPS systems are prone to CT-to-body discrepancies (CT2BD). CT2BD is the mismatch between the electronic virtual target and the actual anatomical location of a peripheral lung lesion. CT2BD can occur for a variety of reasons, including atelectasis, anesthesia-induced neuromuscular weakness, tissue distortion caused by the catheter system, bleeding, ferromagnetic interference, and anatomical disturbances such as pleural effusions. Neither the SS system nor the OPRGPS platform can perform intraoperative real-time correction for CT2BD. In particular, CT2BD increases procedure length, causes operator frustration, and ultimately results in nondiagnostic procedures. Summary of the invention

[0005] Recently, digital tomosynthesis algorithms have been introduced for correcting CT2BD. Tomosynthesis is a limited angle tomography scan compared to full angle (e.g., 180 degree tomography). However, tomosynthesis reconstruction does not have uniform resolution. The resolution is worst in the depth direction. The standard method of showing a 3D volume dataset through three orthogonal planes (e.g., axial, sagittal, and coronal) is ineffective because the resolution of two planes is poor. For example, traditional tomosynthesis applied in pulmonology has poor depth resolution (e.g., anterior-posterior (AP)), making it difficult to determine whether the tool is within the target area (e.g., lesion). A common method of viewing a tomosynthesis volume is to scroll in the depth direction, where each slice has good resolution. In the case of pulmonology, the user can view the lesion and the needle in the coronal plane, and can manually scroll the slices in the anterior-posterior (AP) direction to identify the special relationship between the needle and the lesion. However, due to cumbersome manual processes and human errors, such processes may make it difficult to determine the spatial relationship of structures in the depth direction. Specifically, it is challenging to determine with high confidence and accuracy whether a small tool (eg, a biopsy needle) is inside a lesion in the AP direction of a tomosynthesis reconstruction.

[0006] There is a need for methods and systems that can determine whether a tool is within a target area (e.g., a lesion) with improved accuracy and correctness. The present disclosure addresses the above needs by providing methods and systems for tomosynthesis-based tool in-lesion decision-making. Specifically, the methods herein provide users with quantitative information about the spatial relationship between a thin tool (e.g., a needle) and a target area (e.g., a lesion) in the depth direction. The methods and systems herein can identify the positional relationship between the tool and the lesion (in the depth direction) by identifying the depth of the tool and the lesion, respectively, and determine in a quantitative manner whether the (thin) tool is within the lesion. The term "thin tool" as used herein may refer to at least a portion of a tool (e.g., a distal tip) that is no larger than 1 mm, 2 mm, 3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, 9 mm, or 10 mm in size.

[0007] The method of this article can be applied after setting up the robotic platform, identifying and segmenting the target lesion, performing airway registration, and selecting a single target lesion. The robotic system of this article can utilize integrated tomosynthesis to improve lesion visibility and confirmation of tools within the lesion. For example, during operation, when the endoscope is navigated through the airway to reach the target, the tool-in-lesion mode can be activated to verify whether the fine tool (e.g., a needle passing through the working channel of the endoscope) is within the target. The endoscopic navigation system can use different sensing modes (e.g., camera imaging data, electromagnetic (EM) position data, robot position data, etc.). In some cases, the navigation method may depend on an initial estimate of the position of the endoscope tip relative to the airway to start tracking the endoscope tip. Some endoscopic techniques may involve a three-dimensional (3D) model of the patient's anatomical structure (e.g., a CT image) and use EM fields and position sensors to guide navigation.

[0008] In some cases, a 3D image of the patient's anatomical structure can be taken once or multiple times for various purposes. For example, before a medical operation, a 3D model of the patient's anatomical structure can be created to identify the target position. In some cases, the virtual space of the 3D model, the physical space of the patient's anatomical structure represented by the 3D model, and the precise alignment (e.g., registration) between the EM field can be unknown. Therefore, before the registration is generated, the endoscope position in the patient's anatomical structure cannot be accurately mapped to the corresponding position in the 3D model. In another case, during a surgical operation, 3D imaging can be performed to update / confirm the position of the target (e.g., lesion) in the case of target problem or lesion movement. In some cases, in order to help reach the target tissue position, the position and movement of the medical instrument can be registered with the intraoperative image of the patient's anatomical structure. By registering the image-guided instrument to the image, the instrument can navigate to the natural or surgically created channel in the anatomical system, such as the lungs, colon, intestines, kidneys, heart, circulatory system, etc. In some cases, after a medical instrument (eg, needle, endoscope) reaches a target location or after a surgical procedure is completed, 3D imaging may be performed to confirm whether the instrument or procedure is at the target location.

[0009] In some cases, a fluoroscopic imaging system can be used to determine the position and orientation of medical instruments and patient anatomical structures within the coordinate system of the surgical environment. In order for the imaging data to help correctly position the medical instrument, the coordinate system of the imaging system may be required to reconstruct the 3D model. As described above, multiple 2D fluoroscopic images acquired in different directions / angles can be used to create tomosynthesis or cone beam CT (CBCT) reconstructions to better visualize and provide 3D coordinates of anatomical structures. However, conventional tomosynthesis has poor depth resolution (AP direction), making it difficult to determine whether the tool is within the target area (e.g., lesion). The systems and methods herein beneficially provide confirmation of the tool within the lesion in a quantitative manner, thereby improving the accuracy and correctness of positioning the tool (e.g., needle) relative to the target area.

[0010] In one aspect, a method for navigating a robotic endoscopic device is provided. The method includes: (a) navigating the robotic endoscopic device to a target area inside a body part; (b) obtaining one or more fluoroscopic images using a fluoroscopic imager when a tool is extended into the target area through the robotic endoscopic device, and reconstructing a 3D fluoroscopic image based on the one or more fluoroscopic images; (c) identifying a first slice having a first coordinate corresponding to a center of the target area in a depth direction, and identifying a second slice having a second coordinate corresponding to the tool in the depth direction; and (d) determining whether the tool is within the target area based at least in part on a comparison of a difference between the first coordinate and the second coordinate with a threshold.

[0011] In a related but independent aspect, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations include: a) navigating a robotic endoscopic device to a target region inside a body part; b) acquiring one or more fluoroscopic images using a fluoroscopic imager as a tool is extended through the robotic endoscopic device into the target region, and reconstructing a 3D fluoroscopic image based on the one or more fluoroscopic images; c) identifying a first slice having a first coordinate corresponding to a center of the target region in a depth direction, and identifying a second slice having a second coordinate corresponding to the tool in the depth direction; and d) determining whether the tool is within the target region based at least in part on a comparison of a difference between the first coordinate and the second coordinate to a threshold.

[0012] In some embodiments, the target area is a lesion visible in a 3D fluoroscopic image. In some cases, the first slice is identified by i) displaying the 3D fluoroscopic image within a graphical user interface (GUI), ii) selecting the first slice from the stack of slices when the lesion is in focus. In some cases, the second slice is identified when the tool is in focus. In some cases, the threshold is determined at least in part based on the size of the lesion. For example, the size of the lesion is calculated at least in part based on a 3D model of the lesion obtained from an image acquired prior to (a).

[0013] In some embodiments, the first slice or the second slice is automatically identified based on a clarity metric or a contrast metric for each slice in the depth direction. In some cases, the method further includes displaying the 3D fluoroscopic image in a graphical user interface (GUI) and displaying an overlay of the lesion on each slice in the plurality of stacks in the depth direction. In some cases, the overlay is generated based at least in part on a 3D model of the lesion intersecting each slice. The method may also include determining whether the tool is within the target area by identifying whether the overlay of the lesion appears in the second slice.

[0014] In some embodiments, the method may further include displaying the 3D fluoroscopic image, a first graphical visual indicator representing the first coordinates, and a second graphical visual indicator representing the second coordinates on a graphical user interface (GUI).

[0015] In some embodiments, a 3D fluoroscopic image is reconstructed based on a pose of the fluoroscopic imager. In some cases, the pose of the fluoroscopic imager is estimated based on markers contained in one or more acquired fluoroscopic images. In some cases, the pose of the fluoroscopic imager is obtained based on position sensor data. In some embodiments, the threshold value includes a margin, wherein the margin is determined based on empirical data.

[0016] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in the art from the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. As will be appreciated, the present disclosure is capable of other and different embodiments, and its several details are capable of modification in various obvious respects, all without departing from the present disclosure. Therefore, the drawings and description are to be regarded as illustrative in nature, rather than restrictive.

[0017] Incorporation by reference

[0018] All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. In the event that the publications and patents or patent applications incorporated by reference conflict with the disclosure contained in the specification, the specification is intended to supersede and / or take precedence over any such conflicting material. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The novel features of the present invention are set forth in detail in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by referring to the following detailed description and accompanying drawings (also referred to herein as "diagrams" and "figures") which set forth illustrative embodiments utilizing the principles of the present invention, in which:

[0020] Figure 1 An example workflow for lung cancer diagnosis enabled by the robotic bronchoscopy system described herein is shown.

[0021] Figure 2 Examples of a tool being within a lesion and a tool being outside of a lesion are schematically illustrated.

[0022] Figure 3 An example of a graphical user interface (GUI) showing a reconstructed tomosynthesis image with information about the quantitative tool within a lesion is shown.

[0023] Figure 4 An example of an optimal slice of the tool is shown.

[0024] Figure 5-Figure 9 Various examples of GUIs for assisting in the confirmation of tools within a lesion are shown.

[0025] Fig.10 An example of a tool-in-lesion algorithm according to some embodiments of the present disclosure is shown.

[0026] Fig.11 An exemplary workflow for integrating tool intralesion determination into robotic bronchoscopy system navigation is shown.

[0027] Fig.12 An example of a robotic bronchoscopy system according to some embodiments of the present invention is shown.

[0028] Fig.13 An example of a fluoroscopic (tomosynthesis) imaging system is shown.

[0029] Fig.14 and Fig.15 An example of a flexible endoscope is shown.

[0030] Fig.16An example of an instrument drive mechanism providing a mechanical interface to the handle portion of a robotic bronchoscope is shown.

[0031] Fig.17 An example of the distal tip of an endoscope is shown.

[0032] Fig.18 An example distal portion of a catheter with integrated imaging and lighting devices is shown.

[0033] Fig.19 and Fig. 20 Other examples of GUIs displaying information of quantification tools within lesions are shown.

[0034] Fig.21 An example process for tomosynthesis image reconstruction is shown. DETAILED DESCRIPTION

[0035] Although various embodiments of the present invention have been shown and described herein, it will be readily understood by those skilled in the art that such embodiments are provided by way of example only. Many variations, changes and substitutions may occur to those skilled in the art without departing from the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed.

[0036] Although the exemplary embodiments are primarily directed to bronchoscopy, those skilled in the art will appreciate that this is not intended to be limiting and that the devices described herein may be used in other therapeutic or diagnostic procedures and in other anatomical regions of a patient's body, such as the digestive system, including but not limited to the esophagus, liver, stomach, colon, urinary tract, or the respiratory system, including but not limited to the bronchi, lungs, etc.

[0037] The embodiments disclosed herein can be combined in one or more of many ways to provide improved diagnosis and treatment for patients. The disclosed embodiments can be combined with existing methods and devices to provide improved treatment, such as with known lung diagnosis, surgical methods, and surgical methods of other tissues and organs. It should be understood that any one or more of the structures and steps described herein can be combined with any one or more additional structures and steps of the methods and devices described herein, and the accompanying drawings and supporting text provide descriptions according to the embodiments.

[0038] Although the definitions of treatment planning and diagnostic or surgical procedures described herein are presented in the context of pulmonary diagnosis or surgery, the methods and devices described herein can be used to treat any tissue of the body and any organ and vessel of the body, such as the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testicles, bladder, ears, nose, mouth, soft tissue (such as bone marrow, adipose tissue, muscle, glandular and mucosal tissue, spinal cord and nerve tissue, cartilage), hard biological tissue (such as teeth, bones, etc.), and body cavities and ducts (such as sinuses, ureters, colon, esophagus, lung tract, blood vessels and throat).

[0039] Whenever the term "at least", "greater than", or "greater than or equal to" precedes the first value in a series of two or more values, the term "at least", "greater than", or "greater than or equal to" applies to each value in the series. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0040] Whenever the term "no more than," "less than," or "less than or equal to" precedes the first value in a series of two or more values, the term "no more than," "less than," or "less than or equal to" applies to every value in the series. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0041] As used herein, a processor includes one or more processors, such as a single processor, or multiple processors of a distributed processing system, for example. The controller or processor described herein generally includes a tangible medium for storing instructions for executing process steps, and the processor may include, for example, one or more central processing units, programmable array logic, gate array logic, or field programmable gate arrays. In some cases, one or more processors may be a programmable processor (e.g., a central processing unit (CPU) or a microcontroller), a digital signal processor (DSP), a field programmable gate array (FPGA), and / or one or more advanced RISC machine (ARM) processors. In some cases, one or more processors may be operably coupled to a non-transitory computer-readable medium. A non-transitory computer-readable medium may store logic, code, and / or program instructions that may be executed by one or more processor units to perform one or more steps. A non-transitory computer-readable medium may include one or more memory units (e.g., a removable medium or an external memory, such as an SD card or a random access memory (RAM)). One or more methods or operations disclosed herein may be implemented in a hardware component or a combination of hardware and software, such as an ASIC, a special-purpose computer, or a general-purpose computer.

[0042] As used herein, the terms distal and proximal may generally refer to a position referenced from a device and may be as opposed to an anatomical reference. For example, a distal position of a bronchoscope or catheter may correspond to a proximal position of an elongated member of a patient, and a proximal position of a bronchoscope or catheter may correspond to a distal position of an elongated member of a patient.

[0043] The systems described herein include an elongated portion or elongated member, such as a catheter. Unless the context indicates otherwise, the terms "elongated member", "catheter", "bronchoscope" are used interchangeably throughout the specification. The elongated member can be placed directly into a body lumen or body cavity. In some embodiments, the system may also include a supporting device, such as a robotic manipulator (e.g., a robotic arm), to drive, support, position or control the movement and / or operation of the elongated member. Alternatively or in addition, the supporting device may be a handheld device or other control device, which may or may not include a robotic system. In some embodiments, the system may also include peripheral devices and subsystems, such as an imaging system, which will assist and / or facilitate the navigation of the elongated member to a target site in the subject's body. Such navigation may require a registration process, which will be described later in this article.

[0044] In some embodiments of the present disclosure, a robotic endoscope (e.g., bronchoscopy) system is provided for performing surgical procedures or diagnoses with improved performance at low cost. For example, a robotic bronchoscopy system may include a fully disposable steerable catheter. This can beneficially reduce sterilization requirements, which may be costly or difficult to operate, but sterilization or disinfection may be ineffective. In addition, one challenge of bronchoscopy is reaching the upper lobes of the lungs while navigating through the airways. In some cases, the provided robotic bronchoscopy system may be designed to be able to navigate through airways with small bends in an autonomous or semi-autonomous manner. Autonomous or semi-autonomous navigation may require a registration process. Alternatively, an operator may navigate the robotic bronchoscopy system via a control system with visual guidance.

[0045] The typical lung cancer diagnosis and surgical treatment process can vary widely depending on the technology used by healthcare providers, clinical protocols, and clinical locations. Inconsistent processes can result in delayed diagnosis of early lung cancer, high costs to healthcare systems and patients for diagnosing and treating lung cancer, and high risks of clinical and surgical complications. The provided robotic bronchoscopy system can enable standardized early lung cancer diagnosis and treatment. Figure 1 An example workflow 100 for standardized lung cancer diagnosis enabled by the robotic bronchoscopy system described herein is shown.

[0046] like Figure 1As shown, in some cases, preoperative imaging can be performed to identify lesions 101, and / or to identify airways that will be used for registration and navigation during surgery. Any suitable imaging modality, such as magnetic resonance (MR), positron emission tomography (PET), X-ray, computed tomography (CT), and ultrasound can be used to identify lesions or areas of interest. For example, a preoperative CT scan can be performed on a patient suspected of lung cancer, and suspected lung nodules can be identified in the CT image. The preoperative imaging process can be performed before bronchoscopy. The CT image can be analyzed to generate a map to guide the navigation of the robot bronchoscope 103 during bronchoscopy. For example, lesions or regions of interest (ROI) can be segmented on the image. When imaging the lungs, the channel or path of the lesion can be highlighted on the reconstructed image to plan the navigation path. The reconstructed image can guide the robot bronchoscope to navigate to the target tissue or target site. In some cases, the navigation path can be pre-planned using 3D image data. For example, the catheter can be advanced to the target site under the robot control of the robot bronchoscope system. The catheter can be steered or advanced toward the target site manually, autonomously, or semi-autonomously. In one example, the movement of the catheter can be image-guided so that the insertion and / or steering direction can be automatically controlled. In some cases, a virtual model of the airway and an overlay of the navigation path can be generated using preoperative image data.

[0047] In some cases, the location of the lesion in the preoperative imaging may not be accurate due to various reasons, such as the deviation of the CT from the body. In this case, before the surgical procedure (e.g., biopsy or treatment), the lesion location 107 can be verified when the tip of the endoscope is close to or in proximity to the target (e.g., lesion 105). The exact location of the lesion can be verified or updated with the help of a robotic bronchoscopy system. For example, the bronchoscopy system can provide an interface with an imaging mode (e.g., real-time fluoroscopy) to provide in vivo real-time imaging of the target site and surrounding areas to locate the lesion. In one example, a C-arm or O-arm fluoroscopy imaging system can be used to generate a tomosynthesis or cone beam CT image to verify or update the location of the lesion 107.

[0048] When performing a surgical procedure such as a biopsy 109, various surgical tools (e.g., a biopsy tool, a brush, or forceps) may be inserted into the working channel of the catheter to manually or automatically perform the biopsy or other surgical procedure. In some cases, another fluoroscopy (tomosynthesis) may be performed to confirm that the tool has reached the target site, i.e., confirmation that the tool is within the lesion. Confirmation of the tool within the lesion may be repeated as needed, such as with respect to Fig.11 Further described. The present disclosure provides a navigation method with integrated tools for intralesion detection based on digital tomosynthesis and tomographic coordinate reconstruction technology. Details of the method will be described in detail in Figures 2 to 10Give a description.

[0049] In some cases, once the tool is confirmed to be within the target (e.g., a lesion), a sample of the lesion or any other target tissue can be obtained by inserting the tool through the working channel of the catheter 109. The system allows camera visualization to be maintained throughout the entire procedure, including during the insertion of the tool through the working channel. In some cases, tissue samples can be quickly evaluated on the spot through a rapid field assessment process to determine whether repeated tissue sampling is needed, or to decide on further actions 111. In some cases, the rapid field assessment process can also quickly analyze the tissue sample to determine the next surgical treatment. For example, if the tissue sample is determined to be malignant according to the rapid field assessment process, a manual or robotic treatment instrument can be inserted through the working channel of the robotic bronchoscope and endobronchial treatment of lung cancer 113 can be performed. This facilitates diagnosis and treatment in one session, thereby providing targeted, painless and rapid treatment of early lung cancer.

[0050] As described above, once the catheter is navigated to the target area (e.g., a lesion), a tool such as a needle can be extended onto the catheter and inserted into the target area. For example, after the biopsy needle is placed in the corrected target area (e.g., a lesion), it is necessary to confirm whether the tool is actually in the lesion. In some cases, a fluoroscopic image can be acquired after the catheter tip reaches the target site to confirm the position of the catheter relative to the target position in real time. In some embodiments, the present disclosure provides methods and systems for providing confirmation of the tool in the lesion in a quantitative manner. The method for confirming that the tool is in the lesion may include a tomosynthesis-based method, which will be described later in this article. In some cases, once the final position is confirmed, a cone beam CT scan (e.g., an 8 second scan, 0.5 projections / degree, 396 projections) can be captured and used to perform confirmation of the CBCT tool in the lesion.

[0051] In some cases, confirmation of tool-in-lesion may be defined as placement of a thin tool (eg, needle) within or tangential to the lesion in three orthogonal planes (axial, sagittal, and coronal). Alternatively, tool-in-lesion may not include tangential conditions. Figure 2 An example of a tool being within a lesion 200 and an example of a tool not being within a lesion 210 are schematically illustrated. As shown in the example, a tool being within a lesion can be defined as a tool (e.g., a biopsy needle) that is in a lesion or tangential to a lesion. It should be noted that although the method described herein is described with respect to determining that a biopsy needle is within a lesion, the method is not limited to the type, shape, or size of the tool, or the type, shape, or size of the target area. For example, the tool can be any tool other than a needle, and the target area can be or not be a lesion. The method of confirming that a tool is within a lesion can be applied to any situation where the relative position of a tool relative to a target area is required.

[0052] In aspects of the present disclosure, a method for real-time confirmation that a tool is within a lesion is provided. In some embodiments, the method may be a tomosynthesis-based method. As described above, tomosynthesis reconstruction does not have a uniform resolution. The scanning angle and the number of projections determine the resolution. Due to the limited scanning angle and the small number of projections, the tomosynthesis imaging data in the Fourier domain is incomplete, and it is challenging to create accurate tomosynthesis image reconstructions. The non-isotropic geometry reduces the z-axis (i.e., depth direction) image resolution, which is derived from the Fourier domain rather than directly acquired. The z-axis resolution is improved by increasing the range of scanning angles. However, considering dose limitations, for a certain scanning angle range, there is an optimal number of projections, beyond which the in-plane (i.e., x-axis, y-axis) image quality decreases with the number of projections without further improving the z-axis resolution. The term "depth direction" used herein may refer to the vertical direction or the direction of the central ray. Slices stacked in the depth direction may be in-plane slices.

[0053] In some cases, the reconstructed 3D volumetric image is viewed as multiple thin (e.g., 0.5 to 1 mm) slices parallel to the detector plane. Unlike CT, tomosynthesis slice thickness and reconstruction interval do not correspond one-to-one to compression thickness. The standard method of showing a 3D volumetric data set in three orthogonal planes (e.g., axial, sagittal, and coronal) is ineffective because the resolution of two of the planes is poor. When a stack of 2D reconstructed image slices is presented on a display screen, the user can view each image slice in the coronal plane and navigate through the slice stack in the anterior-posterior (AP) direction. The image stack is typically oriented parallel to the detector plane, with each image separated by a predetermined depth (e.g., 1 mm).

[0054] In some embodiments, the system provides real-time intraoperative imaging to confirm that the tool is within the lesion and overcome the deviation of CT from the body. The digital tomosynthesis tool may involve tomosynthesis reconstruction coordinate technology in lesion confirmation. In some cases, the reconstructed tomosynthesis image is displayed in the user interface along with quantitative tool in lesion information. In some cases, the quantitative tool in lesion information can be based on coordinates representing the depth of the slice displayed in the reconstruction in the anterior-posterior (AP) direction. Details about tomosynthesis image reconstruction will be described later in this article.

[0055] Figure 3 An example of a graphical user interface (GUI) for displaying a reconstructed tomosynthesis image with quantitative tool-in-lesion information is shown. The tomosynthesis image can be constructed from multiple projections using any suitable reconstruction algorithm (e.g., filtered back projection (FBP), iterative algorithm (e.g., algebraic reconstruction technique (ART)), etc.), thereby generating a 3-D volume data set. As shown in the example, the coordinates (e.g., depth) of the displayed slice in the AP direction are displayed with the image.

[0056] In the first example, the coordinates of the slice of the lesion 301 (e.g., 143.5 mm) and the coordinates of the slice of the tool 303 (e.g., 145.0 mm) are determined and displayed on the screen. In some cases, the slice of the lesion 301, 305 may be the best image slice to focus on the lesion. This may be determined based on a sharpness or contrast metric of the slice. In some cases, the slice of the tool 303, 307 may be the best image to focus on the tool. This may also be determined based on the sharpness or contrast of the image slice. Figure 4 An example of a best slice for a tool is shown. The best slice for a tool (eg, needle tip) may be a slice showing the entire needle 401 in focus, or a slice 403 showing a portion of the needle within the lesion area in focus.

[0057] In some embodiments, the coordinate information is used to determine whether the tool is in the lesion by calculating the difference between the optimal slice depth coordinates of the needle and the lesion, respectively. Figure 3 , the distance 311 between the best image slice of the needle 301, 305 and the best image of the lesion 305, 507 can then be calculated and used to determine whether the needle is within the lesion. The distance 311 can be the depth difference (e.g., coordinate in the depth / AP direction) between the slice with the tool in focus and the slice with the lesion in focus.

[0058] In some embodiments, the method may include comparing the distance 311 to a threshold to determine whether the tool is in the lesion. In some cases, the threshold may be determined based on a nodule size (or lesion size) determined by the average of the longest and shortest dimensions on a pre-planning CT scan of the subject (patient). For example, if the average of the longest and shortest dimensions of the target lesion on the pre-planning CT scan is 4 mm, and the depth difference between the best image of the lesion 301 and the best image of the needle 303 is 1.5 mm, which is below the 4 mm threshold, the GUI may display the result as the tool is in the lesion. In another example, the distance is 7.6 mm, which is above the threshold, and the GUI may display the result as the tool is not in the lesion.

[0059] A variety of methods can be used to generate a threshold for determining a tool within a lesion. In some embodiments, the threshold can be determined at least in part based on the lesion size in the same subject. In some cases, the threshold can be the average of the longest and shortest dimensions on a pre-planned CT scan of the subject. In some cases, the threshold can be the radius of the lesion in the AP direction obtained from a pre-planned scan of the subject or any scan performed during the procedure (e.g., a 3D scan performed during navigation or when the endoscope is close to the target). In some cases, the threshold can include a margin such that if abs(xy) < r - e (where r is the radius of the lesion in the AP direction, e is the margin, and x, y are the depth coordinates of the best tool slice and the best lesion slice), then the tool is within the lesion, otherwise if abs(xy) > r + e, then the tool is not within the lesion, and otherwise if abs(xy) = r + e, then the tool is on the boundary / tangent line. In some cases, the margin e can be determined based on empirical data. For example, empirical data, data provided by a doctor, or historical data of similar lesion sizes can be utilized to determine the margin.

[0060] A variety of suitable methods can be used to determine the best slice of the tool and / or the best slice of the lesion. In some embodiments, the best slice of the lesion can be selected via the GUI by visually identifying the slice with a focused lesion by the user. For example, the user can scroll through the slices in the depth direction and can mark the slice with the clearest edge or best contrast of the lesion or needle as the best slice. Alternatively, the best slice of the lesion and / or the tool can be automatically determined. For example, the software can perform an autofocus analysis on the image stack, e.g., by defining a sharpness or contrast metric and selecting the slice that maximizes the selected metric. For example, the sharpness and / or contrast of the lesion area in each slice can be calculated, and the slice with the maximum sharpness measurement can be automatically determined as the best slice of the lesion.

[0061] In some embodiments, the GUI herein can provide a visual indicator or interactive visual guidance to enable the user to interact with the image slices and assist in determining the tool within the lesion. Figures 5 to 9 Various examples of the GUI for assisting in confirming the tool within the lesion are shown. Figure 5An example of a GUI for tool detection within a lesion is shown. The GUI can display a tomosynthesis volume image, where an interactive graphical element (e.g., a slider) 505 allows the user to scroll a stack of slices in the AP direction (depth direction). As shown in the example, depth coordinates 501, 511 associated with the currently displayed slice can also be displayed, for example by superimposing the coordinates on the image slice. The user can slide the slider to scroll the slice. When the lesion 503 is in focus, the user can identify the first slice 500 and obtain the associated depth coordinate 501. For example, the user can click on the slice and mark it as the best slice, and the system automatically records the associated depth coordinate. In some cases, the coordinates of the first slice can be identified as the center of the lesion in the AP direction. Then, when the tool 513 is in focus, the user can identify the second slice 510 and obtain the associated depth coordinate 511. Next, the coordinate difference between the second slice and the center of the lesion in the AP direction (e.g., abs (138mm-134.5mm)) is calculated and compared with a threshold to determine whether the tool is within the lesion.

[0062] In some cases, a mask, outline, or contour of the lesion in each slice may be superimposed on top of the tomosynthesis image to assist in confirmation of the tool within the lesion. Figure 6 An example of a GUI that displays the outline or boundary of a lesion in each slice is shown. A user can scroll through the slice stack and identify the first slice with a focused lesion (e.g., the lesion boundary has the best clarity), and can provide input via the GUI indicating that the coordinates of the first slice are the center of the lesion. For example, a user can click on the first slice 601 to select it as the best slice for the lesion. Based on the center coordinates of the lesion, the system can automatically register the 3D model of the lesion to the image and determine the intersection between the 3D lesion model and each slice. The 3D lesion model can be obtained from a previous scan of the subject (e.g., from a CT pre-planning process). For example, a virtual model of the target tissue can be created using preoperative images by segmenting the target lesion from the image data. The intersection can be used to generate a mask or outline of the lesion 605, 607 in each slice and superimpose it on the image slice. Then, the user can scroll through the slices and identify the second slice 603 when the tool (e.g., a biopsy needle) is focused or at least a portion of the tool is focused. When the mask or outline of the lesion 607 appears in the second slice, the user can determine that the tool is within the lesion. In another example, if the mask or outline does not appear in the second slice that the tool is focused on, it may indicate that the tool is not within the lesion. Such a visual indicator beneficially helps the user confirm that the tool is within the lesion in an intuitive manner.

[0063] In some cases, the radius of the lesion in the AP direction may be displayed as a reference rather than showing the intersection / cross section of the lesion in each slice. Figure 7Another example of a GUI is shown. In this example, once the user identifies a slice 701 of a lesion that is in focus (e.g., best clarity of outline or best contrast), the coordinates of the slice can be used as the center of the lesion. Next, the radius of the lesion in the AP direction (e.g., R=6 mm AP) and the distance 705 between the current slice 703 and the center of the lesion (e.g., d=3.5 mm) can be displayed in each slice. The user can find a second slice with the tool in focus and check whether the corresponding distance d of the second slice is greater than the radius R. If so, it indicates that the tool is not within the lesion.

[0064] Figure 8 and Fig. 9 An example of a GUI showing the depth difference between a slice of a lesion and a slice of a tool is shown. In the illustrated example, once the user identifies the slice of the focused lesion, i.e., the best slice of the lesion, the user can provide input via the GUI, such as by right-clicking the image to mark the current slice as the center of the lesion. A visual indicator (e.g., a thick horizontal line) representing the slice of the center of the lesion and an indicator (e.g., a thin horizontal line) representing the current slice can be displayed, as well as the distance between the two slices. As the user scrolls through the slices, the distance and indicator of the current slice can be dynamically presented.

[0065] In some embodiments, the software can automatically determine the best slice for the lesion (i.e., where the lesion is most focused) or the best slice for the tool (i.e., where the tool tip, tool portion, or entire tool is most focused). For example, a sharpness or contrast metric can be calculated for each slice, and the slice that maximizes the selected metric can be selected. In some cases, the automatically selected slice can be suggested to the user within the GUI, and the user can confirm the automatically selected slice or select a different slice to represent the lesion center and / or tool slice.

[0066] Fig.19 and Fig. 20 Other examples of GUIs displaying information of a quantitative tool within a lesion 1900, 2000 are shown. Fig.19As shown, an overlay of a lesion 1901 can be displayed on a slice. The overlay of lesion 1901 can be generated based on a 3D model of the lesion and an intersection with the slice, as described above. The overlay of the lesion can be turned on or off by the user. The user can scroll through the slices and select the slice with the best focus of the lesion as the best slice. Then, the depth of the best slice can be automatically recorded as the center of the lesion in the depth direction. Then, the user can scroll through the slices and select the slice with the best focus of the needle tip as the best slice for the needle. In some cases, when the user scrolls through the slices, quantitative information indicating the spatial relationship between the tool in the current slice and the center of the lesion can be dynamically displayed on the GUI. For example, a dotted line 1907 can represent the depth position of the center of the lesion, and a solid line 1905 can represent the depth position of the current slice of the needle. The distance between the needle and the center of the lesion (in the current slice or the best slice) can be displayed (e.g., 0.1 mm before, 3.8 mm before, etc.). The GUI can also display the AP diameter (e.g., 10.0 mm) of the target object or lesion 1909, and display the target 1907 observed from the right and left directions. In some cases, in addition to the quantitative indicator, color coding can be used to indicate whether the tool is inside or outside the lesion. For example, when the tool is inside the lesion (such as shown in examples 1900, 1910), lines 1905, 1907 can be displayed in green, and when the tool is outside the lesion (as shown in example 1920), line 1921 representing the best slice for the tool can be displayed in red. Fig. 20 An example of a GUI 2000 is shown that displays a virtual view of a target (e.g., a lesion) and a tool tip. As shown in example 2000, a graphical visual indicator 2005 can be displayed on the virtual view representing the location of the needle tip in 3D space. A second graphical indicator 2003 can be displayed that represents the target or lesion in 3D space. A third indicator 2001 can be displayed that represents the tip of the endoscope. The virtual view can be created based on preoperative image data and / or later acquired 3D image data. The virtual view 2000 superimposed with the needle tip location, lesion location, and endoscope tip can beneficially allow the user to visualize the spatial relationship between these three objects.

[0067] Fig.10An example of the tool-in-lesion algorithm 1000 according to some embodiments of the present disclosure is shown. The method may include identifying a first image slice (e.g., a tomosynthesis slice) representing a target center (e.g., lesion center 1001). The user may identify the first image slice within the GUI, as described above. For example, the user may identify the slice focused on the lesion based on the clarity and / or contrast of the slice. Alternatively or additionally, the first slice may be automatically identified based on the clarity and contrast metrics calculated for each slice. For example, the algorithm may calculate the clarity and / or contrast in the lesion area of each slice (e.g., the user draws a box or region in the image slice indicating the possible location of the lesion), and may select the slice with the maximum metric value as the first slice. The coordinates of the first slice in the AP / depth direction may be used to mark the depth coordinate of the lesion center.

[0068] Next, a second image slice (e.g., a tomosynthesis slice) 1003 corresponding to the tool at the best focus may be identified. The second slice may also be manually selected by the user via the GUI or automatically selected by the system by calculating image metrics (e.g., clarity or contrast). The difference 1005 between the coordinates of the first and second image slices in the AP / depth direction may be calculated and used to determine whether the tool is within the lesion. In some cases, the difference may be automatically compared with a threshold 1007. If the difference is equal to or below the threshold, the algorithm may output a result confirming that the tool is within the lesion. Alternatively, if the difference is above the threshold, the algorithm may output a result indicating that the tool is not within the lesion. Any suitable method as described above may be used to determine the threshold. For example, the threshold may be the average of the longest and shortest dimensions on the pre-planned CT scan of the subject. Alternatively, the threshold may be the radius of the lesion in the AP direction obtained from the pre-planned scan of the subject or any scan performed during the operation. In some cases, the threshold may include a margin such that if abs(xy)<r - e (where r is the radius of the lesion in the AP direction, e is the margin, and x, y are the depth coordinates of the best tool slice and the best lesion slice), the tool is within the lesion, otherwise if abs(xy)>r + e, the tool is not within the lesion, and if abs(xy)=r + e, the tool is on the boundary. In some cases, the margin e may be determined based on empirical data.

[0069] Optionally, the algorithm may generate a visual indicator of the lesion intersecting each image slice (e.g., a mask or contour of the lesion) based on the 3D lesion model and the lesion center coordinates determined in step 1001. The user may determine whether the tool is within the lesion by viewing the visual indicator superimposed on the second slice (when the tool is in focus).

[0070] Fig.11An exemplary workflow for integrating tool in-lesion determination into robotic bronchoscopy system navigation is shown. As shown in the example, planning software can be used to identify and segment the target lesion as well as plan the path. The robotic platform can be set up before airway alignment is performed and a single target lesion is selected. Geolocation guidance can be used to guide the catheter system to the desired target lesion. Next, a tool in-lesion tomosynthesis technique with enhanced fluoroscopy (TILT+) scanning is performed using a 2D fluoroscopic C-arm. In the example, the C-arm scan includes a limited angle circular rotation from 30 degrees left anterior oblique to 30 degrees right anterior oblique. The bronchoscope tip position is marked. Based on a reconstruction algorithm, the two-dimensional images are stacked to create a partial image. The target location is also marked on the partial image. Details regarding the reconstruction of the 3D volumetric tomosynthesis image will be referred to. Fig.13 and Fig.21 Give a description.

[0071] The catheter is then navigated to the correction target and the needle is placed. If needed, the operator can utilize enhanced fluoroscopy to help optimize bronchoscope and tool position. Repeat TILT scans are performed to confirm confirmation of the tool within the lesion. Repeat attempts are allowed at the user's discretion until the needle is in the optimal position. Confirmation of the digital tomosynthesis tool within the lesion is based on the TOMO reconstructed coordinate technique as described above.

[0072] Once the final position is confirmed, a cone beam CT scan can be captured. Confirmation of the CBCT tool within the lesion can be defined as the needle position in three orthogonal planes (axial, sagittal, and coronal) or tangential to the lesion.

[0073] Fig.12 An example of a robotic bronchoscopy system 1200, 1230 is shown according to some embodiments of the present invention. Fig.12As shown, the robotic bronchoscopy system 1200 may include a manipulable catheter assembly 1220 and a robotic support system 1210 for supporting or carrying the manipulable catheter assembly. The manipulable catheter assembly may be a bronchoscope. In some embodiments, the manipulable catheter assembly may be a disposable robotic bronchoscope. In some embodiments, the robotic bronchoscopy system 1200 may include an instrument drive mechanism 1213, which is attached to the arm of the robotic support system. The instrument drive mechanism may be provided by any suitable controller device (e.g., a handheld controller), which may or may not include a robotic system. The instrument drive mechanism may provide a mechanical and electrical interface for the manipulable catheter assembly 1220. The mechanical interface may allow the manipulable catheter assembly 1220 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the manipulable catheter assembly may be attached to the instrument drive mechanism via a quick installation / release device (e.g., a magnet, a spring-loaded level, etc.). In some cases, the manipulable catheter assembly may be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0074] The steerable catheter assembly 1220 may include a handle portion 1223, which may include components configured to process image data, provide power, or establish communication with other external devices. For example, the handle portion 1223 may include circuits and communication elements that enable electrical communication between the steerable catheter assembly 1220 and the instrument drive mechanism 1213 and any other external system or device. In another example, the handle portion 1223 may include circuit elements, such as a power supply for powering the electronic equipment (e.g., camera and LED light) of the endoscope. In some cases, the handle portion may be electrically communicated with the instrument drive mechanism 1213 via an electrical interface (e.g., printed circuit board) so that the communication module of the instrument drive mechanism can receive image / video data and / or sensor data, and can transmit it to other external devices / systems. Alternatively or additionally, the instrument drive mechanism 1213 may only provide a mechanical interface. The handle portion may be electrically communicated with a modular wireless communication device or any other user device (e.g., portable / handheld device or controller) to transmit sensor data and / or receive control signals. Details about the handle portion will be described later in this article.

[0075] Manipulable catheter assembly 1220 can include a flexible slender member 1211 coupled to a handle portion. In some embodiments, the flexible slender member can include a shaft, a manipulable tip, and a manipulable portion. The manipulable catheter assembly can be a disposable robotic bronchoscope. In some cases, only the slender member can be disposable. In some cases, at least a portion of the slender member (e.g., shaft, manipulable tip, etc.) can be disposable. In some cases, the entire manipulable catheter assembly 1220 including the handle portion and the slender member can be disposable. The flexible slender member and the handle portion are designed so that the entire manipulable catheter assembly can be discarded at a low cost. Details about the flexible slender member and the manipulable catheter assembly will be described later in this article.

[0076] In some embodiments, the provided bronchoscopy system may further include a user interface. As shown in the example system 1230, the bronchoscopy system may include a treatment interface module 1231 (user console side) and / or a treatment control module 1233 (patient and robot side). The treatment interface module may allow an operator or user to interact with the bronchoscope during the surgical procedure. In some embodiments, the treatment control module 1233 may be a handheld controller. In some cases, the treatment control module may include a proprietary user input device and one or more additional elements that are detachably coupled to an existing user device to improve the user input experience. For example, a physical trackball or scroll wheel may replace or supplement the functionality of at least one of the virtual graphical elements (e.g., navigation arrows displayed on a touchpad) displayed on a graphical user interface (GUI) by giving it a function similar to the replaced graphical element. Examples of user devices may include, but are not limited to, mobile devices, smart phones / cellular phones, tablet computers, personal digital assistants (PDAs), notebook computers, desktop computers, media content players, and the like. Details about the user interface device and the user console will be described later in this article.

[0077] The user console 1231 may be mounted to the robotic support system 1210. Alternatively or additionally, the user console or a portion of the user console (eg, a therapy interface module) may be mounted to a separate mobile cart.

[0078] The present disclosure provides a robotic intraluminal platform with integrated tool-in-lesion tomosynthesis technology. In some cases, the robotic intraluminal platform can be a bronchoscopy platform. The platform can be configured to perform one or more operations consistent with the methods described herein. Fig.13 An example of a robotic endoluminal platform and its components or subsystems according to some embodiments of the present invention is shown. In some embodiments, the platform can include a robotic bronchoscopy system and one or more subsystems that can be used in conjunction with the robotic bronchoscopy system of the present disclosure.

[0079] In some embodiments, one or more subsystems may include an imaging system, such as a fluoroscopic imaging system for providing real-time imaging of a target site (e.g., including a lesion). Multiple 2D fluoroscopic images may be used to create tomosynthesis or cone beam CT (CBCT) reconstructions to better visualize and provide 3D coordinates of anatomical structures. Fig.13 An example of a fluoroscopic (tomosynthesis) imaging system 1300 is shown. For example, the fluoroscopic (tomosynthesis) imaging system can perform accurate lesion location tracking or confirmation of tool within a lesion before or during a surgical procedure as described above. In some cases, the lesion location can be tracked based on position data about the fluoroscopic (tomosynthesis) imaging system / station (e.g., C-arm) and image data captured by the fluoroscopic (tomosynthesis) imaging system. The lesion location can be registered with the coordinate system of the robotic bronchoscopy system.

[0080] In some cases, the position, posture or motion of the fluoroscopic imaging system can be measured / estimated to align the coordinate system of the image to the robotic bronchoscopy system, or for building a 3D model / image. Any suitable motion / position sensor 1310 provided on the fluoroscopic (tomosynthesis) imaging system can be used to measure the posture or motion of the fluoroscopic (tomosynthesis) imaging system. The motion / position sensor may include, for example, an inertial measurement unit (IMU), one or more gyroscopes, velocity sensors, accelerometers, magnetometers, position sensors (e.g., global positioning system (GPS) sensors), visual sensors (e.g., imaging devices capable of detecting visible light, infrared light, or ultraviolet light, such as cameras), proximity or distance sensors (e.g., ultrasonic sensors, lidar, time of flight or depth cameras), height sensors, posture sensors (e.g., compasses), and / or field sensors (e.g., magnetometers, electromagnetic sensors, radio sensors). In some cases, one or more sensors for tracking the motion and position of the fluoroscopic (tomosynthesis) imaging station may be provided on the imaging station, or away from the imaging station, such as a wall-mounted camera 1320. The C-arm fluoroscopic (tomosynthesis) imaging system is in different (rotational) postures when taking images of a subject. As described above, one or more sensors can capture the various postures.

[0081] In some cases, optical methods can be used to estimate the pose of a C-arm fluoroscopic (tomosynthesis) imaging system. Fig.21 An example process 2100 for tomosynthesis image reconstruction is shown. In some cases, the tomosynthesis image reconstruction of process 2100 may include generating a 3D volume using a combination of X-ray projection images acquired at different angles (acquired by any type of C-arm system).

[0082] The process may include obtaining C-arm video or imaging data using an imaging device (e.g., a C-arm imaging system 105). The C-arm imaging system may include a source (e.g., an X-ray source) and a detector (e.g., an X-ray detector or an X-ray imager). The C-arm system may generate video or imaging data that may be used for tomosynthesis and enhanced fluoroscopy (e.g., a display of real-time 2D fluoroscopy enhanced with superposition of lesions, tools, or other information). In some cases, the original video frame may be used for tomosynthesis and fluoroscopy. However, tomosynthesis may require a unique frame from the C-arm, while enhanced fluoroscopy may operate using duplicate frames from the C-arm, and since it is real-time video, the method herein may provide a unique frame check algorithm to process tomosynthesis video frames to ensure uniqueness. For example, when a new image frame is received, if the current mode is tomosynthesis, the image frame may be processed to determine whether it is a unique frame or a duplicate. The uniqueness check may be based on an image intensity comparison threshold. For example, when the intensity difference from the previous frame is below a predetermined threshold, the frame may be identified as a duplicate frame and may be deleted for tomosynthesis reconstruction. If the current mode is fluoroscopy, the image frames may not be processed to check for uniqueness.

[0083] Operation 2110 may include detecting a video or imaging frame from a C-arm source, and may normalize the video or imaging frame. Normalization may be an image processing technique that changes the range of pixel intensity values ​​in a video or imaging frame. In general, normalization may transform an n-dimensional grayscale image I: (intensity value is in the range (Min,Max)) converted to a new image I NEW : (Intensity value in the range (Min NEW ,Max NEW )). Examples of possible normalization techniques that may be applied to the C-arm video or image frames may include linear scaling, clipping, logarithmic scaling, z-scores, or any other suitable type of normalization.

[0084] Accurate camera pose and camera parameters are important for tomosynthesis image reconstruction and enhanced perspective overlay. In some cases, marker detection 2115 can be performed for pose estimation. In some examples, the marker can be a 2D or 3D marker on the tomosynthesis plate, so that the X-ray projection of the marker on the tomosynthesis plate can be the marker to be detected in the X-ray image. Any suitable image processing or computer vision technique (e.g., motion structure) can be used to detect the marker. For example, OpenCV's blob detection algorithm can be used to detect blob-like markers. In some cases, the detected marker (e.g., blob) can be detected to have certain attributes, such as position, shape, size, color, darkness / brightness, opacity, or other suitable attributes of the marker.

[0085] In some cases, method 2100 may include matching a marker to a plate pattern at 2120. The marker detected in the fluoroscopic image may be matched to a tomosynthesis plate. As described above, the marker may exhibit any number of different physical characteristics (e.g., position, shape, size, color, darkness / brightness, opacity, etc.) that may be detected and used to match the marker to a marker pattern on a tomosynthesis plate. For example, a tomosynthesis plate may have different types of markers, such as large spots and small spots. In some cases, the large spots and small spots may create a pattern that may be used to match the marker pattern in a video or image frame to a pattern on a tomosynthesis plate.

[0086] In some cases, the method 2100 may include finding the best marker match among all video or image frames 2125. The initial marker match may be a match between a marker in a frame and a tomosynthesis plate. In some cases, the pattern of matching markers may be compared on the tomosynthesis plate to find the best match using a Hamming distance. For each frame, a match with a pattern matching score (e.g., the number of matching markers divided by the total number of detected markers) may be obtained. The best match may be determined as the match with the highest pattern matching score among all frames 2125.

[0087] Process 2100 may perform frame-to-frame tracking 2130. At a high level, frame-to-frame tracking 2130 may include propagating marker matches from the best match determined at operation 2125 to the remaining tomosynthesis video and image frames through robust tomosynthesis marker tracking. In some cases, (i) markers in a pair of consecutive frames may be initially matched; (ii) each marker in the first frame may then be matched to the k closest markers in the second frame; (iii) for each pair of matched markers, a motion displacement between the two frames may be calculated; (iv) all markers in the first frame may be transferred to the second frame along with the motion displacement; (v) if the motion displacement between a given transfer point from the first frame and a given point location in the second frame is less than a threshold and the two given markers are of the same type, then the match may be an inlier; (vi) the best match may be the motion with the most inliers. Starting with computed tomosynthesis marker tracking 2130, existing marker matches in the current frame are transferred to marker matches in the next frame. In some cases, this process may be repeated for all frames 2135 to find a marker match for all frames where the markers in all frames match the tomosynthesis slab.

[0088] In some cases, imaging device pose estimation 2140 may include recovering rotation and translation by minimizing reprojection errors from 3D-2D point correspondences to perform pose estimation. In some cases, the camera pose may be recovered from n pairs of point correspondences using perspective-n-point (PnP) pose computation. The minimal form of the PnP problem may be P3P and may be solved by three point correspondences. For each tomosynthesis frame, there may be multiple marker matches and a RANSAC variant of a PnP solver may be used for pose estimation. In some cases, the pose estimate 2140 may be further refined by minimizing the reprojection error using a nonlinear minimization method and starting from an initial pose estimate using a PnP solver.

[0089] The tomosynthesis reconstruction 2145 can be based on the pose estimation results. In some cases, the tomosynthesis reconstruction 2145 can be implemented as a model in Python (or other suitable programming language) using the open source ASTRA (MATLAB and Python Toolbox for High-Performance GPU Primitives for 2D and 3D Tomography) toolbox (or other suitable toolbox or package). In the tomosynthesis reconstruction, the input of the model can be: (i) undistorted and repaired (repair: the process of restoring a damaged image) projection images; (ii) an estimated projection matrix, such as the pose of each projection; and (iii) the size, resolution, and estimated position of the target tomosynthesis reconstruction volume. The output of the model is a tomosynthesis reconstruction (e.g., a volume in NifTI format) 2145. For example, the tomosynthesis reconstruction can include 3D volume data of a surgical area or target scene, where lesions and tools are visible within the 3D volume data.

[0090] Return to reference Fig.13In some embodiments, the location of the lesion can be segmented in the image data (e.g., 3D volume data) captured by the fluoroscopic (tomosynthesis) imaging system with the aid of the signal processing unit 1330. One or more processors of the signal processing unit can be configured to further superimpose the treatment location (e.g., lesion) on the real-time fluoroscopic image / video. For example, the processing unit can be configured to generate an enhancement layer, which includes enhancement information, such as the location of the treatment location or the target site. In some cases, the enhancement layer may also include a graphic mark indicating the path to the target site. The enhancement layer can be a substantially transparent image layer, which includes one or more graphic elements (e.g., boxes, arrows, etc.). The enhancement layer can be superimposed on the optical view of the optical image or video stream captured by the fluoroscopic (tomosynthesis) imaging system, and / or displayed on a display device. The transparency of the enhancement layer allows the user to view the optical image, wherein the graphic elements are superimposed on the optical image. In some cases, the segmented lesion image and the optimal path for guiding the elongated member to the lesion can be superimposed on the real-time tomosynthesis image. This can allow the operator or user to see the exact location of the lesion and the planned path for the movement of the bronchoscope. In some cases, segmented and reconstructed images (eg, CT images as described elsewhere) provided prior to operation of the systems described herein may be superimposed on the real-time images.

[0091] In some embodiments, one or more subsystems of the platform may include one or more therapeutic subsystems, such as manual or robotic instruments (e.g., biopsy needles, biopsy forceps, biopsy brushes) and / or manual or robotic therapeutic instruments (e.g., radiofrequency ablation instruments, cryogenic instruments, microwave instruments, etc.).

[0092] In some embodiments, one or more subsystems of the platform may include a navigation and positioning subsystem. The navigation and positioning subsystem may be configured to build a virtual airway model based on preoperative images (e.g., preoperative CT images or tomosynthesis). The navigation and positioning subsystem may be configured to identify the segmented lesion location in the 3D rendered airway model, and based on the location of the lesion, the navigation and positioning subsystem may generate an optimal path from the main bronchus to the lesion at a recommended approach angle toward the lesion to perform a surgical procedure (e.g., a biopsy).

[0093] At a registration step before driving the bronchoscope to the target site, the system can align the rendered virtual view of the airway with the patient's airway. Image registration can consist of a single registration step or a combination of a single registration step with real-time sensor updates of the registration information. The registration process can include finding a transformation that aligns objects (e.g., airway model, anatomical site) between different coordinate systems (e.g., EM sensor coordinates based on preoperative CT imaging and patient 3D model coordinates). Details on registration are described later in this article.

[0094] Once registered, all airways can be aligned with the preoperatively rendered airways. The position of the bronchoscope within the airway can be tracked and displayed during the driving of the robotic bronchoscope toward the target site. In some cases, a positioning sensor can be used to track the position of the bronchoscope relative to the airway. Sensor fusion techniques can also be used instead of or in combination with the positioning sensor to use other types of sensors (e.g., cameras). A positioning sensor (e.g., an electromagnetic (EM) sensor) can be embedded in the distal end of the catheter, and an EM field generator can be placed next to the patient's torso during surgery. The EM field generator can locate the EM sensor position in 3D space, or it can locate the EM sensor position and orientation in 5D or 6D space. This can provide a visual guide for the operator as he drives the bronchoscope toward the target site.

[0095] In real-time EM tracking, an EM sensor, consisting of one or more sensor coils at one or more locations and orientations embedded in a medical instrument (e.g., the tip of an endoscopic tool), measures changes in the EM field generated by one or more static EM field generators located near the patient. The position information detected by the EM sensor is stored as EM data. An EM field generator (or transmitter) can be placed near the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field senses a small current in the sensor coils of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be registered with the patient's anatomy (e.g., a 3D model) during surgery to determine a registration transformation that aligns a single location in a coordinate system with a location in a preoperative model of the patient's anatomy.

[0096] In some embodiments, the platform of this article can use a fluoroscopic imaging system to determine the position and orientation of medical instruments and patient anatomical structures within the coordinate system of the surgical environment. Specifically, the system and method of this article can use mobile C-arm fluoroscopy as a low-cost and mobile real-time qualitative assessment tool. Fluoroscopy is an imaging method that obtains real-time moving images of patient anatomical structures and medical instruments. The fluoroscopic system may include a C-arm system that provides position flexibility and can be moved in orbit, horizontally and / or vertically via manual or automatic control. Fluoroscopy image data from multiple viewpoints in the surgical environment (i.e., the fluoroscopic imager moves between multiple positions) can be compiled to generate two-dimensional or three-dimensional tomographic images. When a fluoroscopic imager system including a digital detector (e.g., a flat panel detector) is used, the generated and compiled fluoroscopic image data can allow the planar image to be sliced ​​in parallel planes according to tomosynthesis imaging technology. The C-arm imaging system may include a source (e.g., an X-ray source) and a detector (e.g., an X-ray detector or an X-ray imager). The X-ray detector can generate an image representing the intensity of the received X-rays. The imaging system can reconstruct a 3D image based on multiple 2D images acquired from a large angle range. In some cases, the rotation angle range can be at least 120 degrees, 130 degrees, 140 degrees, 150 degrees, 160 degrees, 170 degrees, 180 degrees, or more. In some cases, the 3D image can be generated based on the posture of the X-ray imager.

[0097] The bronchoscope or catheter may be disposable. Fig.14 An example of a flexible endoscope 1400 according to some embodiments of the present disclosure is shown. Fig.14 As shown, the flexible endoscope 1400 may include a handle / proximal portion 1409 and a flexible elongated member to be inserted into the subject. The flexible elongated member may be the same as the flexible elongated member described above. In some embodiments, the flexible elongated member may include a proximal shaft (e.g., an insertion shaft 1401), a manipulable tip (e.g., a tip 1405), and a manipulable segment (active bending segment 1403). The active bending segment and the proximal shaft segment may be the same as the active bending segment, the anti-prolapse passive segment, and the proximal shaft segment described elsewhere herein. Endoscope 1400 may also be referred to as a manipulable catheter assembly, as described elsewhere herein. In some cases, endoscope 1400 may be a single-use robotic endoscope. In some cases, the entire catheter assembly may be disposable. In some cases, at least a portion of the catheter assembly may be disposable. In some cases, the entire endoscope may be released from the instrument drive mechanism and may be discarded. In some embodiments, the endoscope may include different levels of stiffness along the axis to improve functional operation.

[0098] The endoscope or steerable catheter assembly 1400 may include a handle portion 1409 that may include one or more components configured to process image data, provide power, or establish communications with other external devices. For example, the handle portion may include circuitry and communication elements that enable electrical communication between the steerable catheter assembly 1400 and an instrument drive mechanism (not shown) and any other external systems or devices. In another example, the handle portion 1409 may include circuit elements, such as a power source for powering the electronics of the endoscope (e.g., a camera, an electromagnetic sensor, and an LED light).

[0099] One or more components located at the handle can be optimized so that expensive and complex components can be assigned to a robot support system, a handheld controller, or an instrument drive mechanism, thereby reducing costs and simplifying the design of a disposable endoscope. The handle portion or the proximal portion can provide an electrical and mechanical interface to allow electrical and mechanical communication with the instrument drive mechanism. The instrument drive mechanism can include a group of motors that are actuated to drive a group of pull wires of a catheter rotationally. The handle portion of the catheter assembly can be assembled to the instrument drive mechanism so that its pulley / capstan assembly is driven by the group of motors. The number of pulleys can vary based on the pull wire configuration. In some cases, one, two, three, four or more pull wires can be used to articulate a flexible endoscope or catheter.

[0100] The handle portion can be designed to allow the robotic bronchoscope to be disposable at a reduced cost. For example, a classic manual bronchoscope and a robotic bronchoscope can have a cable at the proximal end of the bronchoscope handle. The cable typically includes an illumination fiber, a camera video cable, and other sensor fibers or cables (such as an electromagnetic (EM) sensor or a shape sensing fiber). Such complex cables can be expensive, increasing the cost of the bronchoscope. The provided robotic bronchoscope can have an optimized design so that a simplified structure and components can be employed while retaining mechanical and electrical functionality. In some cases, the handle portion of the robotic bronchoscope can be designed without a cable while providing a mechanical / electrical interface for the catheter.

[0101] An electrical interface (e.g., a printed circuit board) can allow image / video data and / or sensor data to be received by a communication module of an instrument drive mechanism and can be transmitted to other external devices / systems. In some cases, the electrical interface can establish electrical communication without cables or wires. For example, the interface can include pins soldered to an electronic board such as a printed circuit board (PCB). For example, a socket connector (e.g., a female connector) is provided on the instrument drive mechanism as a mating interface. This can beneficially allow the endoscope to be quickly inserted into an instrument drive mechanism or a robot support without the use of additional cables. This type of electrical interface can also be used as a mechanical interface so that when the handle portion is inserted into the instrument drive mechanism, both mechanical coupling and electrical coupling are established. As an alternative or in addition, the instrument drive mechanism can only provide a mechanical interface. The handle portion can be electrically communicated with a modular wireless communication device or any other user device (e.g., a portable / handheld device or a controller) for transmitting sensor data and / or receiving control signals.

[0102] In some cases, the handle portion 1409 may include one or more mechanical control modules, such as a Luer interface 1411, for connecting an irrigation system / suction system. In some cases, the handle portion may include a lever / knob for articulation control. Alternatively, the articulation control may be located at a separate controller attached to the handle portion via the instrument drive mechanism.

[0103] The endoscope can be attached to a robotic support system or a handheld controller via an instrument drive mechanism. The instrument drive mechanism can be provided by any suitable controller device (e.g., a handheld controller), which may or may not include a robotic system. The instrument drive mechanism can provide a mechanical and electrical interface to the steerable catheter assembly 1400. The mechanical interface can allow the steerable catheter assembly 1400 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the steerable catheter assembly can be attached to the instrument drive mechanism via a quick installation / release tool (such as a magnet, a spring-loaded level, etc.). In some cases, the steerable catheter assembly can be manually coupled to the instrument drive mechanism or released from the instrument drive mechanism without the use of a tool.

[0104] In the illustrated example, the distal tip of the catheter or endoscope shaft is configured to articulate / bend in two or more degrees of freedom to provide a desired camera field of view or control the direction of the endoscope. As illustrated in this example, an imaging device (e.g., a camera), a position sensor (e.g., an electromagnetic sensor) 1407 is located at the tip of the catheter or endoscope shaft 1405. For example, the line of sight of the camera can be controlled by controlling the articulation of the active bending segment 1403. In some cases, the angle of the camera can be adjustable so that the line of sight can be adjusted without articulating the distal tip of the catheter or endoscope shaft or in addition to articulating the distal tip of the catheter or endoscope shaft. For example, the camera can be oriented at a certain angle (e.g., tilted) relative to the axial direction of the endoscope tip by means of an optical assembly.

[0105] The distal tip 1405 can be a rigid component that allows sensors (such as electromagnetic (EM) sensors), imaging devices (e.g., cameras), and other electronic components (e.g., LED light sources) to be positioned embedded at the distal tip.

[0106] In real-time EM tracking, an EM sensor consisting of one or more sensor coils in one or more locations and orientations embedded in a medical device (e.g., the tip of an endoscopic tool) measures changes in the EM field generated by one or more static EM field generators positioned at locations close to the patient. The position information detected by the EM sensor is stored as EM data. An EM field generator (or transmitter) can be placed close to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. For example, an EM field generator can be positioned close to the patient's torso during surgery to locate the EM sensor position in 3D space or can locate the EM sensor position and orientation in 5D or 6D space. This can provide visual guidance to the operator when driving the bronchoscope toward the target site.

[0107] The endoscope can have a unique design in the elongated members. In some cases, the active bending section 1403 and the proximal shaft of the endoscope can be composed of a single tube that includes a series of cutouts (e.g., notches, slits, etc.) along its length to allow for improved flexibility, desired stiffness, and anti-prolapse features (e.g., features for defining a minimum bend radius).

[0108] As described above, the active bending segment 1403 can be designed to allow bending (e.g., articulation) in two or more degrees of freedom. Greater degrees of bending, such as 180 degrees and 270 degrees (or other articulation parameters for clinical indications) can be achieved through the unique structure of the active bending segment. In some cases, a variable minimum bending radius along the axial axis of the slender member can be provided, so that the active bending segment or the passive segment can include two or more different minimum bending radii.

[0109] The articulation of the endoscope can be controlled by applying force to the distal end of the endoscope via one or more pull wires. One or more pull wires can be attached to the distal end of the endoscope. In the case of multiple pull wires, pulling one wire at a time can change the orientation of the distal tip to tilt it upward, downward, left, right, or in any desired direction. In some cases, the pull wire can be anchored at the distal tip of the endoscope, travel through the curved segment, and enter the handle where the pull wire is coupled to a drive assembly (e.g., a pulley). The handle pulley can interact with an output shaft from a robotic system.

[0110] In some embodiments, the proximal end or proximal portion of one or more pull wires can be operably coupled to various mechanisms (e.g., gears, pulleys, capstans, etc.) in the handle portion of the catheter assembly. The pull wire can be a metal wire, cable or filament, or it can be a polymer wire, cable or filament. The pull wire can also be made of natural or organic materials or fibers. The pull wire can be any type of suitable wire, cable or filament that can support various loads without deformation, significant deformation or breakage. The distal end / distal portion of one or more pull wires can be anchored or integrated into the distal portion of the catheter so that the control unit's operation of the pull wire can apply a force or tension to the distal portion, which can manipulate or articulate (e.g., upward, downward, pitch, yaw or any direction therebetween) at least the distal portion (e.g., flexible segment) of the catheter.

[0111] The pull wire can be made of any suitable material, such as stainless steel (e.g., SS316), a metal, an alloy, a polymer, nylon, or a biocompatible material. The pull wire can be a wire, a cable, or a filament. In some embodiments, different pull wires can be made of different materials to change the load-bearing capacity of the pull wire. In some embodiments, different segments of the pull wire can be made of different materials to change the stiffness and / or load-bearing capacity along the pull wire. In some embodiments, the pull wire can be used for the transmission of electrical signals.

[0112] The proximal design can improve the reliability of the device without introducing additional costs, thereby allowing a low-cost single-use endoscope. In another aspect of the present invention, a single-use robotic endoscope is provided. The robotic endoscope can be a bronchoscope and can be the same as the steerable catheter assembly described elsewhere herein. Conventional endoscopes can be complex in design and are typically designed to be reused after surgery, which requires thorough cleaning, disinfection, or sterilization after each surgery. Existing endoscopes are typically designed with complex structures to ensure that the endoscope can withstand the cleaning, disinfection, and sterilization processes. The provided robotic bronchoscope can be a single-use endoscope, which can beneficially reduce cross-contamination between patients and infected persons. In some cases, the robotic bronchoscope can be delivered to a medical practitioner in a pre-sterilized package and is intended to be discarded after a single use.

[0113] like Fig.15 As shown, the robotic bronchoscope 1520 may include a handle portion 1513 and a flexible elongated member 1511. In some embodiments, the flexible elongated member 1511 may include a shaft, a steerable tip, and a steerable / active bending section. The robotic bronchoscope 1520 may be used with Fig.14 The robotic bronchoscope may be a single-use robotic endoscope. In some cases, only the catheter may be disposable. In some cases, at least a portion of the catheter may be disposable. In some cases, the entire robotic bronchoscope may be released from the instrument drive mechanism and may be discarded. In some cases, the bronchoscope may include different levels of stiffness along its axis to improve functional operation. In some cases, the minimum bend radius along the axis may vary.

[0114] The robotic bronchoscope can be releasably coupled to an instrument drive mechanism 1520. The instrument drive mechanism 1520 can be mounted to an arm of a robotic support system, or can be mounted to any actuated support system as described elsewhere herein. The instrument drive mechanism can provide a mechanical and electrical interface to the robotic bronchoscope 1510. The mechanical interface can allow the robotic bronchoscope 1510 to be releasably coupled to the instrument drive mechanism. For example, a handle portion of the robotic bronchoscope can be attached to the instrument drive mechanism via a quick mount / release tool (such as a magnet and a spring-loaded level). In some cases, the robotic bronchoscope can be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0115] Fig.16An example of an instrument drive mechanism (IDM) 1620 is shown, which provides a mechanical interface to a handle portion 1613 of a robotic bronchoscope. As shown in this example, the instrument drive mechanism 1620 can include a set of motors that are actuated to rotationally drive a set of pull wires of a flexible endoscope or catheter. The handle portion 1613 of the catheter assembly can be assembled to the instrument drive mechanism so that its pulley assembly or capstan is driven by the set of motors. The number of pulleys can vary based on the pull wire configuration. In some cases, one, two, three, four or more pull wires can be used to articulate a flexible endoscope or catheter.

[0116] The handle portion can be designed to allow the robotic bronchoscope to be disposable at a reduced cost. For example, a classic manual bronchoscope and a robotic bronchoscope can have a cable at the proximal end of the bronchoscope handle. The cable typically includes an illumination fiber, a camera video cable, and other sensor fibers or cables (such as an electromagnetic (EM) sensor or a shape sensing fiber). Such complex cables can be expensive, increasing the cost of the bronchoscope. The provided robotic bronchoscope can have an optimized design so that a simplified structure and components can be employed while retaining mechanical and electrical functionality. In some cases, the handle portion of the robotic bronchoscope can be designed without a cable while providing a mechanical / electrical interface for the catheter.

[0117] Fig.17 An example of a distal tip 1700 of an endoscope is shown. In some cases, the distal portion or tip of the catheter 1700 can be substantially flexible so that it can be manipulated in one or more directions (e.g., pitch, yaw). The catheter can include a tip portion, a curved segment, and an insertion axis. In some embodiments, the catheter can have a variable bending stiffness along the longitudinal axis. For example, the catheter can include multiple segments with different bending stiffnesses (e.g., flexible, semi-rigid, and rigid). The bending stiffness can be changed by selecting materials with different stiffness / rigidity, changing the structure in different sections (e.g., incisions, patterns), adding additional support components, or any combination of the above. In some embodiments, the catheter can have a variable minimum bending radius along the longitudinal axis. Selecting different minimum bending radii at different locations along the catheter can be beneficial to provide anti-prolapse capabilities while still allowing the catheter to reach difficult-to-reach areas. In some cases, the proximal end of the catheter does not need to be highly curved, so the proximal portion of the catheter can be reinforced with additional mechanical structures (e.g., additional material layers) to achieve greater bending stiffness. This design can provide support and stability for the catheter. In some cases, variable bending stiffness can be achieved by using different materials during extrusion of the catheter. This can advantageously allow different stiffness levels along the axis of the catheter during the extrusion manufacturing process without requiring additional fastening or assembly of the different materials.

[0118] The distal portion of the catheter can be manipulated by one or more pull wires 1705. The distal portion of the catheter can be made of any suitable material (such as a copolymer, polymer, metal or alloy) so that it can be bent by the pull wire. In some embodiments, the proximal or terminal end of the one or more pull wires 1705 can be coupled to a drive mechanism (e.g., a gear, pulley, capstan, etc.) via an anchoring mechanism as described above.

[0119] The pull wire 1705 can be a metal wire, cable or silk, or it can be a polymer wire, cable or silk. The pull wire 1705 can also be made of natural or organic materials or fibers. The pull wire 1705 can be any type of suitable wire, cable or silk that can support various loads without deformation, significant deformation or breakage. The distal end or distal portion of one or more pull wires 1705 can be anchored or integrated into the distal portion of the catheter so that the control unit's operation of the pull wire can apply a force or tension to the distal portion, which can manipulate or articulate (e.g., up, down, pitch, yaw or in any direction therebetween) at least the distal portion (e.g., the flexible segment) of the catheter.

[0120] The size of the catheter can enable one or more electronic components to be integrated into the catheter. For example, the outer diameter of the distal tip can be around 4 to 4.4 millimeters (mm), and the diameter of the working channel can be around 2 mm, so that one or more electronic components can be embedded in the wall of the catheter. However, it should be noted that based on different applications, the outer diameter can be in any range less than 4 mm or greater than 4.4 mm, and the diameter of the working channel can be in any range depending on the tool size or specific application.

[0121] One or more electronic components may include an imaging device, an illumination device, or a sensor. In some embodiments, the imaging device may be a video camera 1713. The imaging device may include an optical element and an image sensor for capturing image data. The image sensor may be configured to generate image data in response to the wavelength of light. Various image sensors such as complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) may be used to capture image data. The imaging device may be a low-cost camera. In some cases, the image sensor may be provided on a circuit board. The circuit board may be an imaging printed circuit board (PCB). The PCB may include a plurality of electronic components for processing image signals. For example, a circuit for a CCD sensor may include an A / D converter and an amplifier to amplify and convert an analog signal provided by the CCD sensor. Alternatively, the image sensor may be integrated with an amplifier and a converter to convert an analog signal into a digital signal so that a circuit board may not be required. In some cases, the output of the image sensor or circuit board may be image data (digital signal) that may be further processed by a camera circuit or a processor of the camera. In some cases, the image sensor may include an array of optical sensors.

[0122] The lighting device may include one or more light sources 1711 located at the distal tip. The light source may be a light emitting diode (LED), an organic LED (OLED), a quantum dot, or any other suitable light source. In some cases, the light source may be a small LED or dual tone flash LED lighting for compact designs.

[0123] The imaging device and the lighting device may be integrated into the catheter. For example, the distal portion of the catheter may include a suitable structure that matches at least one size of the imaging device and the lighting device. The imaging device and the lighting device may be embedded in the catheter. Fig.18 An example distal portion of a catheter with an integrated imaging device and lighting device is shown. A camera may be located at the distal portion. The distal tip may have a structure for accommodating a camera, lighting device, and / or a position sensor. For example, a camera may be embedded in a cavity 1810 at the distal tip of the catheter. Cavity 1810 may be integrally formed with the distal portion of the cavity and may have a size that matches the length / width of the camera so that the camera may not move relative to the catheter. The camera may be adjacent to the working channel 1820 of the catheter to provide a near-field view of a tissue or organ. In some cases, the attitude or orientation of the imaging device may be controlled by controlling the rotational movement (e.g., rolling) of the catheter.

[0124] The power of the camera can be provided by a wired cable. In some cases, the cable line can be in a harness, which provides power for the lighting element or other circuit at the distal tip of the camera and the catheter. The camera and / or light source can be powered by a power supply located at the handle portion via a wire, a copper wire, or any other suitable tool that travels through the length of the catheter. In some cases, the real-time image or video of the tissue or organ can be wirelessly transmitted to an external user interface or display. Wireless communication can be WiFi, Bluetooth, RF communication, or other forms of communication. In some cases, the image or video captured by the camera can be broadcast to multiple devices or systems. In some cases, the image and / or video data from the camera can be transmitted to the processor located in the handle portion via a wire, a copper wire, or any other suitable tool along the length of the catheter. The image or video data can be transmitted to an external device / system via the wireless communication component in the handle portion. In some cases, the system can be designed to make the line invisible to the operator or the line is not exposed to the operator.

[0125] In conventional endoscopy, the illumination light can be provided by a fiber optic cable that transmits the light of a light source located at the proximal end of the endoscope to the distal end of the robotic endoscope. In some embodiments of the present disclosure, a small LED lamp can be used and embedded in the distal portion of the catheter to reduce design complexity. In some cases, the distal portion may include a structure 1430 having a size that matches the size of the small LED light source. As shown in the illustrated example, two cavities 1430 can be formed integrally with the catheter to accommodate two LED light sources. For example, the outer diameter of the distal tip can be about 4 to 4.4 millimeters (mm), and the diameter of the working channel of the catheter can be about 2mm, so that two LED light sources can be embedded at the distal end. The outer diameter can be in any range less than 4mm or greater than 4.4mm, and the diameter of the working channel can be in any range according to the size of the tool or the specific application. Any number of light sources can be included. The internal structure of the distal portion can be designed to accommodate any number of light sources.

[0126] In some cases, each LED can be connected to a power cord that can extend to the proximal handle. In some embodiments, the LED can be welded to separate power cords that are subsequently bundled together to form a single strand. In some embodiments, the LED can be welded to a pull cord that supplies power. In other embodiments, the LED can be crimped or directly connected to a single pair of power cords. In some cases, a protective layer such as a thin layer of biocompatible glue can be applied to the front surface of the LED to provide protection while allowing light to be emitted. In some cases, an additional cover 1431 can be placed at the forward end face of the distal tip to provide precise positioning of the LED and sufficient space for glue. The cover 1831 can be composed of a transparent material that matches the refractive index of the glue so that the illumination light can be unblocked.

[0127] Although preferred embodiments of the present invention have been shown and described herein, it will be readily understood to those skilled in the art that such embodiments are provided only as examples. Without departing from the present invention, those skilled in the art may now conceive of many variations, changes and substitutions. It should be understood that various alternatives to the embodiments of the present invention described herein may be adopted when implementing the present invention. The appended claims are intended to define the scope of the present invention, and thus cover methods and structures within the scope of these claims and their equivalents.

Claims

1. A method for navigating a robotic endoscopic device, the method comprising: (a) navigating the robotic endoscopic device to a target area inside a body part; (b) acquiring one or more fluoroscopic images using a fluoroscopic imager while the tool is extended through the robotic endoscopic device to the target area, and reconstructing a 3D fluoroscopic image based on the one or more fluoroscopic images; (c) identifying a first slice having first coordinates corresponding to a center of the target area in a depth direction, and identifying a second slice having second coordinates corresponding to the tool in the depth direction; as well as (d) determining whether the tool is within the target area based at least in part on a comparison of a difference between the first coordinate and the second coordinate to a threshold value. 2 . The method of claim 1 , wherein the target area is a lesion visible in the 3D fluoroscopic image.

3. The method of claim 2, wherein the first slice is identified by: i) displaying the 3D fluoroscopic image within a graphical user interface (GUI), ii) selecting the first slice from a stack of slices when the lesion is in focus. The method of claim 3 , wherein the second slice is identified when the tool is in focus. The method of claim 2 , wherein the threshold is determined based at least in part on a size of the lesion.

6. The method of claim 5, wherein the size of the lesion is calculated based at least in part on a 3D model of the lesion obtained from images acquired prior to (a). 7 . The method of claim 1 , wherein the first slice or the second slice is automatically identified based on a sharpness metric or a contrast metric of each slice in the depth direction.

8. The method of claim 2, further comprising displaying the 3D fluoroscopic image within a graphical user interface (GUI) and displaying a superposition of the lesion on each slice in the plurality of stacks in the depth direction.

9. The method of claim 8, wherein the overlay is generated based at least in part on a 3D model of the lesion intersecting each slice.

10. The method of claim 8, further comprising determining whether the tool is within the target region by identifying whether the superposition of the lesion appears in the second slice.

11. The method of claim 1 , further comprising displaying the 3D fluoroscopic image, a first graphical visual indicator representing the first coordinate, and a second graphical visual indicator representing the second coordinate on a graphical user interface (GUI).

12. The method of claim 1, wherein the 3D fluoroscopic image is reconstructed based on a pose of the fluoroscopic imager.

13. The method of claim 12, wherein the pose of the fluoroscopic imager is estimated based on markers contained in the acquired one or more fluoroscopic images. The method of claim 12 , wherein the pose of the fluoroscopic imager is obtained based on position sensor data.

15. The method of claim 1, wherein the threshold comprises a margin, and wherein the margin is determined based on empirical data.

16. A non-transitory computer-readable storage medium comprising instructions, which, when executed by one or more processors, cause the one or more processors to perform operations comprising: (a) navigating a robotic endoscopic device to a target area inside a body part; (b) acquiring one or more fluoroscopic images using a fluoroscopic imager while the tool is extended through the robotic endoscopic device to the target area, and reconstructing a 3D fluoroscopic image based on the one or more fluoroscopic images; (c) identifying a first slice having first coordinates corresponding to a center of the target area in a depth direction, and identifying a second slice having second coordinates corresponding to the tool in the depth direction; as well as (d) determining whether the tool is within the target area based at least in part on a comparison of a difference between the first coordinate and the second coordinate to a threshold value. 17 . The non-transitory computer-readable storage medium of claim 16 , wherein the target area is a lesion visible in the 3D fluoroscopic image.

18. The non-transitory computer readable storage medium of claim 17, wherein the first slice is identified by: i) displaying the 3D fluoroscopic image within a graphical user interface (GUI), ii) selecting the first slice from a stack of slices when the lesion is in focus.

19. The non-transitory computer readable storage medium of claim 18, wherein the second slice is identified when the tool is in focus.

20. The non-transitory computer-readable storage medium of claim 17, wherein the threshold is determined based at least in part on a size of a lesion.

21. The non-transitory computer-readable storage medium of claim 20, wherein the size of the lesion is calculated based at least in part on a 3D model of the lesion obtained from an image acquired prior to (a). 22 . The non-transitory computer-readable storage medium of claim 16 , wherein the first slice or the second slice is automatically identified based on a sharpness metric or a contrast metric of each slice in the depth direction.

23. The non-transitory computer-readable storage medium of claim 17, wherein the operations further comprise displaying the 3D fluoroscopic image within a graphical user interface (GUI) and displaying an overlay of the lesion on each slice in the plurality of stacks in the depth direction.

24. The non-transitory computer-readable storage medium of claim 23, wherein the overlay is generated based at least in part on a 3D model of the lesion intersecting each slice.

25. The non-transitory computer readable storage medium of claim 23, wherein the operations further comprise determining whether the tool is within the target region by identifying whether the superposition of the lesion appears in the second slice.

26. The non-transitory computer-readable storage medium of claim 16, wherein the operations further comprise displaying the 3D fluoroscopic image, a first graphical visual indicator representing the first coordinate, and a second graphical visual indicator representing the second coordinate on a graphical user interface (GUI).

27. The non-transitory computer-readable storage medium of claim 16, wherein the 3D fluoroscopic image is reconstructed based on a pose of the fluoroscopic imager.

28. The non-transitory computer-readable storage medium of claim 27, wherein the pose of the fluoroscopic imager is estimated based on markers contained in one or more acquired fluoroscopic images.

29. The non-transitory computer-readable storage medium of claim 27, wherein the pose of the fluoroscopic imager is obtained based on position sensor data.

30. The non-transitory computer-readable storage medium of claim 16, wherein the threshold comprises a margin, and wherein the margin is determined based on empirical data.

Images